January 05 2009
Things It Is Good To Know
Not every aspect of search engine optimization is about your Web sites, your visibility, your traffic, and your conversions. There are some very naughty, unscrupulous people out there who will walk all over you if you allow them to. They do this in their ongoing efforts to cheat the search engine community.
Yes, friends, I’m talking about those so-called Black Hat Search Engine Spammers.
There are some things members of the black hat community do that I just don’t appreciate, agree with, or tolerate (when they get in my face). Nor should you tolerate their nefarious activities when these anti-social behaviors abuse your resources. Here is a list of things you should know.
Script Kiddies Rape Your Registrations - If you operate a forum, a blog, or some other UGC Web site where people can register and post content, you’re all but hanging a “Spam me” sign on your back. The script kiddie link spammers (most of whom in my experience promote adult content, illegal pharamaceutical content, and transparent affiliate sites) will hammer your server with bogus registrations. Once registered, they will hammer your server with comments, posts, and other link-filled user-generated content.
You can counteract these jerks’ best efforts by:
- Requiring confirmation of all registrations (force them to respond to confirmation emails)
- Moderating all registrations and first time posts (some scripts will post up to 4 or 5 random linkless comments before they start dropping links)
- Blocking all Russian, Ukrainian, Chinese, and other non-English domains
- Blocking hyperactive IP addresses in your “hosts.deny” and/or .htaccess file (or equivalent)
There ARE people in Russia and the Ukraine (and other eastern European nations) who may want to visit your forums, but if you see 100 registrations in one day from accounts like “asgheod@menshealthusa.ua” it’s a pretty safe bet a script kiddie is trying to abuse your service.
Some people favor using CAPTCHAs over confirmation emails. I actually use both. Scripts have been written that actually get past typical CAPTCHAs. How do they do it? I don’t know. They just do.
Of course, Google claims some people in India sell their registration services. These people sign up for Gmail accounts and other free email accounts. I’m currently blocking Gmail and Hotmail from my own forums because of abuse from their users. So the listen to take away here is: When you use free email services, you look like a spammer.
Your images appear in the strangest places - If you post interesting pictures or pictures of celebrities and politicians on your Web site, the odds of those images appearing in other peoples’ forums and blogs are astronomically high. Over the past couple of years many people in the SEO community have noticed increased search referrals from image search, and there has been growing interesting in optimizing for image search.
But here’s the trick: image search referrals rarely convert well for most sites. Those referrals are usually people looking for images to share in forums and blogs. They tend to hotlink to your images and eat up your bandwidth.
Here are a few ways to counteract hotlinking:
- Restrict access to your images to “known” domains. That includes your domains and search engine domains.
- Substitute a standard promotional ad for all hotlinked images.
- Brand all your images with your domain name.
- Block image search from indexing your images.
If you cannot think of a good reason to be found in image search, then just don’t fall into the image search trap. Many people do block the search engines from indexing their images. But there is no hard and fast rule on this. Consumers often use image search to find specific products (particularly where product names are ambiguous, like “Rolex watch” or “Citizen watch”).
If you do want to leverage image search to draw traffic to your site, then make sure that you discourage hotlinking. You cannot prevent people from capturing images and reusing them (even if you go to extreme lengths, all they have to do is take a snapshot of their screen and they have your image). Image search referral can bring customers to your site, but it will more likely bring bandwidth thieves. Make it hard for them to abuse your bandwidth.
Not all email packages are alike - Believe it or not, the old UNIX sendmail service can be more easily secured than some of today’s popular “safer” email packages. I’ve written about Qmail spam exploits in the past. Qmail’s creator claimed it was not vulnerable to spam exploits. The spammers proved him wrong.
Postfix is another email package that is supposedly better than sendmail. However, the braindead designers of Postfix decided it was better to not allow you to blacklist and whitelist domains with “hosts.allow” and “hosts.deny” (which you can do with both sendmail and Qmail). If you go through enough Rube Goldberg machinations, you can sort of filter spam with Postfix but the learning curve and implementation time are ridiculously high compared to older, “less secure” sendmail and other packages.
Internet Service Providers combat email spam in several ways that may affect you regardless of which email package you use.
- They detect and block open relays.
- They perform reverse DNS lookups and reject emails from servers that fail these tests.
- They subscribe to the wrong black lists.
- They white list domains and require you to ask to be included in the white list.
Not all emails are bounced back when they fail these tests. Theoretically, the ISPs are SUPPOSED to send a bounce message, but some ISPs refuse to do that. These rogue admins either believe you’re a spammer because someone says you are a spammer and therefore you should be ignored like a spammer OR they just don’t have a clue about how to handle spam email.
There are some reputable blacklist services (like Spamhaus.org) that allow you to find out your server is an open relay, fix the problem, and then be removed from their black lists. Then there are guys like Joe Jared who don’t believe in treating people fairly. Jared’s blacklists have been known to block as many as 32,000 IP addresses at a time with no option for vetting or whitelisting individual domains and IP addresses. He and his friends historically pursued economic blackmail against hosting providers whom they concluded were “spam-friendly”. By driving customers away from those services, Jared and his friends thought they were doing themselves and everyone else a favor.
A spammer took down Jared’s blacklist at his OsirusSoft site a few years ago. I don’t know if he is currently engaged in blackmailing hosting providers with another blacklist, but the day Jared’s site went day was the one day in history I was applauding an email spammer. I don’t encourage or condone email spam — but Jared’s solution was worse than the cure and it hurt a lot of people, forcing us to move our domains more than once (an expense Jared and his cronies did not care to share or ameliorate in any way).
If your server is failing reverse DNS lookups despite your best efforts to fix the problem, check with your hosting provider. Many Web host services now routinely control email DNS settings at their level to help fight email spam. They will usually work with clients who need to pass reverse DNS lookup. Don’t be rude and angry with your host for failing to disclose this security practice to you. Just be glad they are trying to stay out of the gunsights of idiots and morons like Joe Jared and his friends.
I have found a growing number of ISPs (like AT&T and all their subsidiaries) require human review for domains or IP addresses that fail spam tests. Although it is inconvenient to have to fill out their forms and wait for the review process, I am grateful to them for taking the time to work with Webmasters on these issues. They’ll remove blocks when you show that you’ve closed your open relay or have resolved your reverse DNS lookup issue.
But be warned: if your site is repeatedly hacked or exploited you may find it elevated to a tighter restricted list, and getting off that won’t be so easy. We may be coming to the day when it’s better to allow a reputable third party (or your hosting ISP) handle your domain’s email.
But I’m not ready to hand the reins over to Gmail just yet. I’ve heard some Web sites actually block their emails because of spam abuses.
Take that for what it’s worth.
Written by Michael Martinez


