Best SEO Blog|Launch your site.

HighRankings Forum moderator Randy made a comment recently that bothered me a little. In a discussion about whether you should use the Hx headers for SEO value (bottom line: do what makes sense for your visitors), Randy wrote “…I can clearly see why the engines would want to reserve the right to see [both CSS and Javascript files]“.

Randy was speaking figuratively (I think) but whenever someone says that a search engine has a right, I stop the horses and get off the wagon.

Here’s the thing: if you extend rights to search engines within the scope of your Web site’s resources, you become obligated to respect those rights (or they can sue you or take other actions against you).

Search engines and Web sites do not have rights with respect to each other. They extend privileges to each other. It is a revocable privilege for any search engine to be able to crawl and index your site. In practice, I revoke that privilege every day for all search engines on some content and for some search engines on all content. That is my right and no search engine has the right to circumvent the blocks I put in place.

On the other hand, there is no law stating that search engines have to observe any particular guidelines on crawling and indexing. That is, we distinguish good search engines from bad search engines on the basis of which ones honor the robots exclusion standard, but no search engine is legally or ethically bound to honor that standard.

Even under common law we do not operate in a contractually defined environment with search engines. They can include whatever content they wish and we exclude whatever search engines we wish, but beyond those inherent rights neither party has any actual defenses that assure them of damages.

If you create a spammy site, Google won’t win any lawsuits against you if your site manages to get past their filters. You just created content and left it for them to find. They took the revocable step of indexing your spam.

And if someone creates a rogue search engine, you won’t win any lawsuits against them if they merely crawl your site. After all, you created a Web site for the purpose of publishing information on the Internet. Everyone is allowed to access that information.

There is a boundary that is seldom crossed. If either party’s behavior becomes abusive to the extent that it interferes with the normal operation of the other party’s services, that might constitute criminal activity. Simply creating Web spam that gets indexed, however, does not interfere with the normal operation of a search engine’s services — it only affects the results of those services.

We cannot inject spam into a search engine. Hence, we cannot be held responsible for what a search engine indexes.

On the other hand, we cannot reasonably argue that a technology which is designed to make information openly available should be considered breached if someone we don’t like merely stops by and grabs a file. If you have secured the data through password protection and someone still grabs it and indexes it, you might have a case. A few courts have issued specific restraining orders in such instances.

John Andrews brought up the issue of “SEO liability” (note: Barry Schwartz followed up on SE Roundtable) on Sphinn. Are we liable for what we do? At some level there are certain obligations you enter into when you take on an SEO contract.

At the heart of the matter is whether you include a clause to ensure that both parties agree to indemnify and hold harmless each other in the event that something doesn’t work right. Without that clause, can your clients sue you?

These two discussions led me to wonder how we might engage in a contractual relationship with search engines. I’m not sure it would be feasible (in fact, there would have to be a three-way contract). Unless a search engine sells you a listing (as some directories do), neither your nor the search engine are offering value or consideration to the other — and as I understand it without exchanging either value or consideration two parties cannot enter into a contractual relationship.

We do see value in being listed in search engines and search engines do see value in indexing content from Web sites, but we are not explicitly exchanging value with the search engines. Our robots.txt files don’t lay down the law, as it were, for either party. After all, no search engine is required by law to either fetch or honor the specifications provided in a robots.txt file.

Which leads me to conclude that if you’re trying to manage your SEO as if you are engaged in a contract with the search engines — that is, if you’re assuming or acting as if the search engines have any rights (or as if you have any rights) — you’re doing it wrong.

Without coercion, without exchange of value or consideration, there is no real legal relationship between search engines and Web sites. We have to operate under the rules of that situation — whatever those rules may be.

Keep in mind that if you’re performing SEO on someone else’s behalf you have to observe the rules of that relationship — which may very well be defined by a contract or statement of work.

I think there is an entire body of search law waiting to be invented.