Best SEO Blog|Launch your site.

Jeremy Bencken wrote an interesting post on his site in which he disclosed an email from 2006 that was accidentally sent to him by a Google engineer. In the message, the engineer mentioned that PageRank determines which pages are assigned to the Supplemental Index.

Let me say something about Google’s Supplemental Results (again). In 2008, I noted on SEO Theory that Danny Sullivan has tried to defuse the SEO industry’s hysteria over Google’s Supplemental Results by pointing out that other search engines have dual indexes, too.

In my SEO Theory post from 2008 I wrote:

Okay, fine. Everyone uses dual indexes. The problem, however, is that pages that rank well in other search engines DUE TO RELEVANCE usually suck in Google’s search results until you point value-passing links at them. That means Google’s Supplemental Index is more than just a “secondary index”.

The Supplemental Index is, so far as we know, still with us today. The earliest confirmation I can find from a Googler that (internal/non-Toolbar) PageRank determined which index a page went into is a post by Matt Cutts from January 2007. Maybe he mentioned it prior to that time. I know Matt brought this up when I asked him a question at SMX Advanced later that year.

Jeremy’s post is trying to support a conjecture Rand Fishkin made on SEOmoz — that Google sets a limit to how many pages it will index for a site. Jeremy argues that your PageRank limits your inclusion in the Main Web Index (really, that is the only index any SEO should want to be in). Jeremy is right, but that doesn’t mean there must be some sort of hard limit. It just means that Matt’s Peanut Butter Principle (”you only get so much PageRank for your site and you have to spread it like peanut butter across a slice of bread”) implies there is a practical limit to how many pages any given amount of PageRank will move into the Main Web Index.

This is, of course, where all the so-called PageRank Sculpting ideas come from. A few people in the SEO industry proposed that it should be possible to direct the flow of your PageRank throughout your site by cutting off “less important” pages so that more PageRank is directed to the “more important” pages.

The concept is theoretically sound. I have said that before and I will say it again. The problem with PageRank Sculpting is that no one outside of Google has been able to do it. Sure, people have changed the flow of PageRank around their sites. In fact, we do THAT constantly by adding and deleting pages, redirecting URLs, building links, etc.

But no one in the SEO industry has the capability to sculpt PageRank flow. They cannot track and measure it — which means they don’t know when Google assesses the PageRank assigned to any given page on a site. If they knew that kind of information, they would be tracking and measuring PageRank. This is the simplest point in the whole debate and yet people continue to miss it.

Do you know how much INTERNAL PageRank Google will assign to any page on your site today, tomorrow, next week, or 2 months from now? Absolutely not. You don’t, you won’t, you can’t, and therefore you cannot sculpt your PageRank.

What you can do is hack your internal navigation structure so that Google and other search engines are less likely to recrawl certain pages as frequently as before. Apparently, enough people did this with such alarming success that Google felt compelled sometime in late 2007 or early 2008 to take the drastic action of changing how it computes PageRank.

Why? Because the brilliant PageRank Sculpting community was screwing up Websites’ search visibility. So let’s zip forward here.

The myth that you can improve your search results by nofollowing or otherwise blocking some of your pages from being indexed persists to this day. We could easily argue that it’s better to leave idiot SEO techniques to the idiots but I feel there is something morally wrong in that position.

Besides, I may one day be asked to help fix a site that has attempted to Sculpt PageRank. I’ve already reviewed quite a few sites in various Webmaster groups and forums where the owners admitted to sculpting PageRank (and they were all complaining about indexing problems — am I the only person to see a correlation here?).

You can’t do it. I admit I cannot do it. And I won’t be stupid enough to try to do it.

But what does all this have to do with the Supplemental Blues? I think I just made that point (twice) but let me put it another way: Sculpting PageRank increases your Supplemental Agony.

The first thing you can do when you’re concerned that your highly relevant content is being overlooked by Google is to see if you’re sculpting PageRank. Someone recently asked me why SEO Theory uses “nofollow” on its link to the root URL. All I can say is that it appears to be built into the Thesis them.

Is it a stupid thing to do? Yes.

Will I fix it? No. Why? Because I don’t care about it.

I can optimize any Website, with or without nofollows embedded on internal pages. It’s not difficult to figure out the solution for when you cannot change the internal structure of a Website that has indexing problems: get more value-passing links to point to the pages that lack PageRank.

I’m pretty sure that all the links people have pointed at SEO-Theory.com through the years will ensure that the root URL has lots of PageRank. Am I sculpting it? No. I’m just not concerned about whether PageRank affects the site’s ability to rank for its own name.

The recent hullabaloo over at SEOmoz concerning their invalid claim to have proven that “PageRank Sculpting still works” shows that people still care about PageRank. The SEO community is extremely hypocritical on this topic. They pay lip service to the principle of “PageRank is not important” but they don’t live it.

Well, I HAVE lived it. I had no idea that the internal links to SEO Theory’s root URL were nofollowed. I never cared about the PageRank. What is the point of caring about PageRank?

Does SEO Theory have pages in the Supplemental Index? I think so. In fact, if I had to bet on whether it does or does not, I would bet that it does. So what?

I had a much bigger cow when I learned that we were using the All-In-One SEO Plug-in for Wordpress. I absolutely hate that plug-in. It uses “rel=’nofollow’” on your category and tag pages by default. Those are some of the most useful pages a blog can have. Who in their right mind would want to block the things?

I find myself talking to Webmasters quite often about which SEO plug-in you need for Wordpress. When I tell them you don’t need any, people get defensive. Why? Because they have read on dozens of blogs and forums that if you’re going to create a blog you need Wordpress but “Wordpress is not SEO-friendly” and therefore you need an SEO plug-in.

Sorry, folks, but that dog won’t hunt. Anyone in the SEO community who is worth their salt knows that the Wordpress development community fixed all those custom field issues a long time ago. They got some very good advice on the matter from a highly reliable source in 2007: Matt Cutts (this is a video).

It doesn’t bother me to know I have Supplemental Pages. It does bother me if I am unable to get pages indexed the way I want them indexed. Sometimes you just need to point a few more links at a deeper page. Sometimes you need to look under the hood and see if maybe someone else’s idea of “good SEO” is pulling you down.

You cannot prove that PageRank Sculpting works because even if you COULD create a set of isolated sites that manage to direct the flow of PageRank as you intend, they would be completely unlike anything in the real world.

Doubling up a document’s PageRank won’t make it more relevant for more queries.

Doubling up a document’s PageRank won’t make it a better quality document.

Doubling up a document’s PageRank MIGHT get it to rank better for its primary keyword if it is stuck in the Supplemental Index.

The real issue facing Webmasters and SEOs alike today, however, is not how much Google’s Supplemental Index is hurting you — it’s how much you’re hurting yourself by implementing truly bad and unreliable SEO methods. You could be doing that right now without realizing it. If you do your job right, though, your site can tolerate a lot of mistakes.

That’s the most important point. It takes a really incompetent SEO to complain that an “About us”, “Privacy Policy”, or other so-called Incidental Page is so powerful it needs to be nofollowed, noindexed, or blocked by robots.txt.

If you are working with a site that has that kind of problem, and you don’t know how to fix it except by hiding the problem, YOU SUCK as an SEO.

And you can quote me on that.