Reduce indexed pages, increase visitors

February 2nd, 2007

I recently read that Shoemoney increased his visitors from Google by 1400% by restricting which pages can be indexed in his robots.txt

It is a common mistake to assume simply that more pages = better Google rankings, there are several reasons why having loads of pages can harm your ranking. Lets explore this some more:

If you have a website about Elvis with 3,000 members and 1,000 pages of article content about Elvis, you may not be doing yourself a favour by showing Google all of this information.

When Google performs a crawl, it can say “okay, I’ve found 1,000 pages about Elvis, I’m not sure what these 3,000 pages are about, but they have a lot of links going to them so they must be fairly important”. The result being Google is slightly confused about the content of your site, it knows it’s about Elvis, but not quite sure the whole site is about that. Links to these member pages are also passing Pagerank (and authority) to these pages which are not important to your search.

By using the robots.txt file to exclude these pages from crawling, you might nudge your PR up a notch by passing the authority to relevant pages and Google will have a better holistic view of the content of your website. Result should be increased visibility for searches around your content!

Have a look at Shoemoney’s robots.txt