Welcome to WebmasterWorld Guest from

Forum Moderators: incrediBILL & martinibuster

Message Too Old, No Replies

Poll: Should Google add urls to its index from the MediaBot crawler?

At first I thought it did, now I think it doesn't

3:42 pm on Aug 28, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 8, 2002
votes: 0

Any time someone visits a page where you have adsense code on it Google sends a MediaBot crawler to the page to determine relevant ads for that page. I think the bot only gets sent if that page is not already in the cache from a previous MediaBot crawl OR from the normal crawl.

The Poll question is: Do you think Google should use the Mediabot crawls in order to add urls to its index?

The pros:
1. They have the data already.
2. It gives a good idea of pages that are actually viewed.
3. They can selectively choose to add only pages viewed many times by many ip addresses.
4. More pages would make it into the index. They could zero out the relevancy of Mediabot-only pages so results would go to the back of the SERPS and only be useful when no other page is relevant.

The cons:
1. Lots more duplicate content might result.
2. Unfair to those who don't use adsense (who cares? heh heh).
3. Opens Google up to abuse (however, see Item 3 in "Pros" for a possible solution)

My vote:
Add 'em.

Your vote?

4:12 pm on Aug 28, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 8, 2001
votes: 0

It already does.

I have few thousand pages in the index that still haven't seen the proper Googlebot yet.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members