homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Is Google dealing with Yahoo (Inktomi)?
Google Sitemaps file spidered by Yahoo

 3:24 pm on Feb 23, 2006 (gmt 0)

Yes Sir, I provide a xhtml-sitemap for Google Sitemaps.
Nobody but google and me knows what's the file name and the storage folder.
The xhtml-sitemap for Google Sitemaps is not linked from any site.
The server is mine, so it's impossible that a provider could 'sell' information about hostet webs.
Nevertheless I today found a bot/spider with the IP 72.30.#*$!.xxx crawling the xhtml-sitemap for Google Sitemaps.
I did a whois-lookup.
The result was: This IP-Range belongs to Yahoo/Inktomi.
Are they dealing with another?



 6:07 pm on Feb 25, 2006 (gmt 0)

Unlikely that they are co-feeding their bots...

Maybe there was some kind of a Yahoo App on your system that called back home. Just a thought...

Do you have Yahoo! Ads on your pages?

Did you access your sitemap while logged into your Yahoo account?

Do you have Yahoo toolbar installed?


 7:10 pm on Feb 25, 2006 (gmt 0)

It is not unusual to see in my stats Inktomi crawling my sites more often than even G or MSN but I have never done SE submission to Inktomi (or Yahoo) so wondering why that happens? That appears to be more so on my newer sites vs older ones where G and MSN are most common.

Sometimes Inktomi is also the first to visit my newly created websites. They will visit a small temp page online for a few hrs, which has no links, traffic or ads, so wondering how they even know the site was resolving in the first place? Just curious as it seems real odd to me.

No Yahoo toolbar is installed. In fact, never was installed and do not even have an account at Yahoo or Inktomi Search (but do have an acct at YPN which is running on several unrelated and non-linked sites).

P.S. Did not even know Inktomi is connected to Yahoo. Are they and if so how?


 7:29 pm on Feb 25, 2006 (gmt 0)

Yahoo bought Inktomi awhile back.


 4:57 am on Feb 26, 2006 (gmt 0)

Ok thanks Vola, did not hear about it for some reason.

Any idea how their spider finds my newly created temporary websites like they often do?

The details are in my earlier post? I am really curious how they know about them as I see no way they could (but they somehow in fact do know).

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved