Forum Moderators: open

Message Too Old, No Replies

Yahoo Crawlers

Which crawlers are from Yahoo?

         

jimh009

5:29 am on Oct 7, 2004 (gmt 0)

10+ Year Member



Hi,

I've been looking over my logs and can't fully identify whatever crawler Yahoo is now using. I see the following crawlers that seem to be from Yahoo, yet they really don't take much in. Yet, I have 1700 pages or so in Yahoo, so it is obviously grabbing pages from somewhere (although at only half the rate that Google does - I have a 4K page site).

The crawlers I see on my site are :

Yahoo MMCrawler
YahooSeeker

Are these the two main crawlers from Yahoo? And if so, what do each seek out? And does Yahoo have other crawlers I need to look out for?

Thanks.

Jim

wilderness

9:13 pm on Oct 10, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Jim,
I have Yahoo-Slurp crawling constantly.

66.196.90.216 - - [10/Oct/2004:06:19:49 -0700] "GET /robots.txt HTTP/1.0" 200 2717 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]

I believe this the former Inktomi bot.

Yahoo MM I sent packing, which I do to ALL media and image bots.
Perhaps somebody else may expand on Seeker. If they are crawling my sites, I don't recall it at the moment.

Bull has links to thread discussion in his most recent [August 22] "Collated Bot List."

[webmasterworld.com...]
msg# 16 & 17

Don

bull

7:47 pm on Oct 11, 2004 (gmt 0)

10+ Year Member



Re MMCrawler recent
[webmasterworld.com...]