Forum Moderators: open
I've been looking over my logs and can't fully identify whatever crawler Yahoo is now using. I see the following crawlers that seem to be from Yahoo, yet they really don't take much in. Yet, I have 1700 pages or so in Yahoo, so it is obviously grabbing pages from somewhere (although at only half the rate that Google does - I have a 4K page site).
The crawlers I see on my site are :
Yahoo MMCrawler
YahooSeeker
Are these the two main crawlers from Yahoo? And if so, what do each seek out? And does Yahoo have other crawlers I need to look out for?
Thanks.
Jim
66.196.90.216 - - [10/Oct/2004:06:19:49 -0700] "GET /robots.txt HTTP/1.0" 200 2717 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]
I believe this the former Inktomi bot.
Yahoo MM I sent packing, which I do to ALL media and image bots.
Perhaps somebody else may expand on Seeker. If they are crawling my sites, I don't recall it at the moment.
Bull has links to thread discussion in his most recent [August 22] "Collated Bot List."
[webmasterworld.com...]
msg# 16 & 17
Don