Forum Moderators: open

Message Too Old, No Replies

         

Strange

2:56 pm on Dec 6, 2004 (gmt 0)

10+ Year Member



A bot with the UA lj1352.inktomisearch.com has nailed one of my sites. Has anyone else seen anything like this?

Thanks,

volatilegx

5:38 pm on Dec 6, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



One of Yahoo's many bots. I've seen Yahoo/Inktomi bots come in on the following User Agents:

# UA "Mozilla/4.05 [en]"
# UA "Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; [inktomi.com...]
# UA "Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]
# UA "YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5; [help.yahoo.com...]
# UA "slurp"
# UA "Fast Crawler v X"
# UA "Fast Crawler v X(compatible; Konqueror/3.2; FreeBSD) (KHTML, like Gecko)"
# UA "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"

fiestagirl

7:19 pm on Dec 6, 2004 (gmt 0)

10+ Year Member



They're also running one with this UA:
Mozilla/4.5 [en] (Win98; I)

JAB Creations

8:54 pm on Dec 6, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yahoo/Slurp from my perspective is the most insistent spider. Last month they hit my site 6500 times. 5182 times in October, at least TWICE as much as google crawls.

Not sure if it's repeating hits on the same files within the same month but I would assume a yes though I have a lot of files!

I do like slurp because it reminds me of old urls I have not set 301s for yet.

Strange

1:57 pm on Dec 7, 2004 (gmt 0)

10+ Year Member



Does this bot have any specific function for yahoo? It isn't identifying itself as slurp which is why I am concerned.

fiestagirl

10:21 pm on Dec 7, 2004 (gmt 0)

10+ Year Member



The robots from the group that you mention, (66.196.91.*) are usually identified with this UA: Mozilla/5.0_(compatible;_Yahoo!_Slurp;_http://help.yahoo.com/help/us/ysearch/slurp).

You don't mention the actual user agent of your culprit so it's hard to help you out. There are robots looking for multi-media files, incl. images and for shopping search in addition to the usual suspects for robots.txt and organic search indexing.

Strange

3:00 pm on Dec 10, 2004 (gmt 0)

10+ Year Member



There was no user agent. It only identified as this lj1352.inktomisearch.com. That is all that showed in logs.