Welcome to WebmasterWorld Guest from 35.153.73.72

Forum Moderators: Ocean10000

Message Too Old, No Replies

     
2:56 pm on Dec 6, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 19, 2003
posts:136
votes: 0


A bot with the UA lj1352.inktomisearch.com has nailed one of my sites. Has anyone else seen anything like this?

Thanks,

5:38 pm on Dec 6, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


One of Yahoo's many bots. I've seen Yahoo/Inktomi bots come in on the following User Agents:

# UA "Mozilla/4.05 [en]"
# UA "Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; [inktomi.com...]
# UA "Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]
# UA "YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5; [help.yahoo.com...]
# UA "slurp"
# UA "Fast Crawler v X"
# UA "Fast Crawler v X(compatible; Konqueror/3.2; FreeBSD) (KHTML, like Gecko)"
# UA "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"

7:19 pm on Dec 6, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:May 14, 2002
posts:378
votes: 0


They're also running one with this UA:
Mozilla/4.5 [en] (Win98; I)
8:54 pm on Dec 6, 2004 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member jab_creations is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 26, 2004
posts:3178
votes: 22


Yahoo/Slurp from my perspective is the most insistent spider. Last month they hit my site 6500 times. 5182 times in October, at least TWICE as much as google crawls.

Not sure if it's repeating hits on the same files within the same month but I would assume a yes though I have a lot of files!

I do like slurp because it reminds me of old urls I have not set 301s for yet.

1:57 pm on Dec 7, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 19, 2003
posts:136
votes: 0


Does this bot have any specific function for yahoo? It isn't identifying itself as slurp which is why I am concerned.
10:21 pm on Dec 7, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:May 14, 2002
posts:378
votes: 0


The robots from the group that you mention, (66.196.91.*) are usually identified with this UA: Mozilla/5.0_(compatible;_Yahoo!_Slurp;_http://help.yahoo.com/help/us/ysearch/slurp).

You don't mention the actual user agent of your culprit so it's hard to help you out. There are robots looking for multi-media files, incl. images and for shopping search in addition to the usual suspects for robots.txt and organic search indexing.

3:00 pm on Dec 10, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 19, 2003
posts:136
votes: 0


There was no user agent. It only identified as this lj1352.inktomisearch.com. That is all that showed in logs.