Strange Inktomi Corporation IP

[SEOPTI]

This has been going on for a few days:

74.6.13.111 - - [05/May/2012:13:34:37 -0400] "GET /....html HTTP/1.1" 200 6297 "-" "Mozilla/5.0 (X11; Linux i686 on x86_64; rv:7.0.1) Gecko/ /7.0.1"

The IP belongs to "Inktomi Corporation, Sunnyvale". I think this is Yahoo.

It loads the whole page with all javascript every few minutes.

[lucy24]

I think this is Yahoo.

I sure hope so, because if it isn't, I've been blocking innocent humans at 74.6 without cause ;)

[wilderness]

If their ability to persist is your only concern?
The following will stop them in their tracks:

RewriteCond %{HTTP_USER_AGENT} Linux
RewriteCond %{REMOTE_ADDR} ^74\.6\.
RewriteRule .* - [F]

[SEOPTI]

Thanks for this. Today Inktomi started to bomb all my sites with thousands of page loads and I had to block them.

They even load the Adsense code with ech request. This could lead to an Adsense ban.

[motorhaven]

Thanks. Found this one in my logs as well, and now blocked.

[SEOPTI]

IP range block was really necessary:

SetEnvIf Remote_Addr "^72\.30\." get_out
SetEnvIf Remote_Addr "^74\.6\." get_out
SetEnvIf Remote_Addr "^98\.137\.72\." get_out


<FILES *>
Order Allow,Deny
Allow from all
Deny from env=get_out
</FILES>

[dstiles]

I have the range 74.6.13.87 - 74.6.13.151 listed as a yahoo bot range. It's hit a few dozen times on the server this month but with a non-slurp IP (as noted in the OP) so it's been rejected.

I have a lot of slurp bot ranges listed at 74.6/16 but the rest is "allowed". Does anyone have a real reason to block the rest of the /16 or is part of it sometimes used by humans?

Same as above for 72.30/16 and 98.137/16, much of which is banned, but the "parent" range 98.136.0.0 - 98.139.255.255 is "allowed".

My current annoyance is 98.139.241.224 - 98.139.241.252 which is used by yahoomobile. I get loads of bad hits on those Ips, plus a few that may be valid. Anyone have anything on those?

[SEOPTI]

This is not a crawler for me. A crawler will NOT execute client side Javascript. What would be the reason for them to do this and even load Adsense ads code thousands of times a day?

What is a Yahoo stealth crawl good for? They have Bingbot.

[lucy24]

A while back, I threw in the towel and went to

BrowserMatch Yahoo keep_out

I'm now trying to figure out why I'm suddenly showing up in Yahoo image search-- with results from roboted-out directories that I keep a close watch on.

[SEOPTI]

This bot really loves eating 403 and it never learns.

[wilderness]

In another thread (unknown to me), I had mentioned that I had not seen the activity of Inktomi that I was used pre-2010.

Since re-activation in Feb 2012 the only Slurp requests I've seen are occasional full-page-requests (with supporting images and CSS), and from the 74.6. range.

This morning the following:
72.30.142.221 - - [06/Nov/2012:12:27:30 +0000] "GET /robots.txt HTTP/1.0" 200 2719 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http:// help.yahoo.com/help/us/ysearch/slurp)"

There were two additioanl requests (same IP and UA) for a sub-sub directory page and CSS.

There's some interesting reading on Inktomi:
Inktomi Traffic Server source
Apache Traffic Server (TS)