Forum Moderators: open

Message Too Old, No Replies

Yahoo search-bot hits from 72.30/16 (Inktomi)

         

SumGuy

2:03 am on Mar 14, 2018 (gmt 0)

5+ Year Member Top Contributors Of The Month



A couple of direct hits to my robots file as well as a non-visible pdf file on my site yesterday from a couple of different IP's in 72.30.14.0/24. These IP's belong to Inktomi but reverse to something.crawl.yahoo.net.

This is the same pdf file that qualityassurance.siteimprove.com (93.160.60.0/24) also likes to access for some reason, for which I have a block in place for, and for which coincidentally it too also tried to access yesterday.

Looking back through 8 years of log files show a grand total of 302 hits from 72.30/16 starting Dec. 2011, all of them showing a Yahoo Slurp user agent (but sometimes the UA is simply LLF). These hits practically ended in August 2013 but the last 8 of them happened Aug/Sept 2017 and Yesterday.

Just mentioning this because of (a) the rarity of seeing Slurp operating from 72.30/16 and (b) having a primary focus on a private pdf on my site that coincidentally is of interest to (c) siteimprove.

keyplyr

3:01 am on Mar 14, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



These IP's belong to Inktomi but reverse to something.crawl.yahoo.net
Yes, Inktomi crawls for Yahoo. Since Yahoo started using the Bing search index, Inktomi has been repurposed for other tasks, but it is normal to see these Inktomi UAs using Yahoo crawl ranges.

Re: coincidences - sometimes it appears that way when we are looking at our logs day to day, but in the big picture these UAs request hundreds of thousands of files across hundreds of thousands of sites, so it's not that coincidental for more than one UA to follow the same link and request the same file.