Welcome to WebmasterWorld Guest from 3.93.74.227

Forum Moderators: Ocean10000

Message Too Old, No Replies

Yahoo search-bot hits from 72.30/16 (Inktomi)

     
2:03 am on Mar 14, 2018 (gmt 0)

Junior Member

Top Contributors Of The Month

joined:Sept 8, 2016
posts: 99
votes: 0


A couple of direct hits to my robots file as well as a non-visible pdf file on my site yesterday from a couple of different IP's in 72.30.14.0/24. These IP's belong to Inktomi but reverse to something.crawl.yahoo.net.

This is the same pdf file that qualityassurance.siteimprove.com (93.160.60.0/24) also likes to access for some reason, for which I have a block in place for, and for which coincidentally it too also tried to access yesterday.

Looking back through 8 years of log files show a grand total of 302 hits from 72.30/16 starting Dec. 2011, all of them showing a Yahoo Slurp user agent (but sometimes the UA is simply LLF). These hits practically ended in August 2013 but the last 8 of them happened Aug/Sept 2017 and Yesterday.

Just mentioning this because of (a) the rarity of seeing Slurp operating from 72.30/16 and (b) having a primary focus on a private pdf on my site that coincidentally is of interest to (c) siteimprove.
3:01 am on Mar 14, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


These IP's belong to Inktomi but reverse to something.crawl.yahoo.net
Yes, Inktomi crawls for Yahoo. Since Yahoo started using the Bing search index, Inktomi has been repurposed for other tasks, but it is normal to see these Inktomi UAs using Yahoo crawl ranges.

Re: coincidences - sometimes it appears that way when we are looking at our logs day to day, but in the big picture these UAs request hundreds of thousands of files across hundreds of thousands of sites, so it's not that coincidental for more than one UA to follow the same link and request the same file.