Forum Moderators: open

Message Too Old, No Replies

IBM Research's: "SAI Crawler"

         

Pfui

7:27 pm on Jun 21, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



yktgi01e0-s3.watson.ibm.com
[domino.research.ibm.com...]

robots.txt? YES

-----
I'm kind of amazed things are still issuing forth from .watson.ibm.com. Threads go back years [google.com], at least as long as I've redirected .ibm.com because of the following kinds of activity:

yktgi01e0-s5.watson.ibm.com
Java/1.5.0_11

robots.txt? NO

blueice3n1.nym.ibm.com
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14

robots.txt? YES but... 64 times in ~90 seconds!

keyplyr

8:41 am on Jul 28, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



All they'd need to do is publish what their crawl is used for and I'd consider removing the ban.