Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies


e-SocietyRobot new IP...

5:15 pm on Feb 4, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 3, 2004
votes: 0

Found that this previously was not obeying robots.txt. I don't know if it is now or not but here is an IP that took forever to track down. Finally pinned it back on e-SocietyRobot (through translated au and jp server logs of other companies...its amazing what google will index... :))

This accounted for 30k+ hits last month...waste of bandwidth? Does anybody know if this is useful? I don't sell in Japan btw.

5:47 pm on Feb 4, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 27, 2002
votes: 0


Netblock owned by National Institute of Informatics Hitotsubashi, Chiyoda-ku, Tokyo. Does not obey robots.txt

Their about: [yama.info.waseda.ac.jp...] says something alltogether different, with respect to robots.txt - at the bottom of the page.

Currently, our crawler checks /robots.txt file every 6 hours.