Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
This accounted for 30k+ hits last month...waste of bandwidth? Does anybody know if this is useful? I don't sell in Japan btw.
Netblock owned by National Institute of Informatics Hitotsubashi, Chiyoda-ku, Tokyo. Does not obey robots.txt
Their about: [yama.info.waseda.ac.jp...] says something alltogether different, with respect to robots.txt - at the bottom of the page.Currently, our crawler checks /robots.txt file every 6 hours.