homepage Welcome to WebmasterWorld Guest from 54.205.241.107
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
e-SocietyRobot
e-SocietyRobot new IP...
beauzero

10+ Year Member



 
Msg#: 543 posted 5:15 pm on Feb 4, 2005 (gmt 0)

Found that this previously was not obeying robots.txt. I don't know if it is now or not but here is an IP that took forever to track down. Finally pinned it back on e-SocietyRobot (through translated au and jp server logs of other companies...its amazing what google will index... :))

133.163.194.50

This accounted for 30k+ hits last month...waste of bandwidth? Does anybody know if this is useful? I don't sell in Japan btw.

 

pendanticist

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 543 posted 5:47 pm on Feb 4, 2005 (gmt 0)

[pgts.com.au...]

Netblock owned by National Institute of Informatics Hitotsubashi, Chiyoda-ku, Tokyo. Does not obey robots.txt

Their about: [yama.info.waseda.ac.jp...] says something alltogether different, with respect to robots.txt - at the bottom of the page.

Currently, our crawler checks /robots.txt file every 6 hours.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved