homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

what is the "KAIST AITrc Crawler" bot?
it ignored my "robots.txt" file and got caught in my bot trap

 1:50 pm on Jan 20, 2004 (gmt 0)

Twice this morning the user-agent "KAIST AITrc Crawler" ignored my robots.txt file and tried to access a link which triggers an .htaccess ban on bad bots (and other bad people). But I can find no reference to what this user-agent is.

Thank you.


[edited by: stapel at 3:00 pm (utc) on Jan. 20, 2004]



 2:08 pm on Jan 20, 2004 (gmt 0)

Not 100% sure, but I believe it's Advanced Information Technology Research Centre at the Korean Advanced Institute of Science and Technology (KAIST AITrc). Involved, among other things, in the development of advanced databases and query mechanisms for applications involving terabytes of infomation (search engines?).


 2:59 pm on Jan 20, 2004 (gmt 0)

But they're not advanced enough to understand a "robots.txt" file? Hmm....

Thank you.


admin amoroustoys

 12:01 pm on Feb 3, 2004 (gmt 0)

I found out that you can find some info on the crwaler by copy pasting the Ip into you Browser.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved