homepage Welcome to WebmasterWorld Guest from 54.196.18.51
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
what is the "KAIST AITrc Crawler" bot?
it ignored my "robots.txt" file and got caught in my bot trap
stapel




msg:1529542
 1:50 pm on Jan 20, 2004 (gmt 0)

Twice this morning the user-agent "KAIST AITrc Crawler" ignored my robots.txt file and tried to access a link which triggers an .htaccess ban on bad bots (and other bad people). But I can find no reference to what this user-agent is.

Thank you.

Eliz.

[edited by: stapel at 3:00 pm (utc) on Jan. 20, 2004]

 

Sanenet




msg:1529543
 2:08 pm on Jan 20, 2004 (gmt 0)

Not 100% sure, but I believe it's Advanced Information Technology Research Centre at the Korean Advanced Institute of Science and Technology (KAIST AITrc). Involved, among other things, in the development of advanced databases and query mechanisms for applications involving terabytes of infomation (search engines?).

stapel




msg:1529544
 2:59 pm on Jan 20, 2004 (gmt 0)

But they're not advanced enough to understand a "robots.txt" file? Hmm....

Thank you.

Eliz.

admin amoroustoys




msg:1529545
 12:01 pm on Feb 3, 2004 (gmt 0)

I found out that you can find some info on the crwaler by copy pasting the Ip into you Browser.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved