Welcome to WebmasterWorld Guest from 54.196.147.57

Forum Moderators: goodroi

Message Too Old, No Replies

what is the "KAIST AITrc Crawler" bot?

it ignored my "robots.txt" file and got caught in my bot trap

     

stapel

1:50 pm on Jan 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Twice this morning the user-agent "KAIST AITrc Crawler" ignored my robots.txt file and tried to access a link which triggers an .htaccess ban on bad bots (and other bad people). But I can find no reference to what this user-agent is.

Thank you.

Eliz.

[edited by: stapel at 3:00 pm (utc) on Jan. 20, 2004]

Sanenet

2:08 pm on Jan 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Not 100% sure, but I believe it's Advanced Information Technology Research Centre at the Korean Advanced Institute of Science and Technology (KAIST AITrc). Involved, among other things, in the development of advanced databases and query mechanisms for applications involving terabytes of infomation (search engines?).

stapel

2:59 pm on Jan 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



But they're not advanced enough to understand a "robots.txt" file? Hmm....

Thank you.

Eliz.

admin amoroustoys

12:01 pm on Feb 3, 2004 (gmt 0)



I found out that you can find some info on the crwaler by copy pasting the Ip into you Browser.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month