1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 10:02 pm Aug 4, 2026

Forum Moderators: open

Message Too Old, No Replies

Anyone know the purpose of this bot?

         

voltrader

5:24 pm on May 7, 2005 (gmt 0)

10+ Year Member



Should I ban it?

versus crawler eda baykan at epfl ch

Brett_Tabke

2:33 pm on May 11, 2005 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Yes, it is a research bot by a phd student.

bull

5:11 pm on May 11, 2005 (gmt 0)

10+ Year Member



Does fetch robots.txt, but does not provide a URL with webmaster information, such as the correct User-agent to specify in robots.txt for Disallow the bot.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

  1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 10:02 pm Aug 4, 2026