Catchbot

Fetches robots.txt now

[Dijkgraaf]

Update on an old thread: [webmasterworld.com...]
/search_engine_spiders/3696307.htm

UA: CatchBot/1.0; +http://www.catchbot.com
IP: 203.6.203.nnn
rDNS: Reverse DNS: 203-6-203-nnn.reed-elsevier.com.au.
Reverse DNS authenticity: [Could be forged: hostname 203-6-203-nnn.reed-elsevier.com.au. does not exist]

This bot now does fetch robots.txt and says it obeys it on it's page.

[enigma1]

Well I checked my logs and does seem to read the robots.txt. It goes straight for pages.

The rDNS I checked with the IPs validate.

[Pfui]

This just in...

203-6-203-80.reed-elsevier.com.au
CatchBot/1.0; +http://www.catchbot.com

robots.txt? Yes