Forum Moderators: open

Message Too Old, No Replies

enoola crawler

         

keyplyr

6:43 pm on Jan 21, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



137.226.70.164 - - [21/Jan/2006:07:42:29 -0800] "GET /robots.txt HTTP/1.1" 200 2525 "-" "enoola (http://www.enoola.com)"

Requested robots.txt then disobeyed it by taking 7 html files, stopping to get robots.txt again, and resuming with requests for another 7 webpages. Their crawler info page says they support robots.txt, so possibly there's a problem with their bot. I've sent them an email, and am now banning UA via mod_rewrite.

Staffa

10:17 pm on Jan 21, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks for the info keyplyr.
Seems to be another Uni project.