Slurp Gone Wild

Took only disallowed files

[GaryK]

slurp, yahoo! slurp, slurp/2.0, inktomi slurp, slurp.so/1.0
72.30.161.222
llf531029.crawl.yahoo.net
-----
OrgName: Inktomi Corporation
OrgID: INKT
Address: 701 First Ave
City: Sunnyvale
StateProv: CA

robots.txt? YES. And then proceeded to take only disallowed files!

[enigma1]

It's the reason I don't pay much attention to the robots.txt and just leave it blank. Plus a server can always emit headers and redirect spiders to a forbidden location and they do follow.