I rejigged my robots.txt file to put the 'deny all bots' code first, but it's still attempting to access the forbidden directory.
physics
9:19 pm on Jan 7, 2007 (gmt 0)
To check whether it's really Slurp you should check if the IP appears to belong to Yahoo or not. If you're on linux try: host x.x.x.x
harrysmit
9:23 am on Jan 17, 2007 (gmt 0)
Some robots don't obey/respect robots.txt file, these are actually bad robots and are sent across net for some specific purpose.
Check whether they are acutually Yahoo's Slurp? You can block the specific ip address from where they belong to.
Marcia
9:27 am on Jan 17, 2007 (gmt 0)
I just got seriously hammered by the Yahoo bot in a totally non-used directory. Yes, it's coming off a Yahoo IP range. I'm having my host check into the server configuration for me, and also sent off a support request to Yahoo to check it out from their end.