Forum Moderators: goodroi

Message Too Old, No Replies

Yahoo Slurp Bot misbehaving

.. or is it someone masquerading?

         

tigertom

9:21 pm on Jan 6, 2007 (gmt 0)

10+ Year Member



[webmasterworld.com...]

I rejigged my robots.txt file to put the 'deny all bots' code first, but it's still attempting to access the forbidden directory.

physics

9:19 pm on Jan 7, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



To check whether it's really Slurp you should check if the IP appears to belong to Yahoo or not.
If you're on linux try:
host x.x.x.x

harrysmit

9:23 am on Jan 17, 2007 (gmt 0)

10+ Year Member



Some robots don't obey/respect robots.txt file, these are actually bad robots and are sent across net for some specific purpose.

Check whether they are acutually Yahoo's Slurp?
You can block the specific ip address from where they belong to.

Marcia

9:27 am on Jan 17, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I just got seriously hammered by the Yahoo bot in a totally non-used directory. Yes, it's coming off a Yahoo IP range. I'm having my host check into the server configuration for me, and also sent off a support request to Yahoo to check it out from their end.