Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Yahoo! Slurp - robots.txt

Not obeyng robots.txt



5:26 pm on Apr 9, 2004 (gmt 0)

10+ Year Member

Hi all

Yahoo have indexed and dredged up a listing from somewhere for an old site of mine (no snippets just the url and company name) where I have had the following robots.txt installed for a number of months. The site has NOT been available or listed anywhere for at least one year.

User-agent: *
Disallow: /

Every page also contains robots exclusion meta tags

Is any one else seeing this or can advise any help on the robots.txt? It appears correct to me.

I have posted in the "Questions for Yahoo" forum a while ago without a response.

Ray - EW


5:18 pm on Apr 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I just had yahoo vertical crawler ignore robots.txt last night, it accessed a forbidden page that emails me when it gets accessed by a spider. This is very annoying, obviously.

I posted [webmasterworld.com] in the yahoo forum too, I'll see if anyone can help there.


1:57 pm on Apr 27, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

This is occurring for us to - a complete disregard to the robots.txt file. Yahoo! rep stated that YSlurp will obey the file, but I have weblogs that shows it just doesn't care. This is happening across multiple domains.



2:44 pm on May 20, 2004 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


See this post: [webmasterworld.com...]



1:59 pm on Jun 29, 2004 (gmt 0)

10+ Year Member

I THOUGHT Yahoo was ignoring my robots.txt file too. I sent an email to webmasterworldfeedback@yahoo.com last Thursday and I received a response yesterday. Their response correctly explained MY error.

Wow! Imagine... a search engine that responds to questions within two business days, with useful information. Amazing!


Featured Threads

Hot Threads This Week

Hot Threads This Month