Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Yahoo! Slurp - robots.txt

Not obeyng robots.txt

5:26 pm on Apr 9, 2004 (gmt 0)

Full Member

10+ Year Member

joined:May 24, 2003
votes: 0

Hi all

Yahoo have indexed and dredged up a listing from somewhere for an old site of mine (no snippets just the url and company name) where I have had the following robots.txt installed for a number of months. The site has NOT been available or listed anywhere for at least one year.

User-agent: *
Disallow: /

Every page also contains robots exclusion meta tags

Is any one else seeing this or can advise any help on the robots.txt? It appears correct to me.

I have posted in the "Questions for Yahoo" forum a while ago without a response.

Ray - EW

5:18 pm on Apr 26, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 15, 2004
votes: 0

I just had yahoo vertical crawler ignore robots.txt last night, it accessed a forbidden page that emails me when it gets accessed by a spider. This is very annoying, obviously.

I posted [webmasterworld.com] in the yahoo forum too, I'll see if anyone can help there.

1:57 pm on Apr 27, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 12, 2003
votes: 0

This is occurring for us to - a complete disregard to the robots.txt file. Yahoo! rep stated that YSlurp will obey the file, but I have weblogs that shows it just doesn't care. This is happening across multiple domains.


2:44 pm on May 20, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0


See this post: [webmasterworld.com...]


1:59 pm on June 29, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 30, 2003
votes: 0

I THOUGHT Yahoo was ignoring my robots.txt file too. I sent an email to webmasterworldfeedback@yahoo.com last Thursday and I received a response yesterday. Their response correctly explained MY error.

Wow! Imagine... a search engine that responds to questions within two business days, with useful information. Amazing!


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members