homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Yahoo! Slurp - robots.txt
Not obeyng robots.txt

 5:26 pm on Apr 9, 2004 (gmt 0)

Hi all

Yahoo have indexed and dredged up a listing from somewhere for an old site of mine (no snippets just the url and company name) where I have had the following robots.txt installed for a number of months. The site has NOT been available or listed anywhere for at least one year.

User-agent: *
Disallow: /

Every page also contains robots exclusion meta tags

Is any one else seeing this or can advise any help on the robots.txt? It appears correct to me.

I have posted in the "Questions for Yahoo" forum a while ago without a response.

Ray - EW



 5:18 pm on Apr 26, 2004 (gmt 0)

I just had yahoo vertical crawler ignore robots.txt last night, it accessed a forbidden page that emails me when it gets accessed by a spider. This is very annoying, obviously.

I posted [webmasterworld.com] in the yahoo forum too, I'll see if anyone can help there.


 1:57 pm on Apr 27, 2004 (gmt 0)

This is occurring for us to - a complete disregard to the robots.txt file. Yahoo! rep stated that YSlurp will obey the file, but I have weblogs that shows it just doesn't care. This is happening across multiple domains.



 2:44 pm on May 20, 2004 (gmt 0)


See this post: [webmasterworld.com...]



 1:59 pm on Jun 29, 2004 (gmt 0)

I THOUGHT Yahoo was ignoring my robots.txt file too. I sent an email to webmasterworldfeedback@yahoo.com last Thursday and I received a response yesterday. Their response correctly explained MY error.

Wow! Imagine... a search engine that responds to questions within two business days, with useful information. Amazing!

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved