homepage Welcome to WebmasterWorld Guest from 107.22.37.143
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Yahoo! Slurp - robots.txt
Not obeyng robots.txt
EarWig




msg:1528341
 5:26 pm on Apr 9, 2004 (gmt 0)

Hi all

Yahoo have indexed and dredged up a listing from somewhere for an old site of mine (no snippets just the url and company name) where I have had the following robots.txt installed for a number of months. The site has NOT been available or listed anywhere for at least one year.

User-agent: *
Disallow: /

Every page also contains robots exclusion meta tags

Is any one else seeing this or can advise any help on the robots.txt? It appears correct to me.

I have posted in the "Questions for Yahoo" forum a while ago without a response.

Regards
Ray - EW

 

isitreal




msg:1528342
 5:18 pm on Apr 26, 2004 (gmt 0)

I just had yahoo vertical crawler ignore robots.txt last night, it accessed a forbidden page that emails me when it gets accessed by a spider. This is very annoying, obviously.

I posted [webmasterworld.com] in the yahoo forum too, I'll see if anyone can help there.

cabowabo




msg:1528343
 1:57 pm on Apr 27, 2004 (gmt 0)

This is occurring for us to - a complete disregard to the robots.txt file. Yahoo! rep stated that YSlurp will obey the file, but I have weblogs that shows it just doesn't care. This is happening across multiple domains.

CaboWabo

jdMorgan




msg:1528344
 2:44 pm on May 20, 2004 (gmt 0)

Earwig,

See this post: [webmasterworld.com...]

Jim

dfud




msg:1528345
 1:59 pm on Jun 29, 2004 (gmt 0)

I THOUGHT Yahoo was ignoring my robots.txt file too. I sent an email to webmasterworldfeedback@yahoo.com last Thursday and I received a response yesterday. Their response correctly explained MY error.

Wow! Imagine... a search engine that responds to questions within two business days, with useful information. Amazing!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved