Welcome to WebmasterWorld Guest from 54.226.194.180

Forum Moderators: goodroi

Message Too Old, No Replies

Yahoo! Slurp - robots.txt

Not obeyng robots.txt

     

EarWig

5:26 pm on Apr 9, 2004 (gmt 0)

10+ Year Member



Hi all

Yahoo have indexed and dredged up a listing from somewhere for an old site of mine (no snippets just the url and company name) where I have had the following robots.txt installed for a number of months. The site has NOT been available or listed anywhere for at least one year.

User-agent: *
Disallow: /

Every page also contains robots exclusion meta tags

Is any one else seeing this or can advise any help on the robots.txt? It appears correct to me.

I have posted in the "Questions for Yahoo" forum a while ago without a response.

Regards
Ray - EW

isitreal

5:18 pm on Apr 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I just had yahoo vertical crawler ignore robots.txt last night, it accessed a forbidden page that emails me when it gets accessed by a spider. This is very annoying, obviously.

I posted [webmasterworld.com] in the yahoo forum too, I'll see if anyone can help there.

cabowabo

1:57 pm on Apr 27, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



This is occurring for us to - a complete disregard to the robots.txt file. Yahoo! rep stated that YSlurp will obey the file, but I have weblogs that shows it just doesn't care. This is happening across multiple domains.

CaboWabo

jdMorgan

2:44 pm on May 20, 2004 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Earwig,

See this post: [webmasterworld.com...]

Jim

dfud

1:59 pm on Jun 29, 2004 (gmt 0)

10+ Year Member



I THOUGHT Yahoo was ignoring my robots.txt file too. I sent an email to webmasterworldfeedback@yahoo.com last Thursday and I received a response yesterday. Their response correctly explained MY error.

Wow! Imagine... a search engine that responds to questions within two business days, with useful information. Amazing!

 

Featured Threads

Hot Threads This Week

Hot Threads This Month