Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

MSN and Yahoo both missing robots.txt

4:43 am on Jul 14, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:May 28, 2004
votes: 0

I found it kind of interesting when I used the robots validator (http://www.searchengineworld.com/cgi-bin/robotcheck.cgi) that both Yahoo and MSN don't have a robots file at all - it just 404's out. Google has a very extensive one.

There was previous debate that lack of a Robots file would possibly lead to problems but this doesn't seem to be the case if these major players aren't using them.

5:57 am on July 14, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0

The only problem it leads to is an error log full of 404 errors. That can be cured by putting up a blank file named robots.txt, which is equivalent to allowing all robots access to all pages, but it stops the 404 errors.


9:42 pm on July 14, 2004 (gmt 0)

New User

10+ Year Member

joined:June 19, 2004
votes: 0

I try to use [searchengineworld.com...] and place my URL

There are lots of error. Why?

3:26 pm on July 29, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:May 14, 2003
votes: 0


well, assuming that the domain listed in your profile is the domain you are talking about, it appears that since you grabbed a copy of the robots.txt from searchengineworld that allows everything, that you have the problem fixed...

how do i know about where it was copied from? because it says so right in it ;)

==== domain obfuscated for TOS ===================================
07/29/04 11:16:45 Browsing http://****-xxxxxxxx-xxxx.com/robots.txt
Fetching http://xxx-xxxxxxxx-xxxx.com/robots.txt ...
GET /robots.txt HTTP/1.1
Host: xxx-xxxxxxxx-xxxx.com
Connection: close
User-Agent: Sam Spade 1.14

HTTP/1.1 200 OK
Date: Thu, 29 Jul 2004 15:16:25 GMT
Server: Apache/1.3.31 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.3.3 FrontPage/ mod_ssl/2.8.18 OpenSSL/0.9.7a
Last-Modified: Thu, 15 Jul 2004 05:08:56 GMT
ETag: "1a093-7b-40f6117e"
Accept-Ranges: bytes
Content-Length: 123
Connection: close
Content-Type: text/plain

# Robots.txt file from http://www.searchengineworld.com
# All robots will spider the domain

User-agent: *


anyway, i guess you have this all sussed out now?


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members