homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

MSN and Yahoo both missing robots.txt

10+ Year Member

Msg#: 420 posted 4:43 am on Jul 14, 2004 (gmt 0)

I found it kind of interesting when I used the robots validator (http://www.searchengineworld.com/cgi-bin/robotcheck.cgi) that both Yahoo and MSN don't have a robots file at all - it just 404's out. Google has a very extensive one.

There was previous debate that lack of a Robots file would possibly lead to problems but this doesn't seem to be the case if these major players aren't using them.



WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 420 posted 5:57 am on Jul 14, 2004 (gmt 0)

The only problem it leads to is an error log full of 404 errors. That can be cured by putting up a blank file named robots.txt, which is equivalent to allowing all robots access to all pages, but it stops the 404 errors.



10+ Year Member

Msg#: 420 posted 9:42 pm on Jul 14, 2004 (gmt 0)

I try to use [searchengineworld.com...] and place my URL

There are lots of error. Why?


10+ Year Member

Msg#: 420 posted 3:26 pm on Jul 29, 2004 (gmt 0)


well, assuming that the domain listed in your profile is the domain you are talking about, it appears that since you grabbed a copy of the robots.txt from searchengineworld that allows everything, that you have the problem fixed...

how do i know about where it was copied from? because it says so right in it ;)

==== domain obfuscated for TOS ===================================
07/29/04 11:16:45 Browsing http://****-xxxxxxxx-xxxx.com/robots.txt
Fetching http://xxx-xxxxxxxx-xxxx.com/robots.txt ...
GET /robots.txt HTTP/1.1
Host: xxx-xxxxxxxx-xxxx.com
Connection: close
User-Agent: Sam Spade 1.14

HTTP/1.1 200 OK
Date: Thu, 29 Jul 2004 15:16:25 GMT
Server: Apache/1.3.31 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.3.3 FrontPage/ mod_ssl/2.8.18 OpenSSL/0.9.7a
Last-Modified: Thu, 15 Jul 2004 05:08:56 GMT
ETag: "1a093-7b-40f6117e"
Accept-Ranges: bytes
Content-Length: 123
Connection: close
Content-Type: text/plain

# Robots.txt file from http://www.searchengineworld.com
# All robots will spider the domain

User-agent: *


anyway, i guess you have this all sussed out now?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved