Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: goodroi
There was previous debate that lack of a Robots file would possibly lead to problems but this doesn't seem to be the case if these major players aren't using them.
well, assuming that the domain listed in your profile is the domain you are talking about, it appears that since you grabbed a copy of the robots.txt from searchengineworld that allows everything, that you have the problem fixed...
how do i know about where it was copied from? because it says so right in it ;)
==== domain obfuscated for TOS ===================================
07/29/04 11:16:45 Browsing http://****-xxxxxxxx-xxxx.com/robots.txt
Fetching http://xxx-xxxxxxxx-xxxx.com/robots.txt ...
GET /robots.txt HTTP/1.1
User-Agent: Sam Spade 1.14
HTTP/1.1 200 OK
Date: Thu, 29 Jul 2004 15:16:25 GMT
Server: Apache/1.3.31 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.3.3 FrontPage/184.108.40.20634a mod_ssl/2.8.18 OpenSSL/0.9.7a
Last-Modified: Thu, 15 Jul 2004 05:08:56 GMT
# Robots.txt file from http://www.searchengineworld.com
# All robots will spider the domain
anyway, i guess you have this all sussed out now?