MSNbot

grabs robots.txt and leaves

[textex]

pounding, and I mean pounding my robots.txt, and leaves. Why? I don't even have a robots.txt file. But I did have an .htaccess file with an error 404. Perhaps this is the problem?

[jdMorgan]

Do yourself a favor and upload a robots.txt file. It can be blank or it can contain the following:

 User-agent: *
Disallow:

Include the trailing blank line. The effect of both methods is the same: To explicitly allow robots to spider you entire site. However, even having the blank file will prevent 404 errors on attempted robots.txt fetches and help keep your log files clean. Use a plain-text editor to creat the file, such as NotePad.

Next, use the Server Headers [webmasterworld.com] checker to check your server's response to a request for a non-existent file. A common server configuration error can lead to a 302-Found response, rather than the desired 404-Not Found response, and that can play havoc with search engine spiders.

Jim

[textex]

Thanks Jim.