jimbeetle
msg:1526520 | 1:28 am on Oct 30, 2003 (gmt 0) |
Make it even easier and just through in an empty robots.txt. No muss, no fuss, no confusion, no 404s.
|
tedster
msg:1526521 | 1:40 am on Oct 30, 2003 (gmt 0) |
The robots.txt protocol only allows for "disallow:" statements (not allow: statements) and wildcards don't belong in the disallow. What you need if you want to allow all spiders to roam your site without restriction is:
|
Mohamed_E
msg:1526522 | 2:13 am on Oct 30, 2003 (gmt 0) |
The most logical approach is not to have one; the only function of that file is to disallow access (hence the syntax as explained by tedster). The only downside is that you will get lots of 404s in your log files. Should you want to eliminate them then use an empty file as suggested by jimbeetle.
|
Robert Charlton
msg:1526523 | 7:10 am on Oct 30, 2003 (gmt 0) |
Extended discussion here... [webmasterworld.com...]
Google and having *no* robots.txt file
could this be hurting your site?
|
panic
msg:1526524 | 1:29 am on Nov 5, 2003 (gmt 0) |
I noticed more of my pages are getting indexed now that I'm using a robots.txt that allows everything to be crawled. -p
|
|
