Welcome to WebmasterWorld Guest from 54.146.201.80

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt

To use... or not to use? That is the question

     
1:23 am on Oct 30, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 16, 2003
posts:633
votes: 0


I allow all content on my site to be crawled. With that said, should I simply not use robots.txt, or have one with the following lines :

user-agent: *
allow: *

Would having the latter of the two improve the number of pages crawled by the spiders?

-panic

1:28 am on Oct 30, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Oct 26, 2002
posts:3292
votes: 6


Make it even easier and just through in an empty robots.txt. No muss, no fuss, no confusion, no 404s.
1:40 am on Oct 30, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


The robots.txt protocol only allows for "disallow:" statements (not allow: statements) and wildcards don't belong in the disallow.

What you need if you want to allow all spiders to roam your site without restriction is:

user-agent: *
disallow:
2:13 am on Oct 30, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 14, 2002
posts:1192
votes: 0


The most logical approach is not to have one; the only function of that file is to disallow access (hence the syntax as explained by tedster).

The only downside is that you will get lots of 404s in your log files. Should you want to eliminate them then use an empty file as suggested by jimbeetle.

7:10 am on Oct 30, 2003 (gmt 0)

Moderator from US 

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2000
posts:11310
votes: 163


Extended discussion here...

[webmasterworld.com...]
Google and having *no* robots.txt file
could this be hurting your site?

1:29 am on Nov 5, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 16, 2003
posts:633
votes: 0


I noticed more of my pages are getting indexed now that I'm using a robots.txt that allows everything to be crawled.

-p

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members