homepage Welcome to WebmasterWorld Guest from 54.197.147.90
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt
To use... or not to use? That is the question
panic




msg:1526519
 1:23 am on Oct 30, 2003 (gmt 0)

I allow all content on my site to be crawled. With that said, should I simply not use robots.txt, or have one with the following lines :

user-agent: *
allow: *

Would having the latter of the two improve the number of pages crawled by the spiders?

-panic

 

jimbeetle




msg:1526520
 1:28 am on Oct 30, 2003 (gmt 0)

Make it even easier and just through in an empty robots.txt. No muss, no fuss, no confusion, no 404s.

tedster




msg:1526521
 1:40 am on Oct 30, 2003 (gmt 0)

The robots.txt protocol only allows for "disallow:" statements (not allow: statements) and wildcards don't belong in the disallow.

What you need if you want to allow all spiders to roam your site without restriction is:

user-agent: *
disallow:

Mohamed_E




msg:1526522
 2:13 am on Oct 30, 2003 (gmt 0)

The most logical approach is not to have one; the only function of that file is to disallow access (hence the syntax as explained by tedster).

The only downside is that you will get lots of 404s in your log files. Should you want to eliminate them then use an empty file as suggested by jimbeetle.

Robert Charlton




msg:1526523
 7:10 am on Oct 30, 2003 (gmt 0)

Extended discussion here...

[webmasterworld.com...]
Google and having *no* robots.txt file
could this be hurting your site?

panic




msg:1526524
 1:29 am on Nov 5, 2003 (gmt 0)

I noticed more of my pages are getting indexed now that I'm using a robots.txt that allows everything to be crawled.

-p

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved