homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Ive never used robots.txt

5+ Year Member

Msg#: 3748388 posted 2:58 am on Sep 20, 2008 (gmt 0)

Hi Just a question. Am I missing somthing. I have a site that I want all the pages crawled and they have been. Ive never used a robots.txt file. Shoul I at least have one that allows all or are some sites missing me because I dont have one. Any help is appreciated.



WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 3748388 posted 3:33 am on Sep 20, 2008 (gmt 0)

If you don't have a robots.txt file, then all crawlers will deem themselves allowed to fetch all pages and resources of your site.

However, you will find that once you move into the finer points of refining your site to attract more (and more appropriate) visitors, the fact that your server access logs and "Website statistics" reports are filled with 404-Not Found errors caused by robots trying to fetch robots.txt will become a nuisance.

You can always upload a blank file called robots.txt to avoid this.

In addition most Webmasters would Disallow robots from fetching any page that triggers an action, such as "voting" or sending an e-mail. Otherwise, you might find that between Googlebot and all the others, your "vote count" would be seriously skewed, and your in-box would be quite full. Could be worse -- They might get into your shopping cart and deplete your inventory in a matter of hours. ;)

Be aware that a Disallow in robots.txt is a "request" -- Malicious and incompetent robots can ignore it at will. For those, a bit of server-side code to actually block access is needed.


[edited by: jdMorgan at 3:34 am (utc) on Sep. 20, 2008]


5+ Year Member

Msg#: 3748388 posted 3:25 pm on Sep 20, 2008 (gmt 0)

Thank you Jim. I appreciate your help


5+ Year Member

Msg#: 3748388 posted 3:21 am on Sep 23, 2008 (gmt 0)

Remember too, that the game isn't just about getting heaps of traffic, it is about getting conversion driven traffic. The more pages and areas that you let the SEs index the more chances for dupe content issues, and unqualified traffic.

If you have a sound information architecture, then you might be fine. But do you really need your Privacy page indexed?

Cutting the fat is key.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved