homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

should i even bother if i want the whole site indexed?

 3:47 pm on May 7, 2003 (gmt 0)

Hi All,

Should I even bother creating robots.txt if there are no areas within my site that should be hidden from the bots. My logs show quite a few of 404's looking for robots.txt. Will i gain any site traffic if i create and properly populate one?



[edited by: agerhart at 4:40 pm (utc) on May 7, 2003]
[edit reason] No sigs per WebmasterWorld TOS [/edit]



 3:52 pm on May 7, 2003 (gmt 0)

Hi James, welcome to webmasterworld!

All bots I've seen will still spider even without the file present.


 3:54 pm on May 7, 2003 (gmt 0)


build a robots.txt to keep the nasties out, and let the nice ones in.


creative craig

 3:54 pm on May 7, 2003 (gmt 0)

I have put up a file on my site called robots.txt telling all spiders that they can spider, it keeps the 404's down.



 4:03 pm on May 7, 2003 (gmt 0)

Do you exclude crawlers that produce few results like WiseNut or AltaVista? Or let them through hoping they might one day become relevant. On a large site, extraneous crawlers can chew lots of bandwidth.


 4:05 pm on May 7, 2003 (gmt 0)

If its not worth its weight in sales then it doesnt get in. Thats my motto, which is probably about to get rubbished.


 4:18 pm on May 7, 2003 (gmt 0)

Naw, that's not rubbish. I just think you have to make sure to replace "weight in sales" with a more generalized term that can apply across the board. For me, there are no sales involved in any of my websites so I determine a crawlers worth by how many visitors I wind up getting from the crawler's website. For others it might also take into account how many of those visitors are converted into customers. :)


 4:36 pm on May 7, 2003 (gmt 0)

thanks to all!

i think i might create one just in case it could be helpful in the long run. as a benefit, this will decrease 'page not founds'.


 4:37 pm on May 7, 2003 (gmt 0)

Very valid point GaryK, formally ammended. Thanks :)

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved