homepage Welcome to WebmasterWorld Guest from 50.17.66.61
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt
should i even bother if i want the whole site indexed?
madeonmoon

10+ Year Member



 
Msg#: 18 posted 3:47 pm on May 7, 2003 (gmt 0)

Hi All,

Should I even bother creating robots.txt if there are no areas within my site that should be hidden from the bots. My logs show quite a few of 404's looking for robots.txt. Will i gain any site traffic if i create and properly populate one?

thanks,

james

[edited by: agerhart at 4:40 pm (utc) on May 7, 2003]
[edit reason] No sigs per WebmasterWorld TOS [/edit]

 

BGumble

10+ Year Member



 
Msg#: 18 posted 3:52 pm on May 7, 2003 (gmt 0)

Hi James, welcome to webmasterworld!

All bots I've seen will still spider even without the file present.

Mike12345

10+ Year Member



 
Msg#: 18 posted 3:54 pm on May 7, 2003 (gmt 0)

[webmasterworld.com...]

build a robots.txt to keep the nasties out, and let the nice ones in.

:)

creative craig

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 18 posted 3:54 pm on May 7, 2003 (gmt 0)

I have put up a file on my site called robots.txt telling all spiders that they can spider, it keeps the 404's down.

Craig

BGumble

10+ Year Member



 
Msg#: 18 posted 4:03 pm on May 7, 2003 (gmt 0)

Do you exclude crawlers that produce few results like WiseNut or AltaVista? Or let them through hoping they might one day become relevant. On a large site, extraneous crawlers can chew lots of bandwidth.

Mike12345

10+ Year Member



 
Msg#: 18 posted 4:05 pm on May 7, 2003 (gmt 0)

If its not worth its weight in sales then it doesnt get in. Thats my motto, which is probably about to get rubbished.

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 18 posted 4:18 pm on May 7, 2003 (gmt 0)

Naw, that's not rubbish. I just think you have to make sure to replace "weight in sales" with a more generalized term that can apply across the board. For me, there are no sales involved in any of my websites so I determine a crawlers worth by how many visitors I wind up getting from the crawler's website. For others it might also take into account how many of those visitors are converted into customers. :)

madeonmoon

10+ Year Member



 
Msg#: 18 posted 4:36 pm on May 7, 2003 (gmt 0)

thanks to all!

i think i might create one just in case it could be helpful in the long run. as a benefit, this will decrease 'page not founds'.

Mike12345

10+ Year Member



 
Msg#: 18 posted 4:37 pm on May 7, 2003 (gmt 0)

Very valid point GaryK, formally ammended. Thanks :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved