Welcome to WebmasterWorld Guest from 54.227.1.130

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt

should i even bother if i want the whole site indexed?

     

madeonmoon

3:47 pm on May 7, 2003 (gmt 0)

10+ Year Member



Hi All,

Should I even bother creating robots.txt if there are no areas within my site that should be hidden from the bots. My logs show quite a few of 404's looking for robots.txt. Will i gain any site traffic if i create and properly populate one?

thanks,

james

[edited by: agerhart at 4:40 pm (utc) on May 7, 2003]
[edit reason] No sigs per WebmasterWorld TOS [/edit]

BGumble

3:52 pm on May 7, 2003 (gmt 0)

10+ Year Member



Hi James, welcome to webmasterworld!

All bots I've seen will still spider even without the file present.

Mike12345

3:54 pm on May 7, 2003 (gmt 0)

10+ Year Member



[webmasterworld.com...]

build a robots.txt to keep the nasties out, and let the nice ones in.

:)

creative craig

3:54 pm on May 7, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have put up a file on my site called robots.txt telling all spiders that they can spider, it keeps the 404's down.

Craig

BGumble

4:03 pm on May 7, 2003 (gmt 0)

10+ Year Member



Do you exclude crawlers that produce few results like WiseNut or AltaVista? Or let them through hoping they might one day become relevant. On a large site, extraneous crawlers can chew lots of bandwidth.

Mike12345

4:05 pm on May 7, 2003 (gmt 0)

10+ Year Member



If its not worth its weight in sales then it doesnt get in. Thats my motto, which is probably about to get rubbished.

GaryK

4:18 pm on May 7, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Naw, that's not rubbish. I just think you have to make sure to replace "weight in sales" with a more generalized term that can apply across the board. For me, there are no sales involved in any of my websites so I determine a crawlers worth by how many visitors I wind up getting from the crawler's website. For others it might also take into account how many of those visitors are converted into customers. :)

madeonmoon

4:36 pm on May 7, 2003 (gmt 0)

10+ Year Member



thanks to all!

i think i might create one just in case it could be helpful in the long run. as a benefit, this will decrease 'page not founds'.

Mike12345

4:37 pm on May 7, 2003 (gmt 0)

10+ Year Member



Very valid point GaryK, formally ammended. Thanks :)
 

Featured Threads

Hot Threads This Week

Hot Threads This Month