robots.txt

should i even bother if i want the whole site indexed?

[madeonmoon]

Hi All,

Should I even bother creating robots.txt if there are no areas within my site that should be hidden from the bots. My logs show quite a few of 404's looking for robots.txt. Will i gain any site traffic if i create and properly populate one?

thanks,

james

[edited by: agerhart at 4:40 pm (utc) on May 7, 2003]
[edit reason] No sigs per WebmasterWorld TOS [/edit]

[BGumble]

Hi James, welcome to webmasterworld!

All bots I've seen will still spider even without the file present.

[Mike12345]

[webmasterworld.com...]

build a robots.txt to keep the nasties out, and let the nice ones in.

:)

[creative craig]

I have put up a file on my site called robots.txt telling all spiders that they can spider, it keeps the 404's down.

Craig

[BGumble]

Do you exclude crawlers that produce few results like WiseNut or AltaVista? Or let them through hoping they might one day become relevant. On a large site, extraneous crawlers can chew lots of bandwidth.

[Mike12345]

If its not worth its weight in sales then it doesnt get in. Thats my motto, which is probably about to get rubbished.

[GaryK]

Naw, that's not rubbish. I just think you have to make sure to replace "weight in sales" with a more generalized term that can apply across the board. For me, there are no sales involved in any of my websites so I determine a crawlers worth by how many visitors I wind up getting from the crawler's website. For others it might also take into account how many of those visitors are converted into customers. :)

[madeonmoon]

thanks to all!

i think i might create one just in case it could be helpful in the long run. as a benefit, this will decrease 'page not founds'.

[Mike12345]

Very valid point GaryK, formally ammended. Thanks :)