Welcome to WebmasterWorld Guest from 54.167.83.224

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt

should i even bother if i want the whole site indexed?

     
3:47 pm on May 7, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 6, 2003
posts:54
votes: 0


Hi All,

Should I even bother creating robots.txt if there are no areas within my site that should be hidden from the bots. My logs show quite a few of 404's looking for robots.txt. Will i gain any site traffic if i create and properly populate one?

thanks,

james

[edited by: agerhart at 4:40 pm (utc) on May 7, 2003]
[edit reason] No sigs per WebmasterWorld TOS [/edit]

3:52 pm on May 7, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 17, 2003
posts:172
votes: 0


Hi James, welcome to webmasterworld!

All bots I've seen will still spider even without the file present.

3:54 pm on May 7, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 28, 2002
posts:317
votes: 0


[webmasterworld.com...]

build a robots.txt to keep the nasties out, and let the nice ones in.

:)

3:54 pm on May 7, 2003 (gmt 0)

Senior Member from ZA 

WebmasterWorld Senior Member 10+ Year Member

joined:July 15, 2002
posts:1720
votes: 1


I have put up a file on my site called robots.txt telling all spiders that they can spider, it keeps the 404's down.

Craig

4:03 pm on May 7, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 17, 2003
posts:172
votes: 0


Do you exclude crawlers that produce few results like WiseNut or AltaVista? Or let them through hoping they might one day become relevant. On a large site, extraneous crawlers can chew lots of bandwidth.
4:05 pm on May 7, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 28, 2002
posts:317
votes: 0


If its not worth its weight in sales then it doesnt get in. Thats my motto, which is probably about to get rubbished.
4:18 pm on May 7, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
posts:2251
votes: 0


Naw, that's not rubbish. I just think you have to make sure to replace "weight in sales" with a more generalized term that can apply across the board. For me, there are no sales involved in any of my websites so I determine a crawlers worth by how many visitors I wind up getting from the crawler's website. For others it might also take into account how many of those visitors are converted into customers. :)
4:36 pm on May 7, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 6, 2003
posts:54
votes: 0


thanks to all!

i think i might create one just in case it could be helpful in the long run. as a benefit, this will decrease 'page not founds'.

4:37 pm on May 7, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 28, 2002
posts:317
votes: 0


Very valid point GaryK, formally ammended. Thanks :)
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members