Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

will all robots crawl the sites without robot.txt?

robot crawl, robot.txt

2:50 am on Feb 1, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 22, 2003
votes: 0


Two questions,

1> if don't have robot.txt file, will all the robot crawl the site correctly? Google, Inkotomei etc..

2> If no robot.txt fiel, no <meta index, follow> at pages, will all robots crawl the pages correctly?



3:08 am on Feb 1, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:July 16, 2001
votes: 0

One of the things that has pissed off content creators for years is opt-out rather than opt-in.

1)If you've got conent up, got inbound links, and not too much bad luck, the spiders will come.

2)Spidering content correctly is mostly a feature of your web site. The only thing that comes to mind is when URLs mutate continuously. If your site generates links on the fly, make sure you watch out for spiders so they don't get caught following hundreds of links that go nowhere.