will all robots crawl the sites without robot.txt?

robot crawl, robot.txt

[johnlim]

Hi,

Two questions,

1> if don't have robot.txt file, will all the robot crawl the site correctly? Google, Inkotomei etc..

2> If no robot.txt fiel, no <meta index, follow> at pages, will all robots crawl the pages correctly?

Thanks,

John

[Slade]

One of the things that has pissed off content creators for years is opt-out rather than opt-in.

1)If you've got conent up, got inbound links, and not too much bad luck, the spiders will come.

2)Spidering content correctly is mostly a feature of your web site. The only thing that comes to mind is when URLs mutate continuously. If your site generates links on the fly, make sure you watch out for spiders so they don't get caught following hundreds of links that go nowhere.