Welcome to WebmasterWorld Guest from 54.211.86.24

Forum Moderators: goodroi

will all robots crawl the sites without robot.txt?

robot crawl, robot.txt

   
2:50 am on Feb 1, 2004 (gmt 0)

10+ Year Member



Hi,

Two questions,

1> if don't have robot.txt file, will all the robot crawl the site correctly? Google, Inkotomei etc..

2> If no robot.txt fiel, no <meta index, follow> at pages, will all robots crawl the pages correctly?

Thanks,

John

3:08 am on Feb 1, 2004 (gmt 0)

10+ Year Member



One of the things that has pissed off content creators for years is opt-out rather than opt-in.

1)If you've got conent up, got inbound links, and not too much bad luck, the spiders will come.

2)Spidering content correctly is mostly a feature of your web site. The only thing that comes to mind is when URLs mutate continuously. If your site generates links on the fly, make sure you watch out for spiders so they don't get caught following hundreds of links that go nowhere.

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month