homepage Welcome to WebmasterWorld Guest from 54.205.188.59
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
will all robots crawl the sites without robot.txt?
robot crawl, robot.txt
johnlim




msg:1529563
 2:50 am on Feb 1, 2004 (gmt 0)

Hi,

Two questions,

1> if don't have robot.txt file, will all the robot crawl the site correctly? Google, Inkotomei etc..

2> If no robot.txt fiel, no <meta index, follow> at pages, will all robots crawl the pages correctly?

Thanks,

John

 

Slade




msg:1529564
 3:08 am on Feb 1, 2004 (gmt 0)

One of the things that has pissed off content creators for years is opt-out rather than opt-in.

1)If you've got conent up, got inbound links, and not too much bad luck, the spiders will come.

2)Spidering content correctly is mostly a feature of your web site. The only thing that comes to mind is when URLs mutate continuously. If your site generates links on the fly, make sure you watch out for spiders so they don't get caught following hundreds of links that go nowhere.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved