homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

need robots.txt for pages that are not linked?

 6:41 am on Jan 29, 2004 (gmt 0)

I'm trying to decide whether I need a robots.txt file. Does the google bot crawl pages that are in the same directory as index.html but that are not linked to index.html?



 2:13 pm on Jan 29, 2004 (gmt 0)

My thought would be no it would not. As long as there is no link to that page and it is not listed as a default web page for your site. Maybe test it by putting a dummy page up and checking after your site has been crawled to see if it finds it. But I don't think it would.



 2:16 pm on Jan 29, 2004 (gmt 0)

It still wouldn't hurt to add it to a robots.txt file though. Unless you don't want people knowing the file even exists. People can read the robots.txt file and see it and then look for it.


 3:34 pm on Jan 29, 2004 (gmt 0)

I'm just wondering of if such pages could be considered doorway pages or something like cloaking. We've vanished from google after being on page 1 for our primary search terms and I'm looking for answers as to why...kind of like being dumped by someone out of the blue without any discussion/communication whatsoever.


 5:09 pm on Jan 29, 2004 (gmt 0)

Im not sure why Google would just "dump" you, but I don't think that your non-linked page would have anything to do with it.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved