Welcome to WebmasterWorld Guest from 54.159.246.164

Forum Moderators: goodroi

Message Too Old, No Replies

need robots.txt for pages that are not linked?

robots.txt

   
6:41 am on Jan 29, 2004 (gmt 0)

10+ Year Member



I'm trying to decide whether I need a robots.txt file. Does the google bot crawl pages that are in the same directory as index.html but that are not linked to index.html?
2:13 pm on Jan 29, 2004 (gmt 0)

10+ Year Member



My thought would be no it would not. As long as there is no link to that page and it is not listed as a default web page for your site. Maybe test it by putting a dummy page up and checking after your site has been crawled to see if it finds it. But I don't think it would.

regards...

2:16 pm on Jan 29, 2004 (gmt 0)

10+ Year Member



It still wouldn't hurt to add it to a robots.txt file though. Unless you don't want people knowing the file even exists. People can read the robots.txt file and see it and then look for it.
3:34 pm on Jan 29, 2004 (gmt 0)

10+ Year Member



I'm just wondering of if such pages could be considered doorway pages or something like cloaking. We've vanished from google after being on page 1 for our primary search terms and I'm looking for answers as to why...kind of like being dumped by someone out of the blue without any discussion/communication whatsoever.
5:09 pm on Jan 29, 2004 (gmt 0)

10+ Year Member



Im not sure why Google would just "dump" you, but I don't think that your non-linked page would have anything to do with it.