I'm trying to decide whether I need a robots.txt file. Does the google bot crawl pages that are in the same directory as index.html but that are not linked to index.html?
DavidAtWork
2:13 pm on Jan 29, 2004 (gmt 0)
My thought would be no it would not. As long as there is no link to that page and it is not listed as a default web page for your site. Maybe test it by putting a dummy page up and checking after your site has been crawled to see if it finds it. But I don't think it would.
regards...
DavidAtWork
2:16 pm on Jan 29, 2004 (gmt 0)
It still wouldn't hurt to add it to a robots.txt file though. Unless you don't want people knowing the file even exists. People can read the robots.txt file and see it and then look for it.
rld3
3:34 pm on Jan 29, 2004 (gmt 0)
I'm just wondering of if such pages could be considered doorway pages or something like cloaking. We've vanished from google after being on page 1 for our primary search terms and I'm looking for answers as to why...kind of like being dumped by someone out of the blue without any discussion/communication whatsoever.
DavidAtWork
5:09 pm on Jan 29, 2004 (gmt 0)
Im not sure why Google would just "dump" you, but I don't think that your non-linked page would have anything to do with it.
