Page is a not externally linkable
luma - 9:21 pm on Aug 14, 2002 (gmt 0)
It fetched two pages but no robots.txt. The last time "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http*//fast.no/support/crawler.asp)" fetched robots.txt was on July, 23rd. It didn't fetch any pages. So, is the Fast-Firstpage crawler the regular guy or did they finally read my e-mail and send some special bot? What do you think, (when) will those pages make it in the index?
It's been some time, and yesterday I found the following in my logfile: access.log.33.2:66.77.73.254 - - [13/Aug/2002:15:04:43 +0200]
"GET /widgets/blue.html HTTP/1.0" 200 21053 www.domain.com "-"
"FAST-WebCrawler/3.6/FirstPage (crawler @fast.no;
http*//fast.no/support.php?c=faqs/crawler)" "-"