Forum Moderators: open
66.151.181.4 - - [12/Aug/2004:18:09:57 -0700] "GET /robots.txt HTTP/1.1"
200 - "-" "FAST Enterprise Crawler 6 used by Cobra Development
(admin@fastsearch.com)"
66.151.181.4 - - [27/Feb/2004:16:09:55 -0800] "GET /../myfolder/mypage.html
HTTP/1.1" 400 - "h**p://www.mysite.com/" "FAST Enterprise
Crawler/6 (www.fastsearch.com)"
151.138.18.36 - - [13/Sep/2004:20:02:21 -0700] "GET
Myothersite/MyFolder/Mypage.html HTTP/1.1" 200 31761 "-" "FAST Data Search
Document Retriever/4.0 04072100 2004-07-21 07:54:10 UTC"
I haven't a clue if the two crawls are related.
The second crawl did NOT begin with a reading of the robots.txt