Forum Moderators: open

Message Too Old, No Replies

Fast?

         

wilderness

8:06 pm on Sep 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



66.151.181.4 - - [02/Sep/2004:10:17:01 -0700] "GET /robots.txt HTTP/1.1" 200 - "-" "Sensis.com.au Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au)"

66.151.181.4 - - [12/Aug/2004:18:09:57 -0700] "GET /robots.txt HTTP/1.1"
200 - "-" "FAST Enterprise Crawler 6 used by Cobra Development
(admin@fastsearch.com)"

66.151.181.4 - - [27/Feb/2004:16:09:55 -0800] "GET /../myfolder/mypage.html
HTTP/1.1" 400 - "h**p://www.mysite.com/" "FAST Enterprise
Crawler/6 (www.fastsearch.com)"

wilderness

11:37 pm on Sep 14, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



51.138.18.35 - - [12/Sep/2004:21:35:54 -0700] "GET /robots.txt HTTP/1.1"
200 2592 "-" "FastSearch Web Crawler for Verizon SuperPages
(kevin.watters@fastsearch.com)"

151.138.18.36 - - [13/Sep/2004:20:02:21 -0700] "GET
Myothersite/MyFolder/Mypage.html HTTP/1.1" 200 31761 "-" "FAST Data Search
Document Retriever/4.0 04072100 2004-07-21 07:54:10 UTC"

I haven't a clue if the two crawls are related.
The second crawl did NOT begin with a reading of the robots.txt

wilderness

1:01 am on Sep 15, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I've apprently left of the numeral "1" of the first IP range when copying.

Should read 151.138.18.35