Here's the scoop....
10/21 & 10/22 - Submitted pages to FAST/All The Web
10/28 - Submitted a few more to All the Web and Lycos
I got hit on 10/28 and 11/03, both by FAST-WebCrawler/3.6 but they only requested my robots.txt file and went on their merry way.
cr048r01-2.sac2.fastsearch.net - - [03/Nov/2002:06:09:57 -0500] "GET /robots.txt HTTP/1.0" 200 36 "-" "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]
cr048r01-2.sac2.fastsearch.net - - [28/Oct/2002:05:17:30 -0500] "GET /robots.txt HTTP/1.0" 200 36 "-" "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]
Does anyone have any idea why this is happening? I realllllly would like to get into Fast/All the Web/Lycos, but I'm feeling doubtful, as they can't even get past robots.txt. I just have a general robots.txt set up like this....
I only did the disallow above because I had some funky stuff happening with requests for scripts that didn't even exist. I don't know what I could be doing, but if anyone has any ideas, I'd appreciate it. I seem to be getting hit ok, by other spiders.