Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: open
10/21 & 10/22 - Submitted pages to FAST/All The Web
10/28 - Submitted a few more to All the Web and Lycos
I got hit on 10/28 and 11/03, both by FAST-WebCrawler/3.6 but they only requested my robots.txt file and went on their merry way.
cr048r01-2.sac2.fastsearch.net - - [03/Nov/2002:06:09:57 -0500] "GET /robots.txt HTTP/1.0" 200 36 "-" "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]
cr048r01-2.sac2.fastsearch.net - - [28/Oct/2002:05:17:30 -0500] "GET /robots.txt HTTP/1.0" 200 36 "-" "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]
Does anyone have any idea why this is happening? I realllllly would like to get into Fast/All the Web/Lycos, but I'm feeling doubtful, as they can't even get past robots.txt. I just have a general robots.txt set up like this....
I only did the disallow above because I had some funky stuff happening with requests for scripts that didn't even exist. I don't know what I could be doing, but if anyone has any ideas, I'd appreciate it. I seem to be getting hit ok, by other spiders.
Anyway: Fast knows your site now. Doing some preliminary crwls before they send the regular spider for content crawling seems nothing unusual.
I would wait for some more weeks before starting to worry.
Your robots txt anyhow looks just about right.
One more unrelated question though, if I may?
On 11/01 I had the following entry...
inktomi5-bre.server.ntl.com - - [01/Nov/2002:20:44:40 -0500] "GET / HTTP/1.0" 200 8115 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
Is this really Inktomi Spider, or a fake? They grabbed a few pages, but no robots.txt. Could it possibly be that an inclusion to Inktomi's SERPS could be near?
I'm sorry to ask dumb questions, but I'm still learning when it comes to differentiating who is who and why they're there. :)
But with listings in OPD and other directories underway you have done what it takes to make it into every free spidering engine.
Not at all. What you are doing - looking at your logs, identifying the bots and figuring out what's what is the best way to get a grip on things.
Thanks for reassuring me that I am normal. :)
I hope the art of waiting is something I can learn to do more gracefully...lol
BTW, although I'm learning new things every day, I've done some editing for Zeal to bide my time, and have also applied for editing at DMOZ. Hopefully, between the two, I will be able to suffice some of my boredom :)