My site has been up for ~6 months now. FAST-Webcrawler came about 10 times in the first 4 months, got my robots.txt (which did NOT exclude it, for sure) and went away without getting anything else.
The whole time it had a 'blank' record for my homepage - it didn't come up in any searches, but if I put in the URL as my search term, it would come up as:
Description: my title from my home page in lowercase
and the 'FAST facts' were:
Last changed: 1970-01-01, 00:00 GMT
Document size: 42 bytes
If I clicked 'who links to this page' I'd get from 40-60 inbounds listed.
If you'd like to see it in person, sticky me and I'll be happy to send you my URL and gripe in private. :)
I had some very tentative hits, finally, last month, it touched a few more pages and did the same thing with those - 'blank' records with no last changed and 42b size (untrue of course).
Why won't it just get my site? Or at least the pages it pretends to have? Any ideas?
Do I care? What % of traffic do you typically get from alltheweb.com? Any?
[edited by: engine at 11:52 am (utc) on Nov. 14, 2003]
[edit reason] examplified & de-linked [/edit]