ses4j - 9:28 am on Nov 13, 2003 (gmt 0)

My site has been up for ~6 months now. FAST-Webcrawler came about 10 times in the first 4 months, got my robots.txt (which did NOT exclude it, for sure) and went away without getting anything else.

The whole time it had a 'blank' record for my homepage - it didn't come up in any searches, but if I put in the URL as my search term, it would come up as:

www.example.com
Description: my title from my home page in lowercase
http://www.example.com

and the 'FAST facts' were:
Last changed: 1970-01-01, 00:00 GMT
Document size: 42 bytes

If I clicked 'who links to this page' I'd get from 40-60 inbounds listed.

If you'd like to see it in person, sticky me and I'll be happy to send you my URL and gripe in private. :)

I had some very tentative hits, finally, last month, it touched a few more pages and did the same thing with those - 'blank' records with no last changed and 42b size (untrue of course).

Why won't it just get my site? Or at least the pages it pretends to have? Any ideas?

Do I care? What % of traffic do you typically get from alltheweb.com? Any?

Thanks!
Scott

[edited by: engine at 11:52 am (utc) on Nov. 14, 2003]
[edit reason] examplified & de-linked [/edit]