Deepcrawl, why won't he crawl my site?

Only robots.txt and index page....WHY ME?

[derekwit]

For the last 4 months deepcrawl comes and this is what he does:

2003-02-06 19:40:55 216.239.46.147 - 216.119.105.20 GET /robots.txt - 200 252 395 15 HTTP/1.0 www.exmaple.com Googlebot/2.1+(+http://www.googlebot.com/bot.html) - -
2003-02-06 19:40:55 216.239.46.147 - 216.119.105.20 GET /index.cfm - 200 14576 385 94 HTTP/1.0 www.example.com Googlebot/2.1+(+http://www.googlebot.com/bot.html) - -

Any suggestions? Last move Freshbot crawled all over the entire site. I was hoping to see something show up in the dance but only my index page which has been listed now for about 4 months. Freshbot visits me about everyday. Any suggestions. Looks like I will have to wait another month to find out.

[edited by: Woz at 9:54 pm (utc) on Feb. 6, 2003]
[edit reason] neutralized log entry [/edit]

[VictorE]

Give him some time. He has been looking at mysite off and on over the past day and a half. Just because he only stopped by for your homepage, doesn't mean he won't be back for more.

[derekwit]

I have gone back through (6) months of logs and found where deep crawl has come and gone. Six months ago I had a Flash intro and lots of Flash on my site. I had (2) pointer domains pointing to my main site. It just so happens that Gbot deepcrawl my entire site on using 1 of the pointer domains after seeing this I immedately canceled the pointer domains. My problem is I have been told that I should get rid of the Flash (which I did) and down menus because Gbot can't read them (which I did) and yet for the past 5 months Gbot has only come and read robots.txt file and my index page. I have looked over my code and meta tags and do not see anything that should scare Gbot off. If anybody here has any FREE time (Ha Ha) please take a look at my code and see if anything looks out of wack.

Thanks in advance for the help.

[hetzeld]

derekwit,

Did you run your robots.txt file through a validator?
I would double check that one if I were you.

Dan

[derekwit]

This is my robots.txt file

User-agent: *
Disallow:

[derekwit]

Is there a chance that deepcrawl will come back this month?