FAST crawler - very strange behaviour

Hundreds of hits for the same page in the past 4 days.

[Vlas_L]

Has anyone seen FAST Crawler request the same page hundreds of times a day?

Here are the stats:

UA - FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]

IP - 66.77.73.77

Number of requests for the same page
February 7, 2003 559
February 8, 2003 771
February 9, 2003 699
February 10, 2003 386
(there are still about 7 hrs left today - PST)

Has anyone seen this before?

How is FAST about feedback?

I've been a little hesitant about contacting them as they have been very good to me. Wouldn't want to change any of that - most of my content is targeted at Canadians.

Thanks for any insight.

[Rumbas]

Hi Vlas - Welcome to the board.

Hmm, that's certainly strange crawler behaviour and I have no idea why the crawler would hit the same page over and over again.

FASTs crawler usually behaves like a good boy (girl?) and imo their feedback is really good. I wouldn't hesitate to contact them on this matter.

Good luck and let us know if you get it solved.

[Big_Dug]

Do you issue session IDs at all?
These can create spider traps.

[Vlas_L]

The site runs on ASP and issues session ID's. But if that would be the case I don't understand why that would affect just one page. This is the first time I've seen this; the site has been around for almost 3 years and indexed by FAST for much of that.

Big_Dug: Can you explain little more what you mean by session IDs spider traps?

BTW - The crawler quit requesting that page about 1/2 a day after posting the topic.

[Vlas_L]

I've been in contact with FAST and problem was on my end. The page that got all the requests has many links pointing to it with variable info. My tracking didn't record the variable strings so it appeared to me that FAST was retrieveing identical url.

FAST feedback was excellent, next time I won't hesitate emailing them right away.

Thanks.