Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

FAST crawler - very strange behaviour

Hundreds of hits for the same page in the past 4 days.

1:13 am on Feb 11, 2003 (gmt 0)

New User

10+ Year Member

joined:Jan 6, 2003
votes: 0

Has anyone seen FAST Crawler request the same page hundreds of times a day?

Here are the stats:

UA - FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]

IP -

Number of requests for the same page
February 7, 2003 559
February 8, 2003 771
February 9, 2003 699
February 10, 2003 386
(there are still about 7 hrs left today - PST)

Has anyone seen this before?

How is FAST about feedback?

I've been a little hesitant about contacting them as they have been very good to me. Wouldn't want to change any of that - most of my content is targeted at Canadians.

Thanks for any insight.

12:43 pm on Feb 11, 2003 (gmt 0)

Moderator from DK 

WebmasterWorld Administrator 10+ Year Member

joined:Oct 23, 2000
votes: 4

Hi Vlas - Welcome to the board.

Hmm, that's certainly strange crawler behaviour and I have no idea why the crawler would hit the same page over and over again.

FASTs crawler usually behaves like a good boy (girl?) and imo their feedback is really good. I wouldn't hesitate to contact them on this matter.

Good luck and let us know if you get it solved.

2:18 pm on Feb 12, 2003 (gmt 0)

New User

10+ Year Member

joined:Nov 12, 2002
votes: 0

Do you issue session IDs at all?
These can create spider traps.
6:28 pm on Feb 13, 2003 (gmt 0)

New User

10+ Year Member

joined:Jan 6, 2003
votes: 0

The site runs on ASP and issues session ID's. But if that would be the case I don't understand why that would affect just one page. This is the first time I've seen this; the site has been around for almost 3 years and indexed by FAST for much of that.

Big_Dug: Can you explain little more what you mean by session IDs spider traps?

BTW - The crawler quit requesting that page about 1/2 a day after posting the topic.

10:00 pm on Feb 13, 2003 (gmt 0)

New User

10+ Year Member

joined:Jan 6, 2003
votes: 0

I've been in contact with FAST and problem was on my end. The page that got all the requests has many links pointing to it with variable info. My tracking didn't record the variable strings so it appeared to me that FAST was retrieveing identical url.

FAST feedback was excellent, next time I won't hesitate emailing them right away.



Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members