Welcome to WebmasterWorld Guest from 54.226.147.190

Forum Moderators: open

Message Too Old, No Replies

FAST crawler - very strange behaviour

Hundreds of hits for the same page in the past 4 days.

     

Vlas_L

1:13 am on Feb 11, 2003 (gmt 0)

10+ Year Member



Has anyone seen FAST Crawler request the same page hundreds of times a day?

Here are the stats:

UA - FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]

IP - 66.77.73.77

Number of requests for the same page
February 7, 2003 559
February 8, 2003 771
February 9, 2003 699
February 10, 2003 386
(there are still about 7 hrs left today - PST)

Has anyone seen this before?

How is FAST about feedback?

I've been a little hesitant about contacting them as they have been very good to me. Wouldn't want to change any of that - most of my content is targeted at Canadians.

Thanks for any insight.

Rumbas

12:43 pm on Feb 11, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



Hi Vlas - Welcome to the board.

Hmm, that's certainly strange crawler behaviour and I have no idea why the crawler would hit the same page over and over again.

FASTs crawler usually behaves like a good boy (girl?) and imo their feedback is really good. I wouldn't hesitate to contact them on this matter.

Good luck and let us know if you get it solved.

Big_Dug

2:18 pm on Feb 12, 2003 (gmt 0)

10+ Year Member



Do you issue session IDs at all?
These can create spider traps.

Vlas_L

6:28 pm on Feb 13, 2003 (gmt 0)

10+ Year Member



The site runs on ASP and issues session ID's. But if that would be the case I don't understand why that would affect just one page. This is the first time I've seen this; the site has been around for almost 3 years and indexed by FAST for much of that.

Big_Dug: Can you explain little more what you mean by session IDs spider traps?

BTW - The crawler quit requesting that page about 1/2 a day after posting the topic.

Vlas_L

10:00 pm on Feb 13, 2003 (gmt 0)

10+ Year Member



I've been in contact with FAST and problem was on my end. The page that got all the requests has many links pointing to it with variable info. My tracking didn't record the variable strings so it appeared to me that FAST was retrieveing identical url.

FAST feedback was excellent, next time I won't hesitate emailing them right away.

Thanks.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month