homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Gold Sponsor 2015!
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

FAST crawler - very strange behaviour
Hundreds of hits for the same page in the past 4 days.

10+ Year Member

Msg#: 799 posted 1:13 am on Feb 11, 2003 (gmt 0)

Has anyone seen FAST Crawler request the same page hundreds of times a day?

Here are the stats:

UA - FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]

IP -

Number of requests for the same page
February 7, 2003 559
February 8, 2003 771
February 9, 2003 699
February 10, 2003 386
(there are still about 7 hrs left today - PST)

Has anyone seen this before?

How is FAST about feedback?

I've been a little hesitant about contacting them as they have been very good to me. Wouldn't want to change any of that - most of my content is targeted at Canadians.

Thanks for any insight.



WebmasterWorld Administrator 10+ Year Member

Msg#: 799 posted 12:43 pm on Feb 11, 2003 (gmt 0)

Hi Vlas - Welcome to the board.

Hmm, that's certainly strange crawler behaviour and I have no idea why the crawler would hit the same page over and over again.

FASTs crawler usually behaves like a good boy (girl?) and imo their feedback is really good. I wouldn't hesitate to contact them on this matter.

Good luck and let us know if you get it solved.


10+ Year Member

Msg#: 799 posted 2:18 pm on Feb 12, 2003 (gmt 0)

Do you issue session IDs at all?
These can create spider traps.


10+ Year Member

Msg#: 799 posted 6:28 pm on Feb 13, 2003 (gmt 0)

The site runs on ASP and issues session ID's. But if that would be the case I don't understand why that would affect just one page. This is the first time I've seen this; the site has been around for almost 3 years and indexed by FAST for much of that.

Big_Dug: Can you explain little more what you mean by session IDs spider traps?

BTW - The crawler quit requesting that page about 1/2 a day after posting the topic.


10+ Year Member

Msg#: 799 posted 10:00 pm on Feb 13, 2003 (gmt 0)

I've been in contact with FAST and problem was on my end. The page that got all the requests has many links pointing to it with variable info. My tracking didn't record the variable strings so it appeared to me that FAST was retrieveing identical url.

FAST feedback was excellent, next time I won't hesitate emailing them right away.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved