Scooter/3.2.SF0 using Cookies & Referrer?

Is this something new or am I just showing my age...

[Dreamquick]

Through my own laziness I have ended up with a load of notes about the "Scooter/3.2.SF0" crawler stacked up on one of my systems, now I was just about to go and delete them when I noticed something odd...

It appeared to be handling my session cookies and supplying a referral string which in the limited exampes I have appeared to be working as you would expect a referal string to.

In other words it arrived with no cookies or referrer, got redirected since it wanted the www. version, its next request came with the session cookie and a referral string stating its origin within my site.

Now as far as I was aware very few of the commercial crawlers "do" cookies, let alone referral strings. Anybody else care to confirm or deny my rantings?

My first hit appeared on Nov 25th, always from 209.73.164.50 (which belongs to the AltaVista company).

2002-11-30 01:38:16 209.73.164.50 GET /robots.txt 200 HTTP/1.1 Scooter/3.2.SF0 - -

2002-11-30 01:38:16 209.73.164.50 GET /index.asp 301 HTTP/1.1 Scooter/3.2.SF0 - -

2002-11-30 01:38:17 209.73.164.50 GET /robots.txt 200 HTTP/1.1 Scooter/3.2.SF0 [ASP_SESSIONID] [mysite.net...]

2002-11-30 01:38:20 209.73.164.50 GET /index.asp 200 HTTP/1.1 Scooter/3.2.SF0 [ASP_SESSIONID] [mysite.net...]

-Tony

[spinnercee]

I use LYNX sometimes to "crawl" and test things on my sites (like dump source and expose response headers)... it does cookies and referrer, honors redirects (ignores javascript), but most importantly, it can run "as" any UA you tell it to be. So don't be surprised if some bots use cookies, especially since some sites won't "work" unless they can perform "cookie negotiation" with a client.

[Dreamquick]

I did think that it might be a fake but the fact that it came from an altavista owned IP and exhibited crawler behaviour suggested it was a crawler rather than a fake or an employee checking something out.

It's just that none of the major crawlers I know of support cookies at the moment which took me by surprise.

- tony