I don't know if anyone cares...

[littleman]

...but NL is cloak sniffing.
HTTP_USER_AGENT = libwww-perl/5.50
REMOTE_ADDR = 205.181.75.60
Name: tantalum.northernlight.com

[toolman]

Oh No!
Oh My Gosh!
I've got to go....

[littleman]

You know, at one time I had 60+ thousand pages in NL, and it was worth maybe 50 refs a day.

[toolman]

>>>>60+ thousand pages in NL

So that means you had about 1/4 of their database and 1/4 of their daily traffic.

But why, oh why did you ever give it all up LM?

[awoyo]

Does this mean Fast is doing the same?

fpcr01.bos2.fastsearch.net (209.202.148.100)
This visitor used libwww-perl/5.53 FP/2.1.

[littleman]

Could very well be.

[idiotgirl]

So, does this mean that NL and Fast is looking to see if the pages are cloaked, correct? Which would include possibly parsing for SSI or embedded exec commands?

Idiotgirl

[awoyo]

So, does this mean that NL and Fast is looking to see if the
pages are cloaked, correct? Which would include possibly
parsing for SSI or embedded exec commands?

I wouldn't think so. So many sites use SSI or embedded exec commands in dynamic situations that restricting their index like this would exclude perhaps thousands of sites that are not cloaking. They may be checking file sizes of pages that may be cloaked by User Agent, it may be a human looking at "suspect" pages, or any number of things.

[WebGuerrilla]

I don't know of any way a spider could possibly detect SSI. The command is executed server side before the page is ever delivered. Neither of the IP's are new, so the best they could do with it is catch UserAgent stuff.

Did either of you notice whether these bots requested a robots file?