CJ Spider

?

[DavidT]

207.71.241.81 - - [01/Apr/2003:20:13:40 -0800] "GET / HTTP/1.1" 200 6805 "-" "CJ Spider/"

Can't find anything out about this one. Anyone else know?

[msgraph]

Noticed this spider last night too.

Do you have CJ links on your site(s)?

I wonder if it has to do with the new stuff Commission Junction is supposed to be rolling out soon.

[DavidT]

I'm a member of CJ, just joined a new program there the day before the spider came but as yet none of their links on the site.

[korkus2000]

I have basic cj links. I got the spider last night also. I think they are using it to check invalid links.

[EliteWeb]

Yah CJ Bot is a robot to check for the links on the sites. You know what they said about charging the people who dont make x amount per pay period maybe they will see if the url's in the users profiles apear on those pages and use it for reference or a way to disable accounts when it doesnt find the links.

[carfac]

Well, I just got hit by this bot- and I can tell you, it will NOT be back on MY sites!

Looks like it was hitting about ten pages per second. VERY bad!

Plus, it is STUPID. I have URL's that are upper and lower case- all the requests were lower case.

I do not know for sure that it follows robots.txt, but I did not catch it cheating. Hit me 120 times.

The URL checks out to a Santa Barbara location, which makes sense for Commission Junction.

Here is a hit:

207.71.241.81 - - [08/Apr/2003:03:21:24 -0600] "GET /a/lower/case/path/that/is/wrong HTTP/1.1" 404 11948 "-" "CJ Spider/"

dave

[nativenewyorker]

carfac,

207.71.241.81 is definitely a CJ IP address. I have seen it in previous log entries when CJ support had checked something on my site. If you block that IP address, you will also block all human visitors from CJ. It is probably best to deny it by agent instead.

And yes, I got hit by this one for the first time today.

Ted

[carfac]

nativenewyorker:

Thanks... I will do that!

dave

[nativenewyorker]

CJ is now using a different IP for CJ Spider registered under a Cable & Wireless block.

216.34.209.23

Ted

[nativenewyorker]

CJ responded to an inquiry about their CJ Spider. They described it as "an automated network policeman". It is supposed to look for content to be reviewed by the compliance department.

The dilemma is will banning the spider trigger an automatic review of a website or just eliminate the unwanted bandwidth usage?

Ted

[DavidT]

No matter what I put in my htaccess file to block this spider it just won't work.
Using mod_rewrite what would be the correct syntax to block 'CJ Spider/'?

[wilderness]

SetEnvIf User-Agent ^CJ keep_out (or whatever you use instead of keep_out)

[jdMorgan]

DavidT,

>> Using mod_rewrite what would be the correct syntax to block 'CJ Spider/'?

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^CJ\ Spider/
RewriteRule .* - [F]

Ref: Introduction to mod_rewrite [webmasterworld.com]

HTH,
Jim

[nativenewyorker]

The U/A has changed. It is now CJNetworkQuality; [cj.com...]

Note that their documentation on the above page indicates that their spider

will visit, on a daily basis, all registered sites and all pages that have generated traffic within the past 30 days

Ted

[DavidT]

It went for robots.txt today, first time I think, so it may obey this.

[carfac]

will visit, on a daily basis, all registered sites and all pages that have generated traffic within the past 30 days

That is a bit of an over statement, don't you think? They might TRY, but if they try to visit EVERY page on my site EVERY day, they'd have to pull in 150-200,000 pages a DAY (as a guess). And if they try THAT...

dave