Inktomi bot Session ID problem

Serving Inktomi with a SID

[trillianjedi]

We have a slight problem with our PostNuke site - we are being very deepcrawled by Ink at the moment, and we're serving it up session ID's so it's indexing lots of duplicate content.

Anyone know of a cure for this? What we need is a search engine session id filter for postnuke.

Many thanks,

TJ

[panic]

Oh, the woes of using premade scripts.

If you know how to program, I suggest putting an if/else condition just before the php_session is initialized. It'd start out with :

if(preg_match("/inktomi/",$HTTP_USER_AGENT)){
}
else{
(put the rest of the code here)
}

I know it's out of context, but you should try to avoid pre-made scripts like that at all costs. I certainly do :D

-panic

[trillianjedi]

Thanks for the reply, but the pnsession script is a little more complex than would allow something like that.

I think we've resolved the problem by 301-ing the URL's to one without the SID on the end.

Not before Ink had grabbed several thousand of the same URL's. Anyone know if this will cause a dup. content problem with Inktomi and the likely repurcussions?

Thanks,

TJ

[panic]

Depends if it gets flagged by Inktomi or not.

Are you getting any traffic from those pages? If so, check the referring URL's, and see if your domain is showing up multiple times for the same keyword. If so, deoptimize those pages as much as possible.

-p

[trillianjedi]

I can't really de-optimise, because the only optimisations on the site are ones that benefit users (good titles, good text - no keywords crammed in where there shouldn't be any).

If a ban is forthcoming from Ink, what are they like on reinstating?

TJ

[panic]

Once they see that the problem has been corrected, they'll usually lift the editorial easily (if you're PFI). If not, then you'll either just have to tough it out until you get de-indexed and indexed again, or do PFI for a while (or at least until you can get Inktomi editorial to lift the ban).

-panic

[trillianjedi]

OK, thanks for that. I won't do the PFI (experience has taught me never to do that, instead just wait).

We don't get many referrals from Ink, but I'm aware of the situation with Yahoo! and wouldn't want to mess things up for the future.

I'm happy to drop out of Ink for a month or two if that's the only likely slap on the wrist.

Many thanks,

TJ

[panic]

Let's just pray/hope that Y! uses Ink results before the end of 2003, in time for the Christmas time rush. :D

-panic