vBulletin Spidering Question

Need to find a way to let spiders access members area

[christopher w]

Hi -

I'm not sure if this is possible but I was told it was so hopefully someone here can shed some light on this for me.

I have a client who is using vBulletin for their forum. They have it closed off (other than the different thread titles and links) to everyone but members. They have no interest in opening up the forums without registering but I want to get the spiders in there. So is there a way to allow the spiders to crawl and index the entire forum while still requiring a login?

Thanks in advance for your help!

[robotsdobetter]

You could do this with cloaking. You could do this by only delivering the member's area to spider IPs or User Agent. But you could get kicked out of the search engines for this.

[christopher w]

Yeah - cloaking is not an option - it is a very strong site with excellent rankings already. I was just wondering if there was another way to do it...

Anyone else?

[rogerd]

The same logic that lets members see the content but not others could be allowed to let Googlebot or other spiders see it.

Doing so, though, would certainly be irritating to searchers and might draw the ire of Google et al if someone complained. On the other hand, I encounter news sites that seem to be well indexed but won't let me view without registration.

Be aware, of course, that some members only info will be exposed in snippets, not to mention cached copies of the pages if caching isn't prohibited.

[arrowman]

If cloaking is not an option, and spiders need to see the content, then by definition the public needs to see the content. Because if they wouldn't, it would be called cloaking :-)

Why don't you restrict only posting to members?