amanS - 7:06 pm on Feb 27, 2007 (gmt 0)

In my case, I have intellectual property that I would need someone to login/register to view; some pay per view, some free.

I think something like a Meta Tag indicating what type of content the bot is indexing would solve the problem.
For example:
Meta Content = free/register/pay/subscription/etc

Then the search engines can give an option to their users to view only free or free + “non free” content.
A bot crawling this content would run across the Meta Tags and deal with it accordingly.