How to use AdSense w/o G getting all content?

"kimona" question from newbie

[PassePartout]

Hi,

I'm working on a dynamic site based on a database containing many occurances of a certain type of establishment. The purpose of the site is to help users find these establishments. The database has (will have) value and is costing time and money to populate. Searching and browsing of the db will be supported in ways beyond simple keyword text searching (e.g. by distance, region, etc.)

How can I use AdSense to get a little income from the site while at the same time keep the spiders from sucking down all the data? If all the db records were to end up in G's cache, much of the unique value of my site would be lost.

Also, AdSense seems to do better for static sites than dynamic sites w/r to ad appropriateness.

How to people deal with that? That is, cause appropos ads to appear based on user input (e.g. search criteria).

Sorry if these are simple questions, i'm jus starting out.

tnx,
pp

[Jenstar]

You can use robots.txt to ban Googlebot but allow the Mediapartners bot so you can get targeted ads. They are completely separate from each other, and do not share information between them.

As for how well it will target, it depends on what dynamic elements it uses, how many variables, etc. Some dynamic sites do well, while others do not.

[jomaxx]

You can use a no-cache meta tag to prevent the page from ending up in Google's cache, but that's only part of your problem. Anybody could use a spider to suck down your whole site, regardless of what any meta tag or robots.txt file says.

This is extremely common. I see personal spiders like HTTrack, Wget, Teleport and MSIECrawler in my logs literally every single day. You need to think through how you're going to prevent that from happening.

[jchampliaud]

What about having your visitors’ login before they see your content. Might not be the best for usability, but it should stop any bot from coming in.

[peewhy]

I think password contril sites are against Adsense TOS,

[jchampliaud]

I think password contril sites are against Adsense TOS,

I just had a look at the Google AdSense program policies and didn't see anything about password-controlled sites. I think as long as you let the Mediapartners in Google shouldn’t have any problems.

[peewhy]

I wonder where I got that from?

Good job you checked!

[PassePartout]

I want the site to be easy to use so we are not considering login accounts at this point.

I guess it is OK to let the MediaBot gather the info and I can use robot.txt to keep out other crawlers.

However, isn't opeying robot.txt voluntary?

I will have to learn how to keep out the others.

Thank you for the help. I can see i'll be studying these forums quite a bit.

tnx,
pp

[robho]

keep the spiders from sucking down all the data

You basically can't stop ALL robots (in the long run), just some of them, so you should be concentrating on making the way the data can be found on your site (searches etc) the main selling point, rather than the data itself.

You have to assume anything you make public on the web will be copied and cached, so make sure it is less useful without your dynamic features. Discouraging robots means few pages will be indexed and fewer visitors.

Adsense works fine on dynamically-generated pages where the content is the same each time it is displayed (e.g. widget.html?id=123). Doesn't work very well where it's just a general search response. At best you'll get general theme ads rather than ads related to the search (unless the search is often repeated and has a spiderable url).

CTR can also be low on searches unless the search is terrible. (if the search is good, the user will find what they want and click on that, rather than an ad).

You might do best on limiting the ads to the "end page" - the page where the user has found the establishment they are after. At that point, they'll be looking to do something else. Adding ads to earlier pages may be counterproductive.