Not so friendly URL + SiteMap + Google

Will AdSense be displayed?

[tata668]

This question is related to AdSense, please read until the end!

1. I've heard Search Engines don't like parameters in URL
ie:

?id=1&foo=2&bar=3

They won't index the pages containing too many parameters in the querystring.

2. To browse my items users have to do a search and for some reasons I have to put some parameters in the pages displayed after the searches.

Therefore, an item's page, on which I want to have AdSense, will be something like:

[widgets.com...]

3. To help Search Engines indexe my pages properly, I've created a sitemap from which all pages are accessible directly, with a clean URL:

[widgets.com...]

4. To prevent Search Engines to think both versions of those pages are not the same and think I have duplicate content, I disabled the indexing of the page with the parameters.
So when the URL is:

[widgets.com...]

I output:

<meta name="ROBOTS" content="NOINDEX,FOLLOW">

Doing this I guess Search engines will only indexe the pages using the "clean" URL, the one without the parameters, found from the sitemap.

5. My question related to AdSense:

If Google can only see and can only indexe

[widgets.com...]

Will an Adsense on that page be displayed when the user will do a search and found the version of the page with parameters:

[widgets.com...]

?

My fear is that the version of the page with parameters won't be recognized by Google and therefore no ad will be displayed!

I'd really like to know before publishing my site..

What do you think?
Will Google considere both versions as the same page?

Thanks a bunch in advance for any help.

[leadegroot]

An ad will be displayed but it is possible it will not be as on topic as it could be.
The ad topic will be determined from what else google knows about your site.
I suppose its possible that after a while of not being able to review the page you could stop getting ads, but it wouldn't be immediate.
Alternatives:
- try and rewrite your search page generation so it produces clean urls (and let the bots back in)
- disallow the pages in your robots.txt file (I think you can use pattern matching), rather than using page robots syntax - and allow the adsense bot in. I don't believe the adsense bot will care about duplicate content. (I dont think you can specify which bot in page-based robots commands, but I've never looked forit)

HIH
Lea

[tata668]

Thanks for that answer Lea...

I think you are right. What I'm going to do is to remove the

<meta name="ROBOTS" content="NOINDEX,FOLLOW">

on pages with the extra parameters.

If Google see those as duplicate content, I'll contact them or something.

But my searches can't produce clean URL for some technical reasons...

[leadegroot]

And remember - the duplicate 'penalty' doesn't mean thney penalise both pages, one of them would show.
The main downside is the PR is effectively split. :(

[Alioc]

tata668,

Use Mod ReWrite and avoid carbon copy pages. You can find info about that by searching for "htaccess tricks", "modrewrite, htaccess"

[tata668]

leadegroot:

I don't care that much about PR, my traffic won't come mainly from Google. I just fear to be banned and therefore unable to use Adsense!

Alioc:

I already use mod_rewrite for example to change www.widget.com/item1/ to www.widget.com?itemId=1
Problem is my searches produce some uniqueId that I have to put in the search result pages' URLs. Therefore a search returning item1 will sometimes be:
www.widget.com/item1/?uniqueId=12345
And other time:
www.widget.com/item1/?uniqueId=67890

mod_rewrite can't really help me here...

So I'm pretty sure what I'm going to do is allow Google to access (and even indexe) both search result pages with "extra" parameters and the ones with clean URLs taken from the sitemap.

I only pray I won't be banned in any way!

But my website is clean, original content and no black hat so I guess everything will be ok!

Wish me good luck, this is a big web application I'm working on for about two years and I wish I'll be able to make a buck or two from it! :-)

Thanks to everybody for your replies.

[Alioc]

May your efforts pay dude. Good luck.

[jadebox]

I think you are right. What I'm going to do is to remove the

<meta name="ROBOTS" content="NOINDEX,FOLLOW">


on pages with the extra parameters.

Since you seem to be able to control it programmatically, why don't you remove the meta tag only if the user agent is "Mediapartners-Google?" That way Google wouldn't index the page, but the Adsense bot will be able to see it.

-- Roger

[tata668]

Thanks Alioc!

jadebox:

The idea is good but I would need more informations here.. Are you telling me that the Google bot responsible to display and choose which ad to display is not linked in any way to the regular Google database of indexed content?

If the regular Google bot doesn't index one page, the adSense bot doesn't care and still browses the page where the ad will be displayed? Adsense system would have its own database?

Because if this is the case, I'm really going to use the solution you suggest!

[leadegroot]

Yes, Google has (at least) 2 bots - one for the general index and a separate one for the adsense.
The adsense bot seems to have this agent string:
Mediapartners-Google/2.1
The regular one normally has an agent string like this:
Googlebot/2.1 (+http://www.google.com/bot.html)

[tata668]

leadegroot:

Bots are differents but maybe they share some databases, some informations?

Can you confirm that even if a page is not indexed (and will never be) by the regular bot, AdSense will still be displayed on that page?..

[Alioc]

Even though it's not guaranteed that ads will be laser targeted, I see that it works. Yes.

[leadegroot]

Can't confirm anything - Google doesn't tell anyone anything :( - but I wouldn't worry about it, myself, in your circumstances.