Ideas on gathering your own keyword data - adjusting to "not provided"

[FranticFish]

Assuming the (not provided) situation isn't a nightmare that we'll all wake up from, perhaps we should pool ideas on how to gather our own keyword data?

I've been trying to move to more action-based Analytics events (clicks, enquiries, downloads, funnelling people to certain pages etc) - but without knowing which query triggered the action a major piece of the puzzle is missing.

What steps can we take to gather our own keyword data and try to compensate for what we're no longer privy to about our search traffic?

1) Develop / install a good search facility on the site and keep the logs

2) Track clicks on blog tags, or install more query-based navigation systems. Maybe we'll see a resurgence of tag clouds?

3) Get to grips with the Google WMT API (or use a third party provider who have developed their own interface) and grab all the data you can to archive in your own db

4) Consider installing search facility or suggested keywords to click into emails, social widgets, apps

[Robert Charlton]

I like the idea of monitoring engagement on the sites.

Maybe we'll see a resurgence of tag clouds?

I think that tag clouds are more likely to hurt than to help. Unless the number of tags is strictly limited, they can create too many empty pages... and concurrently too many choices for users and for Google. Empty tag pages are likely to give you Panda problems.

If you're going to mine site search, perhaps make it more prominent on your site... but this is a bet you'll be making with yourself that site search is more likely to produce fruitful results for users than well-designed navigation would. Ditto wrt tags vs navigation.

If your site is large enough, site-search with auto-complete that is customized for your site might be the best way of creating a site search that will work for users and provide useful data. (This in addition to well-designed navigation, and then monitoring engagement via conversion goals, funnels, etc).

[FranticFish]

Sorry, I wan't clear; I always 'noindex,follow' tag pages where the user/client is responsible for them and assumed that others did.

I am just interested in tracking the click to the subject page, definitely not having Google index what is on that page.