Getting Google hits query string

[Patrick Taylor]

To track hits, on each of my site's pages I have this:

// Get user agent
$agent = $HTTP_USER_AGENT;
// Get referrer
$refer = $_SERVER['HTTP_REFERER'];
// Get query string
$query_string = $HTTP_SERVER_VARS["QUERY_STRING"];

The agent and referrer go into my database fine but I get no records for where I'm trying to get the query string the visitor typed into Google etc. I have also tried:

$query_string = $_SERVER['QUERY_STRING'];

Thats doesn't work either. Obviously I'm doing something wrong, and would appreciate a pointer. Many thanks.

[Salsa]

$_SERVER['QUERY_STRING'] is only going to hold whatever query string may be in Googles' link to your site. In the current case, apparently none.

The keywords that you are looking for are in $_SERVER['HTTP_REFERER']. Look at the query string in that. For google, the keywords value that a visitor searched for will be right after the '?q=' or '&q='. You'll have to call a preg_match or strpos, etc., to pull them out.

I hope this helps.

[Patrick Taylor]

You'll have to call a preg_match or strpos, etc., to pull them out.

I see. Yes it helps, but I think I will leave it as I don't know preg_match etc - seems quite complex, having looked at the various query strings involved. I thought I could get the query string just with $HTTP_SERVER_VARS["QUERY_STRING"]; but obviously not.

Thanks.

Patrick

[Salsa]

It's not that terribly hard. You could still do it on a single line like:

if (preg_match('!q=(.*)&!', $_SERVER['HTTP_REFERER'], $matches)) 
 $keywords = urldecode($matches[1]);

The way it works is that the regex in the first argument looks at the $_SERVER['HTTP_REFERER'] string. In the regex, the (.*) represents all of the characters found between the first 'q=' and the next '&'--which should contain any keywords. The '!' are just delimiters for the regex. Then, the urldecode functions converts + and %nn back to spaces and ASCII characters.

Many other search engines also use 'q=', but it's not universal. Just the above code should at least get those that do, and you can refine it to engine specific queries later on if it seems worthwhile.

[Patrick Taylor]

Well, thanks for your helpful encouragement. It's something I've been meaning to learn for a while, so I might just start with some experimentation using your example.

Many thanks.

Patrick

[Patrick Taylor]

It's starting to work, using your example, but the end delimiter doesn't seem to delimit. I'm finding the search terms being recorded in my database like this:

keyword1 keyword2&hl=en&lr=&start=10

Otherwise, I think I can make a start with Google, Yahoo!, and MSN, and build on that. However, Yahoo! uses p= instead of Google's q=

I presume I can build "or" into my regex like so:

'!q¦p=(.*)&!'

Also, in usage of the above, does it matter if MSN (and Google sometimes) omit the ampersand at the end of the search terms? It's not easy to test this out other than "live" on the site.

[coopster]

You may also want to have a look at parse_url() [php.net] and parse_str() [php.net]

[Salsa]

Sorry about that, Patrick,

This will work:

if (preg_match('!q=([^&.]*)!', $_SERVER['HTTP_REFERER'], $matches)) 
 $keywords = urldecode($matches[1]);

Before, what the (.*)& was doing was to match every character up to the last & in the string. By changing it to ([^&.]*), it will match every character until the next &.

Regarding managing multiple referrers: For greatest precision, I'd test for the referrer before trying to extract the keywords. That way you can reduce the guessing of what else might be in various referrers' query strings, like pdq=fast...

If you did want to take the shortcut you mentioned, this syntax would work:

preg_match('![pq]=([^&.]*)!',...

[Salsa]

Actually, after thinking about it another moment, you could mitigate the pdq=fast scenario with:

preg_match('![?&][pq]=([^&.]*)!',...

And, hmm. I'm learning here, too. Going back to the original part of my code that didn't work properly: (.*)& Instead of fixing it with: ([^&.]*), a more economical way is: (.*?)&

By default, (.*)& is "greedy" and tries to match as much of the string as possible. By adding the? after the *, it reduces the "greed" and matches the next & in the string instead of the last.

[Patrick Taylor]

Thankyou both. I also have my books out here, and will report back on progress.

Patrick

[Patrick Taylor]

With Salsa's example I'm now able to record search terms for a few search engines - Google, MSN, Yahoo! mainly. However, other search engines would require several different patterns to be developed, plus a means to identify the search engine in order to be able to deploy the appropriate pattern. A useful introduction to preg_match, but I looked coopster's suggestion to see if I can simplify the approach.

Here is some experimental code:

// Test referrer
$refer = "http://aolsearch.aol.com/aol/search?query=dog+cat+mouse+pig+sheep&invocationType=spelling"; // Example referrer

$ptn = '![?&]query=([^&.]*)!'; // Salsa's pattern

// Get query string from referrer
if (preg_match($ptn, $refer, $matches)) {
$qstring = urldecode($matches[1]);
echo '' . $qstring . ''; // This works, but only for AOL
} else {
echo 'No match.<br />';
}

echo '<br />';

// From [uk2.php.net...]
$origin="http://aolsearch.aol.com/aol/search?query=dog+cat+mouse+pig+sheep&invocationType=spelling"; // Example referrer
$tab=parse_url($origin);
$query=$tab["query"];
$variables=explode("&",$query);
for ($i=0;$i<=count($variables);$i++){
$tab=explode("=",$variables[$i]);
$$tab[0]=$tab[1];
}

// Print what $tab is
echo '$tab = ' . $tab . '<br />'; // Just prints the word "Array"

// Loop through $variables array
foreach ($variables as $val) {
echo '$variables = ' . $val . '<br />'; // Prints two lines - not sure why
}

// Loop through $tab array
foreach ($tab as $vals) {
echo 'Array = ' . $vals . '<br />'; // Prints nothing
}

The output is:

dog cat mouse pig sheep
$tab = Array
$variables = query=dog+cat+mouse+pig+sheep
$variables = invocationType=spelling
Array =

The part I'm looking at is the first $variables line, where the search terms are shown, but I think my code is a bit of a hotch potch. I'm not sure which of all those variables is the one that contains the search terms, not how to output it in one line without query= in front of it. And why won't the Array print anything?

Patrick

[Patrick Taylor]

My code is now:

// Get referrer
if ($_SERVER['HTTP_REFERER']); { // Check for referrer
$refer = $_SERVER['HTTP_REFERER'];

$referrer_url = $refer;

if (strstr($referrer_url, "google")) {
$ptn = '![?&]q=([^&.]*)!';
} else if (strstr($referrer_url, "yahoo")) {
$ptn = '![?&]p=([^&.]*)!';
} else if (strstr($referrer_url, "msn")) {
$ptn = '![?&]q=([^&.]*)!';
} else if (strstr($referrer_url, "aol")) {
$ptn = '!query=([^&.]*)!';
}

// Get query string
if (preg_match($ptn, $referrer_url, $matches)) {
$qstring = urldecode($matches[1]);
}
}

It seems to be working okay, though I will have to add extra patterns for other search engines.