Welcome to WebmasterWorld Guest from

Forum Moderators: martinibuster

Message Too Old, No Replies

Log-Walking Your Way to >$$$

Old Content Needs Re-Tuning; Logs Tell You How

7:02 pm on Jan 20, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 16, 2003
votes: 0

The "get rich slow" school of AdSense is about adding valuable content. Preferably, "evergreen" content that, once written, will continue to pay back rewards (however small) for years to come. But over time, the world is changing around that static content you wrote weeks or months or years ago. Different visitors are arriving, with perhaps slightly different neeeds. A page that used to rank highly for "widget repair" may have, over time, come to rank more highly for "widget reconstruction", which is a subtly different thing. If you never retune your "evergreen" content, you may be leaving some easy money on the table.

To retune content, you've got to see how the world has changed since that content was originally written, and see things through fresh eyes. One way to do this is "log walking", which forces you to look at your website like one of your visitors does, and can quickly give you lots of ideas about how to tweak and retune existing content that may only need a small change to increase its profitability. Many AdSensers pore over their logs only with high-level tools like Google Analytics, seeing their visitors only as vague statistics. Log walking will put you in your visitors' shoes, one at a time, and you'll be unable to view them as anything but individuals. It's a fresh perspective.

First, you need a recent day's worth of raw web logs, those big ol' ASCII files that record every page anyone fetches from your website, one fetch per line. They look a little like gobbledygook, but it only takes little effort to see the various parts; you can learn details of that skill elsewhere. Second, you need a web browser so you can look various things up. Beyond that, all you need is your brain and imagination.

  • Find a Google Hit The journey begins with your weblog record of someone arriving at your website from Google. This is easy to spot, since the corresponding line in your weblog will contain something like this:
    Just ignore the parts that don't make sense; Google's home address stands out, and the search terms (with plus signs instead of spaces) used to get to your website should stand out too. Somebody was searching for red widget paint.
  • Know Your Visitor Now at the beginning of that same line is an IP address, a 4-part number that looks like NNN.NNN.NNN.NNN. There are various tools on the web (ask around if you don't know them) that can give you a rough idea of what company that IP address belongs to, and where it is located in the world. Use those tools to get that information.
  • Get In the Visitor's Head You now have enough information to start thinking about this visitor as a real person. Notice that there's a timestamp on that Google hit as well, and combining that with the best-guess geographic location of the IP address and its company name will probably give you a lot of clues about your visitor. Was this someone in a school in France, during their afternoon? Maybe it's a child working in a classroom. Was this someone on the East Coast at 7pm their time? Maybe just got off work, ate some dinner, and started surfing. You may get hints about whether the visitor is young, old, at work, at home, etc. If the IP address belongs to a law firm, this could be a lawyer or a legal secretary. Suddenly, this anonymous weblog entry is turning into a real person.
  • Be The Visitor OK, now you know where/when in the world this person was when they typed "red widget paint" into Google. Put yourself in their shoes, and do the exact same thing yourself. (Beware: Google results vary with geographic location and time, so you may not see exactly what they saw -- but often it will be close or even identical.) Once you type "red widget paint" into Google, look at the resulting Search Engine Result Page (SERP).
  • SERPs: Check the competition.
    Look around the page. Why did the visitor choose your entry over the others? Why did Google like the others well enough to put them on the same page? Are there relevant keywords visible on this page that your website simply doesn't have? If so, maybe there's an opportunity to add closely related content that would make Google give you one of those lovely "double entries" on this SERP. Do you know most of your competitors on this page, or are there surprises that you might want to go check out?
  • SERPs: Check the advertisers.
    Look at the ads on the page (you may want to click to visit the complete list of advertisers). Do those folks advertise on your website? Why not? Do you have content that fits their products really well? If not, make some notes. If you're smart enough to keep a database of relevant advertisers for your site's content, make sure all these folks are in there, associated with your particular page (since Google thought their ads were relevant for that search term, and that your page was also relevant for that search term).
  • SERPs: Check your SERP entry
    What title did Google choose for your SERP entry? Do you now wish it contained some words that it doesn't? What about the page summary that Google extracted? Do you see some obvious tweaks that could improve it? Could changing the wording just a bit make it more likely that someone searching for "red widget paint" would click on you instead of one of the sites above you on this SERP?
  • Enter your website
    Time to do what this visitor did and now click on your SERP entry to go to the exact same page s/he did. You know this page, you wrote this page, you're probably bored with this page. But not today. Today, you're a legal secretary in Chicago, or a French schoolchild, and you arrived at this page because you were hot on the trail of "red widget paint". So don't just look at that familiar content, look at it through the eyes of someone who was searching for that specific thing.
  • Page: Check the relevance
    You often have a matter of seconds to either draw the visitor in or lose them. So the first question is the most important: if you just arrived at this page looking for "red widget paint", is it going to be immediately obvious that this page is relevant, or not? This can go two ways: either this really is the place you want people who are searching for "red widget paint" to go, or it isn't. If so, make sure it's relevant. If not, make sure they can find a link to take them to where you want them to go. Of course, your effort here will be tempered by how valuable terms like "red widget paint" are to you, and how much potential useful traffic that and similar search terms can generate.
  • Page: Check the advertisers
    What are the advertisers for this page? Are you missing some of the ones that were on the Google search page you just came from? Why is that? Are the ads relevant? If not, is it possible that adding a few missing relevant keywords here and there might attract more targeted ads? Is this page in an AdSense channel? Is "red widget paint" (or something close, like "widget paint") an important enough keyword that you should consider put all such relevant pages in an AdSense channel that you export to advertisers by name? In this day and age, even Fortune 500 companies are doing targeted (i.e., selecting individual websites one at a time) AdWords advertising that extend all the way down even to podunk little websites like my own. Do not discount the value of being aware of them and trying to make their job easier.
  • Page: Check the exits
    As you look at this page through the eyes of your visitor, how are they likely to exit this page? Leaving by clicking on an ad is great (if it's not a super-low paying ad), and leaving by clicking on a link to another page in your website ain't bad, but hitting the Back button to go back to Google and click on a competitor definitely is bad -- that tells Google the visitor couldn't get what they wanted, and can help lower this page's ranking for that search term. So just how many relevant exits did you give this visitor if they don't click on an ad? Is there a link to "How to buy widget paint.", for example? How about "How to select a widget color."? If you're smart enough to maintain a database of ideas for future content, you should be able to see some opportunities for adding to it here. Don't ignore the problem of navbar blindness and assume that the visitor will go hunting for a relevant link all over your page; it's easier to hit that Back button. On page, relevant links with well-chosen anchor text make your page more useful for visitors and can raise your Google rankings.
  • What happened next?
    Now that you've tried to view your page through the eyes of one particular visitor, at one particular time/place, searching for one particular thing, it's time to try to see what they did next. Go back to your weblogs, note that IP address of the request that started all this off, and scan down, looking for the next request from that same IP address (you'll want to skip over any requests for images that were on the same page you just looked at). What happened? If there are no other page requests from that IP address, do you think they clicked on an ad, or hit the Back button to go back to Google? If they went to a different page on your website, is that the route you wanted this visitor to take? How long was it before they made that decision? Go to that page and repeat all the Page: checks again.
  • Put it in context
    OK, now you've followed one visitor from Google through your site, learning as much as you can. Now you can go back to all those anonymous statistical tools you have to put this in context. Is that page rising or falling in traffic? (If your fancy stat tool doesn't offer standard variance or similar estimates of significance, you may just be fooling yourself about the answer to that question.) What are all the search terms you've been getting Google traffic for that are similar to "red widget paint"? (What? Your stat tool doesn't let you surf through "similar" search terms? Yeah, most of'em aren't really designed for the way an AdSenser works.) Are there other relevant pages that should have been linking to this one, perhaps with anchor text of "red widget paint"?

Log walking is labor-intensive and has a lot of randomness in it (just like your visitors). But it's a counter-balance to numbing statistics that AdSensers spend too much time staring at, statistics that round the behavior of real people down so much that useful information is lost. Real people come to your website to perform a specific task, and if you can't put yourself in their shoes, you probably can't do a good job of designing your website to meet their needs. Also, no matter what you designed your web pages for, Google is going to refer visitors based on what Google thinks your web pages are designed for. To whatever degree those two different understandings are out of sync (and they're rarely in complete agreement), there will be lost opportunities.

Even log walks chosen completely at random can give you useful insights, but over time you can learn to take targeted log walks to get the best return for your time. For example, AdSense Channels can help you identify pages that are getting lots of impressions, but low profitability. But log walking, where you choose only requests directed at those low-profit pages, can quickly help you identify why they are not profitable, and give you some ideas of what to do about it.

Another application is the situation of a page whose profits are increasing. You might think that's a time to leave well enough alone, but increasing profits may point to a rising in the SERPs for particular keywords -- and those may be keywords the page was not particularly designed to target. The case of a page that's moving up in the rankings may be a golden opportunity to retune, and add additional relevant content that the rising page can help elevate along with it.

If you're a content creator and your AdSense revenues have hit a plateau you just can't budge, try some log walking. It's guaranteed to make you look at old content in a new way.

7:51 pm on Jan 20, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 9, 2007
votes: 0

Just one word: awesome.

Thanks for this post, ronburk.

8:59 pm on Jan 20, 2009 (gmt 0)

Preferred Member

10+ Year Member

joined:July 28, 2000
posts: 580
votes: 0

Excellent post ronburk.

This applies to anyone who runs a website whether they run Adsense or not. Stats programs like Google analytics are great for tracking the high level figures and trends, but there is nothing better than raw logs for getting inside the head of your visitors.

7:07 am on Jan 21, 2009 (gmt 0)

New User

10+ Year Member

joined:Jan 18, 2009
votes: 0

Cool! Well-written!

Thanks for sharing!

Now, I know how to log-walk.

4:14 pm on Jan 21, 2009 (gmt 0)

Senior Member

joined:Aug 12, 2004
votes: 0

Logs are especially valuable to me for long tail stuff. If I'm ranking 130-140 for "do it yourself widget repair +senior citizens" and seeing it on a fairly consistent basis, it tells me that more than one person has gone that deep in the SERPS for information on that particular combination of keywords. It could be that they're just doing thorough research, or it could also mean that they're not finding what they need within the first 130 results. If even 2 people have gone 130 deep, how many have gone 40 and 50 deep? It's an opportunity for me to look at different areas of my site(s) and add or reword text to try to improve my standing with regard to that particular set of keywords.

I can't rank number one for "red widgets." I can, however, make a concerted effort to rank number one for "making red widgets using recycled thing-a-ma-bobs" and this is where logs become invaluable.

Log walking. I like that. Excellent post Ron.

6:20 pm on Jan 21, 2009 (gmt 0)

Junior Member

10+ Year Member

joined:June 7, 2005
votes: 0

Great information. I follow the "get rich slowly" Adsense plan; have been using HitTail to accomplish this. This will make a nice addition to my process.

There are various tools on the web (ask around if you don't know them) that can give you a rough idea of what company that IP address belongs to, and where it is located in the world.

Okay, anyone care to suggest one of these tools?

Is that page rising or falling in traffic? (If your fancy stat tool doesn't offer standard variance or similar estimates of significance, you may just be fooling yourself about the answer to that question.)

I'm curious now- whose fancy stat tool Does do this? Any suggestions here?

10:40 pm on Jan 21, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 19, 2005
votes: 0

fancy stat tool doesn't offer standard variance or similar estimates of


11:16 pm on Jan 21, 2009 (gmt 0)

Senior Member

joined:July 3, 2008
votes: 0

Analytics doesn't analyze log files.
1:42 am on Jan 22, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 17, 2004
votes: 0

2:05 am on Jan 22, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 19, 2005
votes: 0

Analytics doesn't analyze log files.

You're right, and I was unclear. I meant the suggestion to refer to a tool that can show increase/decrease trends for directories or pages.

I use Analytics data along with other tools, including log files.

10:13 am on Jan 22, 2009 (gmt 0)

Full Member

10+ Year Member

joined:Dec 2, 2008
votes: 0

Yup. This is great. Content based path analysis can provide some real insights. A certain other tool I use in conjunction with GA provides a nice grouping of user actions making the whole process much easier. Good guide those for the uninitiated well done :=)
1:59 pm on Jan 22, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:July 18, 2002
votes: 42

This is a good post. I will have to try it.

On my own site, in order to make sure that I capture ideas from users who came to my site looking for something I don't quite have on the site, I have a link at the bottom of every post that says "Didn't find the answer you were looking for? Ask one of our experts."

Chances are, if they are asking the question, not only could they not find it on my site, but they could not find it in the SERPs either. So, it is an easy to fill hole with little competition.

I then take the question, look at my log files to see which keywords apply to that topic from people who came to the site (because if one asks the question, another 10 landed on the site and did not ask) and have a writer write the content based on that topic and those keywords. Then link the original article and the new one to each other.

Not only does this add new content to the site that answers the visitor's questions, but strengthens the original page with better internal linking.

We get hundreds of questions now and almost exclusively write content based on the visitor questions now. It has been very effective for finding holes on our information, or identifying changes to the topic.

2:40 pm on Jan 22, 2009 (gmt 0)

Full Member

10+ Year Member

joined:Feb 18, 2004
votes: 0

BadA$$ post ! Thank you for sharing.
4:12 pm on Jan 22, 2009 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 12, 2004
votes: 0

Excellent job and love the 'evergreen' term :)

I recently did this with one of my sites and it had an impressive impact on the rankings. What was disturbing is there were many errors (spelling, etc) and not one visitor ever took the time to let me know (probably not wanting to offend). I swear those errors were not there when I published the content :)

May I also suggest giving Microsoft adCenter Publisher program a try. For me, it pays the same or more than Adsense on most of my sites and on the bigger ones, it pays double (no kidding here!).

The CTR is less, but the payout is more and that equates to less people leaving my site.

I still use Google on the sites that Microsoft can't match.

5:45 pm on Jan 22, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 20, 2008
votes: 0

good thing there's another thing to do with your log files than pull your hair out or nod out.

great post! love to see the innovative strategy conceptualized by infinitely networked cyborgs like me.

10:40 pm on Jan 22, 2009 (gmt 0)

Full Member

10+ Year Member

joined:Jan 2, 2005
votes: 0

Very interesting read - Great post!
9:10 am on Jan 23, 2009 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2005
posts: 107
votes: 0

You don't need to mine the logs to get a granular view of the user session. There are hosted tools (javascript-based) that allow you to see the exact user path on your website, complete with heatmaps, clickthroughs, and even mouse movements.
11:18 am on Jan 23, 2009 (gmt 0)

New User

10+ Year Member

joined:Jan 23, 2009
votes: 0

This is great. Sometimes, to trace one visitor is much more valuable than the aggregated numbers.
2:37 pm on Jan 23, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 7, 2005
votes: 0

Very good thoughts, thanks for the posting!
3:32 pm on Jan 23, 2009 (gmt 0)

Moderator from GB 

WebmasterWorld Administrator mack is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 15, 2001
votes: 93

Incredible post. Thank you.


8:50 am on Jan 25, 2009 (gmt 0)

Preferred Member from US 

10+ Year Member

joined:Nov 13, 2005
posts: 363
votes: 0

Thank you, great post.

I use my log tool almost every day, it's an invaluable resource. I know what visitors download, pages they come in on and leave, search keywords, came from, popular pages that don't pay or pay well, how long they stay on the site, which page they clicked ads. I pay for statcounter, 25k level. Well worth it.

One thing to add - I also take notes from visitor feedback and adjust the site accordingly. It may be adding images they're looking for to text-only articles, it may be adding content they are requesting, etc. With a small site it's not possible b/c of uneven feedback, but as long as you're big enough to get a few contacts a day, it's enough to have a good sample to start modifying content.

1:17 pm on Jan 26, 2009 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 2, 2005
votes: 0

Thanks, printed out
8:28 pm on Feb 2, 2009 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 7, 2007
posts: 80
votes: 0

Yeah the idea is good but I think it is practically very difficult to achieve on a daily/weekly/monthly basis and do most webmasters really can balance time for this, well yes in some cases but probably no in most cases. Ronburk's idea is good and can deliver solid results but as he himself said this is very labor-intensive. What I did on my site was automated the log walking process a bit. It goes like this:

Created a special database
Coded so that every page visit is tracked (consider this the table fields):
- IP
- Date and Time
- Referrer URL
- Target Page
- Time On Page

Now, the data is further processed by an offline app, which gives me:

xx ip accessed xx page xx times with most visits on monday.
xx page got the maximum visits during time 6:00:01 PM - 6:30:15 PM
Google.com visits: 210
Google Other Visits: 219 (Google's country specific engines)
Top Google's Country Specific Engine: google.de
Other Search Engine Visits: 156
Other Referrer Visit:76
Top 10 Referrers:
Top 10 Keywords from Google.com:
Top 10 Keywords from Google Other:
Top 10 Keywords from Non-Google Engines:

-> Clicking further on a keyword takes me to:

... Keyword Most Usage Days: 2nd Aug, 4th Aug, 14th Aug,...
... Keyword Most Usage Time Frame: 10:32:10 PM to 11:20:54 PM

Since I coded this thing, I just read through the data and I know what to do next :)


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members