[Editor Note: Dave Collins, noted UK micro-ISV marketing expert, is sharing his considerable expertise on marketing, SEO, Google AdWords and more on Fridays at MyMicroISV. Thanks Dave!]
By Dave Collins
Founder, Shareware Promotions
If you’ve heard me speaking at any of the worldwide conferences, you’ll almost certainly remember at least two things about me. 1: I’m British. 2: I’m somewhat obsessive about web log analysis. And I always like to squeeze in at least one extra fact. Or two.
In this day and age, no company has any good reason for not regularly analysing their server logs. And don’t make the mistake of thinking that your server’s free online stats are enough. They’re not.
The bottom line is that without log analysis you are blind to any threats, opportunities, weaknesses and strengths of your website. A reverse SWOT of sorts!
Imagine that your business wasn’t online, but existed only as a physical concrete presence on a high street. Can you imagine not knowing how many people come into your shop each day? Not having a clue what they look at, where they go, and how long they spend before leaving?
Supermarkets realised the importance of this information years ago. They know how to setup their stores to make sure that their consumers buy every single item they want, and then a whole lot more besides.
With detailed analysis of your web server log files, you can do the same with your website.
On a typical working day, I probably look through between five and ten sets of server logs. I won’t tell you how many times I do this at the weekend. You won’t know whether to pity me or be impressed.
But by the time you get to the end of this article, I hope to have shared some of the knowledge that I’ve acquired, in the hope that you too will realise the pure gold that lies within your log files, and how to tap into it.
Before we dive in, a few important pieces of information. Let’s start with two reality checks.
Reality Check 1: If you were to run the same set of log data through five different web log analysis applications, you would get five quite different sets of reported results.
I’ve tried more web log analysis applications that you probably realise exist, and the range of different figures for the same “facts” can be staggering.
The good news is that the better ones more or less agree with each other. Which applications are the accurate ones? Bear with me. We’ll get there.
Reality Check 2: Log analysis is time consuming. No matter which application you use, it takes time. Which is why so many small (and not so small) companies put it off. Bad idea.
Another potential stumbling block is the terminology. So let’s be clear here:
A hit is a request for a file from a web server. If my main index page consists of one HTML page with five images, one person coming to view that page will generate six hits. 1 HTML + 5 images.
Visitors – you can probably work out for yourself. But some log analysis software tries to recognise unique/individual visitors. For example, my RSS reader may hit your feed ten times a day. Some log apps will count that as ten visitors, some will count it as one.
IP Addresses are also important for most log analysis applications, but may be misleading. Think of people sharing IP addresses on a network, dialup accounts, non/fixed IP addresses and so on.
Aside from terminology, there are also three other important points to consider.
Point 1: If a website (for example a software site) pulls in images or data from your own website, this may show up in your logs as a visit. So if Dave’s Cool Site features your app on their front page, and displays a screenshot that is being pulled from your server, Dave’s Cool Site’s 10,000 visitors may look like your visitors in your logs, even though they never visited your site.
Point 2: DOS attacks, spam and hacking attempts sometimes produce all sorts of strange entries in your web logs.
Point 3: Download managers can make a real mess of your download counts or hits to your file. If a download manager creates ten simultaneous connections to your EXE file, this will probably show up as ten hits to the file in your logs. The solution? Divide the bandwidth to each file by the size of the file. That will give you a more accurate figure.
One more thing. I warned you I try to squeeze more information in. Always remember that the period of time you are looking at is important. As a general rule, don’t make decisions on information gleaned from anything less than a full week or two of data, and it’s often a good idea to work in units of seven days. It’s also important to take into account any regional, national or international holidays, festivals, religious events, major sporting events and seasonal trends. Always take a step back from your data.
Enough of the provisos. Let’s get into what I look for in our client’s logs.
Every log analysis app has some sort of overview or summary section. This is a useful snapshot of what’s going on, but in general isn’t worth spending too much time on. The total number of visitors isn’t as much use as how many visitors came on each day, the average page views is utterly meaningless, and the time spent? Don’t even bother reading it.
So what do I look for?
Referrals are obviously of great interest. You want to know who is sending you traffic, whether there are unexpected or unknown names in there, and of course whether there have been any changes and/or trends emerging. Falls or drops in referrals are just as important as rises.
Sites sending two or three visitors a month aren’t worth a second glance, but any generating a significant number of visitors should catch your attention.
You may, for example, find that your company has been discussed on a forum. You may also notice a crack site generating high levels of traffic. The choice is yours as to how to respond to these situations. But without analysis you’d never even know.
Interestingly, we once spotted one of our competitors who had copied and pasted the text from our own site right on to their front page. How did we know? They also copied the link to one of our pages. Don’t you wish you had competitors like that?
Aside from general referrers, many log analysis apps also offer specific information related to which of the search engines are sending you traffic. It’s important not to make the mistake of thinking that only Google matters. If they have one of their renowned/feared updates and pull you from their index for no apparent reason, you’ll wish you’d covered other options. If your website is getting no traffic at all from MSN and Yahoo, then you might want to look into it.
You may also like to drill down a little, and see which of the pages on your website perform well on the engines. Why? Because if or when you make changes to your site, you want to be careful not to delete or drastically alter any of your high performing pages. Again; no analysis means you would never know.
Entrance pages are also extremely important, and surprisingly often overlooked. Most companies will find that the most popular entry page is their home page, but this isn’t always the case. If people are arriving elsewhere, it’s important to make sure that they’re seeing all the relevant information.
Equally important is seeing what they do next. Which pages they go to and which links they click on. Are they behaving as you want them to? By moving links around, playing with different wording, graphics and calls to action, you may be amazed to see how much control you actually have over your visitors. Put this to good use.
Exit pages are equally important, and just as often overlooked. The reality for many companies is that the most popular entry page will also be the most popular exit page. But not always.
And there may be other pages on your website that instead of feeding your visitors with the information they seek, effectively act as black holes, swallowing them up forever. If there are such pages, and they receive a fair number of visitors, then you need to understand why and seal up these holes.
Also bear in mind that different log analysis apps handle the issue of exits differently. Some may report a click on a download as an exit, some may count a click on a mailto link as leaving the site, and if your files are hosted by a third party, then all of them will look at these clicks as a visitor leaving.
The paths that your site visitors take are also of great importance. With a little digging and detective work, you may realise that certain sections of your website are more or less hidden, or that your visitors are having to work hard to find what they are looking for. Hint: They won’t. They’ll give up.
There are also what I call the “basic stats”: which countries your visitors come from, what operating systems they use, what screen resolutions and so on. But bear in mind that as interesting as it may appear, this is often highly inaccurate. Country information is notoriously imprecise, the browser and resolution may be on their work/school/library machines and so on.
Errors, however, are far more important (and accurate), and again, to my constant surprise, these too are often more or less ignored.
If a referrer of a significant number of visitors has the wrong URL, then believe me you’ll want to know about it. A company that we worked with sent out a press release, and got a great write-up in a newsletter. Unfortunately the editor mistyped the URL, and anyone clicking on the link only got a 404 error. How’s that for a missed opportunity?
Note also that incorrect URLs may be from external or internal links. And that in general, there will be a variety of “odd” errors, mostly generated by attempted hacks and exploits. Unless there is a massive number of them, you can probably ignore most of these too.
And if issues such as partial content and bad requests are common, you’ll want to get to the bottom of it as quickly as possible. Find out why and solve the problem.
Finally, how to make your life easier. I belong to a certain association of shareware professionals (you know the one), and every now and then someone shows up who still reads their log files with Excel. I always assume that this person has far too much time on their hands and views their logs as a curiosity, and not a valuable resource.
The fact is that there are numerous good applications out there, two of which I can personally recommend.
The first is ClickTracks at www.clicktracks.com. In a nutshell, it’s the single most powerful log analysis application that I have ever worked with. Nothing else even comes close. A few weeks ago there would have been a “but”. A big “but”; the price. But they’ve just released a free version of their software.
I’m not going to get into it now, but trust me, you want to take a look for yourself. This is a major time saver and a powerful tool.
The other app I recommend is Web Log Storming – www.datalandsoftware.com. It has a very unique approach that allows you to really dig deep into your logs in real time. I can’t recommend this software enough, and at $129 I rate it as a steal.
The bottom line is that too many companies neglect their logs on an ongoing basis. If you’re one of them, I can’t stress enough just how accurately you may be shooting yourself in the foot. You may be oblivious to mistakes, be losing sales, be throwing away prospects and leads, and be wasting money on advertising in the process.
It usually boils down to time. There isn’t enough of it. I understand. But this is just as important as anything else you do in your work hours, and probably a whole lot more important than some of the other things you do.
Make the time. At the very least schedule two hours on the first Tuesday of every month. Stop losing sales.
Dave Collins is the CEO of SharewarePromotions http://www.sharewarepromotions.com, a well established UK-based software marketing company. Dave specialises in Google AdWords, Log Analysis, Online Marketing and Delegation.
Technorati Tags: Dave Collins, micro-ISVs, AdWords, Server Logs, Marketing, SEO

1 Comment

  1. Thanks for this, it’s a most timely post. I’m just deciding which log analysis software to use (finally found some time after deciding to ditch Google Analytics following your presentation at ESWC!)
    Do you have a URL for the free version of clicktracks? I can’t find anything about it on their website and the normal price is a bit above my budget.

Write A Comment