I’m studying different help sites and applying web analytics. I wanted to write up some of the processes, potential wins, and possible short comings of web analytics for technical communication.
When I spoke with a few Google technical writers at the STC Summit, one of them confirmed that their performance reviews include a web analytics component. This concept got me thinking about help sites I’ve worked on and how well they’d stand that test. Or rather, how well my writing and information architecture would stand up to an investigation with web analytics data. I started looking at what I’d measure. I looked at sites I’ve monitored to find examples. I collected some ideas here.
What are the goals?
I believe tech pubs groups may serve different masters or several masters. Pre-sales or marketing goals are different from support goals, and training or education goals are different still. So you would pay more attention to different measures depending on your goals for the site. This section discusses customer support and preventing costs in the company caused by a support issue being filed. The goal here is preventing support calls.
In the Web Analytics Demystified book, Eric Peterson points out a distinct difference in goals for a customer support site.
…while the other business models are driven financially by the idea that “more page views are usually better” the customer support model tends to be the opposite, the more quickly the visitor can find the information they are looking for the better.
I’d highly recommend these books for understanding how web analytics tie into business goals. The book has an entire section devoted to Customer Support sites.
For training sites, the goal would be for the person to spend time with the content, digest it, and meet training objectives because they have learned the material fully. Time spent should be higher for training sites.
Another crucial difference between a support site visitor and a training site visitor is that for the support site visitor, you want to observe the behavior of new visitors, but for the training site visitor, you want to ensure retention and repeat visits. To put it simply, customer support deals with acquisition of visitors, training deals with retention of visitors.
Another business goal is conversion – converting site visitors to paying customers. A technical manual can assist with three main goals – acquisition, retention, and conversion.
Support cost deflection – what to measure?
If the site has a question and answer section, compare the page views of the FAQ or Q&A pages to the other pages in the site – are there more views and longer time spent on pages in the FAQ area? That might be a good sign to indicate the help site is deflecting support calls.
Set up a custom segment to look for pages that have troubleshooting or particular error messages in the title or page content. Next, look at the bounce rate for that segment compared to the rest of the site. You want the bounce rate for the troubleshooting topics to be lower than the overall bounce rate. You want the trend for bounce rate for troubleshooting pages to stay the same or go lower over time. In other words, if visitors do not spend any time reading the troubleshooting information you’ve provided, what can you do to improve the content to prevent a visitor from leaving (bouncing)? This screenshot shows an example of a lower bounce rate for the troubleshooting segment of pages compared to all pages, which you want to maintain if your goal is to help users troubleshoot independently.
Pay attention to the pages that have the highest rate of exit, the page that most people leave the site after viewing. But also look at the percentage of exits. I have seen great pages on tech comm sites that have 9% exit rates but the highest number of page views. One explanation is that non-users found them through organic searches and the entire site wasn’t what they expected, not just that page.
What is the responsiveness of people to updates or comments on the site? If it offers comments, how much time elapses between a comment and a corresponding response? Do the questions get answered by a company representative or by another user? Not all web analytics packages will offer this measurement so you may have to do a sampling yourself. You can also try to get a sense of cadence, the rate of comments per day or per week or per month.
Analyzing searches will go a long way towards understanding whether user’s needs are being met by the help site. Look for searches that have zero yield – that is, the user didn’t click through on any hit or no results appeared at all. You can also look at the search results to site exits ratio.
Watch first time visitors‘ data like a hawk. New visitors may struggle at first while they learn their way around the site. As the number of first time visitors goes up, support call volume may also increase if visitors can’t find what they need or if they find the call-in number quickly. A good method for tracking real-world data along with web site data is to use a special phone number displayed only on the online help site, so that you know only those people who found the page with the special phone number can call it.
You can set up a visualization funnel from the home page to the support site to specific information to generate what Eric Peterson calls the “Information Find” conversion rate. For example, consider a flow of visits to the home page, to a product page, to a list of commonly asked questions, to a page containing a specific answer to a question. You can track travel through this series of pages, measure abandonment along the path and track a conversion as a certain amount of time spent on the final answer page.
Potential problems
Limited data will certainly make it harder to provide a convincing, statistically-significant analysis. One of the problems I foresee with applying web analytics on technical documentation sites is the small number of page views per day. I recently analyzed a help site that had about 40-50 page views per day on weekdays, pretty consistently. That was a software-as-a-service product available online. Another site I’ve watched for more than a year consistently gets 200-300 page views on weekdays and just under 100 page views on weekend days. I hear (don’t have an official citation to point to) that the 10k visitors per month is a common benchmark for starting to pay attention to web analytics. Do we get much value or accuracy from analysis if our sites don’t get to that point? I think it’s okay to monitor but to recognize your data may not have the clout you’d like it to.
Benchmarks to compare your site to would be valuable, but there aren’t categories as specific as “help site for consumer gadget” or “help site for enterprise software” yet.
Site search analytics, while most valuable to us, may be harder to enable unless you use specific tools. Site search analysis shows you what users look for, whether they find anything, and the path they take after clicking on a result link. Search analytics focused only on your tech comm or online help site require you to use a Google Custom Search Engine or the MindTouch 2010 platform which has site search analytics built into their reporting system. It appears that Adobe RoboHelp Server Analytics offers the ability to see what users search for but I don’t know the depth of analysis beyond keywords.
Connecting to the greater web analytics group at your company may be a challenge. Google Analytics is the free offering, so I expect it would have the highest uptake in tech comm in the beginning. Also, tech pubs departments aren’t usually tied in to the web content management systems so Omniture or Coremetrics (now owned by IBM) which are two other web metrics tools may not gather data on a tech comm site.
One takeaway from the Web Analytics Demystified book that makes a lot of sense is to always ask, “Is the information actionable?” In other words, when deciding which metrics to watch, make sure you can do something about the resulting metrics, whether you make changes to content or dive more deeply into the metrics. In certain environments, actionable items could be problematic if politics or tools get in the way.
Definitions to Understand
It’s important to get a handle on the basics of web analytics. I appreciate Avanish Kaushik’s blog, Occam’s Razor, for learning about web analytics and the definitions that are so important to understand. What are visitors and what are visits, and what are clicks and what are views? What’s the difference between clicks, visits, visitors, pageviews, and unique pageviews? is a Google Analytics help topic that explains these well. I also pursued and attained a Google Analytics Individual Certification which was extremely valuable for understanding definitions and the mechanics of web analytics.
Sources
I refer to a great article by Rachel Potts titled, What can web analytics do for technical communications? that she wrote for the ISTC’s Communicator magazine last year.
I downloaded and devoured two web analytics books from John Lovett and Eric T. Peterson on their site at www.webanalyticsdemystified.com. The titles are available for a free (give them an email address) download, Web Analytics Demystified and The Big Book of Key Performance Indicators.