I’ve been thinking a lot about developer support at Cisco recently, especially for the way the world works today with multiple cloud providers. This post is a re-publish of my talk from over five years ago, but the techniques and tools for listening and helping others are still true today.
At Rackspace, we watched several sources to determine if a developer is having trouble with Rackspace Cloud services, many of which are based on OpenStack. We even have a notification tool, aptly named peril, that offers an aggregation of the many sites where developers may seek help. I’ve been on the Rackspace developer experience team since the very early days when we started supporting developers at the code level a few years ago. We monitor stackoverflow.com, serverfault.com, and superuser.com for questions tagged with Rackspace, rackspace-cloud, fog, cloudfiles, jclouds, pyrax, or keystone. At Rackspace we have been supporting cross-cloud SDKs such as Apache jclouds, Node.js pkgcloud, Ruby Fog, Python Pyrax, .NET, and PHP. Let’s look at the data from these many places to find out the patterns for application development.
A developer is in peril!
We have a group email address that we monitor for developer support requests. In the last year, about 6.25% of support requests came in through email. We saw nearly half of support requests on the jclouds bug list (JIRA tracker) and a community forum at community.rackspace.com. Tracking Github issues on our supported SDKs was another 40% with the 14% remaining support requests coming from Stack Overflow, where we track certain tags on questions asked. Here’s a screenshot showing what our notifications look like in a Slack channel internally. I like how it cycles through various alert messages, “Heads up, incoming!” and not shown is “BWEEEEEP BWEEEEP BWEEEEP” which naturally makes us want to help!
True story: sometimes our slurps catch and notify before our email server. We are on it!
Documentation comments
If you’ve read my blog for a while or my book, you know I appreciate documentation that offers back-and-forth discussion in comment threads. We use Disqus comments for developer documentation at Rackspace. These comments tend to uncover three categories of requests:
- Request for help when something doesn’t work as expected
- Request for a feature that doesn’t exist
- Request for correction: pointing out typos or incorrect formatting, such as JSON examples that lose their indentation
We see about 20 comments a month on API docs spanning all our products, with about 27.5% on Cloud Files (Object Storage), about 20% Cloud Servers (Compute), and Identity coming in third at 12%. In OpenStack docs, we have a doc bug link that serves for the third type of comment — pointing out a doc bug — but not on the API reference, yet. One pattern we see that’s comparable to a review of the document is someone asking a bunch of questions at once to gain understanding.
Stack Exchange sites
Stack Exchange sites are question and answer sites with built-in features to boost motivations for answering questions posted by others. Some examples sites include Stack Overflow for developers and Server Fault for administrators.
We track these tags on Stack Overflow with our peril tool:
rackspace rackspace-cloud fog cloudfiles jclouds pyrax keystone
When people don’t get a satisfactory answer on Stack Overflow, one interesting pattern is that they come to ask.openstack.org, the OpenStack open source equivalent site.
The Stack Exchange API is a ton of fun to scrape data. I wrote some Python scripts that request data from these calls, using the tag “openstack”:
- Top Answerers: Matt Joyce, Everett Toews, and Lorin Hochstein are great at answering questions on Stack Overflow, and since Everett’s on the developer support team at Rackspace that makes sense.
- Related Tags: openstack-nova, cloud, python are the top three related tags for openstack on Stack Overflow.
- Top Tags: python, ruby, csharp/.net, php, javascript(node.js). One interesting observation is that overarching concepts like security and networking were often tagged along with the language itself.
- Frequently Asked Questions: It’s not surprising that authentication is the root of the most frequently asked questions. Networking is also particularly complex and it shows in the number of questions asked. What was interesting though is the number of questions about monitoring and metrics on the cloud consumption itself.
- Unanswered Questions: For the openstack tag, the unanswered questions had less than 100 views, compared to over 1000 views for the answered questions.
While your browser automatically decompresses the results when you enter http://api.stackexchange.com/2.2/tags/openstack/top-answerers/all_time?site=stackoverflow in your browser, I had to figure out how to have Python unzip the results and put them into JSON. Then I got stuck trying to automate putting the JSON into CSV, so I used konklone.io/json/ to convert the JSON to CSV.
Once I had the CSV files, I could use Tableau to get interesting data visualizations like a bubble grouping for related tags showing the frequency of the tags.
GitHub data
I found that GitHub issues were a great place to dive into the use cases for particular software development kits. I even discovered that our Austin-based real estate agent’s web site was developed by Rackspace Cloud php-opencloud users!
TryStack
One extremely helpful sandbox for trying OpenStack services through API use is Trystack, with over 15,000 users and growing every year. I find it very helpful to have my own sandbox besides the Rackspace Cloud to test the actual API calls and see what is returned from OpenStack. In looking at the logs, I found many queries about quotas which also explain the many questions about policies. It’s a free, community-donated cloud, and support is entirely through a Facebook group, so it’s a bit unusual but I still found interesting data around it.
Other findings
I found that many of the SDKs do not yet have full support for certain OpenStack services. For example, jclouds, the Java multi-cloud toolkit has a lot of users but doesn’t yet support the newest OpenStack services like Orchestration (heat templates), the Metering module ceilometer, the new versions of APIs for Images and Identity, and the latest storage policies implementation for Object Storage.
I also surmise that the service catalog coming through the Identity service needs stricter documentation and expectations setting. So the OpenStack API Working Group is tackling that issue by first discovering all the common patterns for the service catalog. Feel free to join the OpenStack API Working Group and review the incoming suggestions for consistency going forward or review patches going into OpenStack services that affect APIs.
Presentation
You can get the full presentation from slideshare.net, or watch me give it on YouTube.