Analytics company New Relic made its name with an application performance management tool for engineers, but recently launched Insights, a business intelligence platform that analyzes data of all kinds, from software to customers to operations.
As a flexible querying tool for a near infinite number of business data-based questions, Insights has a huge range of user types and use cases.
“Companies of all sizes use our products, and we want to be flexible enough so that if you’re small it feels natural, and doesn’t feel like a heavy enterprise product,” says New Relic director of user experience design Etan Lightstone. “But as you get larger and larger, and you use our products for more things, the UI transitions with you.”
The product is meant to be whatever any business needs it to be instantly, which for Lightstone presents a real design challenge: For whom do you create your interface, and how do you get meaningful feedback?
To combat these challenges, Lightstone uses three approaches to test and iterate in real time, without relying on manufactured or simulated conditions that may not reflect the product’s true use. They can be helpful to any business looking for authentic ways to improve customer experiences.
Lightstone says that Insights is meant to be used in real time as questions arise–questions that the design team couldn’t possibly predict for every business or use case. For example, instead of generating analytics reports that are printed out and brought to meetings, New Relic client Quizlet, the online study aids site, uses Insights to answer questions and create visualizations about customer product use during the course of development meetings, which they have used to adjust their interface to more closely reflect customer usage.
In order to test Insight’s usability in ad hoc situations, Lightstone’s team conducts what he calls “guerrilla hallway tests”–literally “grabbing other New Relic employees and bringing them in,” he says. “They may not have actually used Insights yet but they could be our target audience. We might grab someone from marketing and sit them in front of a prototype and watch them then and there, just use some of these new features we’re building to evaluate it and then iterate from there.”
The guerrilla tests are effective, says Lightstone, because “the very first thing we do whenever we build a product at New Relic is adopt it ourselves. As we build prototypes, we immediately load them with our own data and try to use it for real to solve our own business cases. When I grab someone from marketing and sit them in front of a prototype, it’s data they’re familiar with, because we’ve actually known about this in advance and try to load in what we think is the right data to be used by our own marketing team.”
To get real-time information on how customers are using New Relic products, the company uses a tool called Ethnio that injects a notification into various pages of the application to intercept sessions and ask if a user wants to be part of an immediate usability test.
“In other words, someone who is a customer of ours might be just wandering around our products, and they’ll get a pop-up that will say, ‘Hey, would you like to get a call right now and screen-share with a designer to talk about what you’re doing?'” says Lightstone. “We don’t have a preset list of things we want to see them do. We just want to know why they’re logging in that day and what brought them into the product and what problem are they solving right now.”
Before the customer is called, they fill out a short survey to answer questions including the size of their business and their role. “We want to make sure that we don’t bias the data accidentally, because people who are in smaller startups have a tendency to say yes to these forms, but people in big enterprise companies have a tendency to say no,” says Lightstone. “We just make sure that we get an even spread of user types.”
Lightstone says the approach has been extremely successful since adopting it broadly. “The key is that I don’t know what people should be doing to analyze their business,” he says. “All we can do is observe them and study them and react and incorporate what we’re learning back into the product.”
Like many technology companies, New Relic uses project tracking software to manage product development and iteration. But Lightstone says that sometimes the best, most authentic internal feedback comes from old-fashioned, spontaneous analog collaboration.
“In the two weeks before we launch a product, we create this poster called the Wall of Tweaks,” says Lightstone. “During the final push to launch there is a lot of little UI polish or usability fixes that need to be done, and one of the things that works out really well socially, I call it social engineering, it’s to print out a big poster.” The wall, says Lightstone, is essentially a bunch of screenshots where designers have mocked up the product before the needed polishes and after. “Every morning the engineers will gather around the poster and cross out the things they’d done and circle the things that they want to do today.”
It’s extremely helpful, he says, because “in most of the project process, they’re really used to just seeing the work item as a line of text in a tracking tool. This helps get them emotionally connected to some of these design issues that they need to help fix. They use a Magic Marker to say, ‘Oh, I’m doing this,’ or ‘I already did this,’ or ‘Here is the challenge for this one.’ Having that physical artifact makes things much quicker, and makes sure everyone is really holding each other accountable for working on the right problems leading to the launch. Ultimately when you’re launching a product for a date, it’s all about how you scope the project, as opposed to making it perfect. It’s impossible to make something perfect.”