This page defines quantitative data, describes the primary advantages and disadvantages of using quantitative data in UX research, describes best practices, and provides examples for quantitative analysis.
Quantitative data is any kind of data that can be quantified through counting or measuring. In contrast, qualitative data is descriptive and not easily distilled into a number. In the context of UX research, quantitative data usually is one of two types:
Quantitative data can tell us more about the who, what, when, and sometimes how of our product’s usage. As an example, quantitative data can answer the following research questions:
Quantitative data cannot tell us the “why”, because there are always aspects of a user’s situation that aren’t understood or under our control. For example:
Imagine two users. One is a student who has been following some tutorials they found online to learn how repositories work. The other user is a software engineer who uses GitLab occasionally on their freelance projects. Both users use the same features at around the same time, and neither have their profile information filled out.
From this point of view, there is no way to tell the difference in use cases through usage analytics, and over time these users will end up being misrepresented in the product. While quantitative data is powerful, it’s important to recognize the limitations of what it can provide.
If your hypotheses and research questions are similar to the examples below, then quantitative data may be one tool you consider using. Remember, this doesn’t mean that quantitative data is the only tool to use, because there are still limitations as to what analytics can tell us.
Examples of questions that point towards using quantitative data:
You can usually spot these questions, as they tend to start with “what,” “how," or “when.” Here is a walkthrough of how designers are using data to make decisions.
The typical process is as follows:
Before starting any kind of quantitative research, it is important to identify your hypotheses and questions to avoid the problem of “data snooping.” In data snooping, a lack of defined hypotheses/questions leads you to find results that seem meaningful, but are actually misleading and misrepresentative of the user population. Hypotheses/questions serve as an anchor for your overall analysis.
At GitLab, there are two main approaches to collecting data: 1) querying from usage data in Sisense see this handbook page for more information on GitLab's data sources and/or 2) using survey data.
You can use these two data sources to build a comprehensive understanding of your problem. For example, you might use usage data to first understand the user population you are looking at and then identify where you will need to go to get more data. Once you understand your user population, you might use a survey to uncover additional insights.
One way to look at data is with data visualizations. Data visualizations are graphical representations of data, commonly seen as charts or tables. Data visualizations are used to represent a trend in your data, or display a summary of your data.
There are many types of visualizations, and each one may suit a particular situation better than another. Below is an overview of common visualizations using example charts driven by SUS data, but you may also want to visit this helpful blog post for additional examples.
There are dozens of ways to lie with visualizations, even by accident. Some tips when creating your visualizations:
This section covers how to read visualizations, look for trends, and turn the results into insights. Start by looking for any kind of patterns. Some examples are:
Once you have identified patterns, summarize those patterns as best you can while considering your original hypotheses/questions.
The following is an example of using quantitative data to investigate Actionable Insights (AI). The research question was, “What are the potential factors in whether an (AI) issue is successfully completed or not?” First, the researcher pulled usage data in Sisense related to GitLab issues with the Actionable Insight label. Then, they created these charts:
Based on these charts, it is evident that there is a huge decline in issues closed after 7 months. So the finding were presented as, “If Actionable Insight issues are not addressed within 7 months of creation, they are likely to be left open.”
Additional examples of Product Designers using data for decisions
When sharing your research, state what you believe are the limitations that may have impacted your results. For example, if your data:
Be sure to include a description of the limitations in your summary. This information is just as relevant to the research as the metrics, and it provides context to the data. In some cases, you may decide it is not worth presenting the data, because of multiple problems.
Tracking key metrics that are formed through analytics can help you understand how a feature is performing in production over time.
Consider adding analytics to features as they are developed and improved. This will enable you to track how successful the design or new feature was.
There are some instances where you can use quantitative data alone. Some examples are:
While the examples above can be answered solely with quantitative data, pairing with qualitative data will provide you with a more complete analysis. That's because qualitative data helps you to understand the "why" behind what's happening.
Some of the most useful research designs come from mixed methods. Both quantitative and qualitative methods have their limitations, but you can use them together to answer complex questions and provide a more complete story. By combining methods, you are able to answer both the "why" and the "how" or "what" of the problem.
Example 1: Understand how users interact with a specific page.
Example 2: Finding areas for improvement of a new feature for future iterations.
There are many common scenarios where you might not have enough data. For example:
There are a couple of different ways you might address this. (1) Expand research by supplementing your quantitative data with qualitative research. For example:
(2) Continue forward with the current product roadmap while summarizing your insights and the possible shortcomings. For example:
It is NOT recommended to fill your analysis with data from another project. This can create problems due to sample sizes from different populations making the data look different than reality.