Diary studies

What are diary studies and when to use them

A diary study is a research methodology used to obtain feedback from participants over a period of time (days, weeks, or months). In these studies, participants are asked to document information on their own when they perform a specific action or the participant has reached a specific period in the research.

The purpose of doing diary studies

Diary studies can be very informative since we, as researchers, cannot observe everything a participant does within a given timeframe. A diary study helps by having participants keep track of information for us. The data from diary studies help researchers and teams learn about specific patterns or changes that emerge over time.

When to conduct diary studies

Diary studies are useful when there is interest in participants beyond a single interaction or touchpoint. This kind of research can examine participant feedback about a product (GitLab or other application), behavior (DevOps tool usage at work), or activity (submitting a merge request or searching for content online). Some examples of participant data that is common to collect in diary studies includes:

Documenting what participants think when completing tasks
Documenting the context in which participants complete tasks
Documenting what participants do within a given timeframe
Documenting pain points experienced while using the product over a month
Documenting how participants learn using a new product within the first few weeks
Documenting the number of times a participant performs an action or task (Note: Other research methods such as monitoring customer analytics data are more accurate in assessing the frequency of an action)

How diary studies are different from other types of research

Diary studies are unique compared to surveys, interviews, or focus groups because they gather data from the same group of participants, multiple times. This method overlaps more with longitudinal studies in situations where one or more quantitative measures (for example: confidence or effectiveness ratings) are collected at distinct points in time. The main distinction between the two methods is that diary studies do not require using quantitative metrics. Many diary studies can involve just collecting qualitative data (open ended feedback or self-reported observations).

How diary studies differ from longitudinal studies

Diary studies require many of the same steps outlined for longitudinal studies. One additional step is to determine when the participant will record entries:

On specific time intervals (for example: record once a day, two times a week, or every other week)
On a set schedule (for example: record only at day 10, day 20, and day 30)
While performing an action (for example: record as you interact with a feature in GitLab or use GitLab in a specific way)

When planning out the appropriate timeframes or touchpoints for the research, take into account the overall research questions.

Example: If the research question is meant to understand how participants learn GitLab as a first time user, it would be better to have participants record entries once every few days to reflect on their experience at that point in time.

Other factors that can impact diary study research planning include the types of participants recruited, deadline for the research, and number of research questions.

How to set up a diary study

When setting up a diary study, it is important to think through all of the necessary parts participants will fill out within their diaries. Researchers should create a diary document that is easy to view, access, and edit. At GitLab, it is common to use collaborative digital files such as Google Docs or Google Sheets, so the researcher can share them with a participant and monitor their progress throughout the study.

When using Google Docs or Sheets, make sure to follow these steps and guidelines to ensure data privacy within your study:

Create a new Google Doc or Google Sheet
Add your instructions, tasks, and questions for the participant
1. Make sure to provide a clear space or location within the file for participants to add their comments and feedback, so participants are not confused about what to do
In the initial meeting with a participant, go through the diary document in detail and answer any questions they have
Once the document is ready to share with a participant, click the share button at the top right hand corner
Keep general access defaulted to restricted
In the add people and groups field, add the email address of the participant
Give them editor access and type in a short note (if needed)
Select send

Depending on the number of expected diary entries and touchpoints, it is advised to open participants’ diary documents often to monitor their current progress. If participants have not filled out specific entries on time, you can leave a comment and @ mention them using their email address to send a reminder to complete the task(s).

Tips for conducting diary studies

Diary studies require providing a large amount of participant feedback, so it is crucial to find participants who are detailed in their diary entries and participants who are motivated to continue for the entire study.
- Include an open ended question in your screener (for example: describe the ways you currently look for content on GitLab) or set up an initial interview with them to determine how verbose they are in their responses before inviting them to participate in the study.
- Share study expectations with participants upfront, so they know what to expect from start to finish. By stating the overall research timeline to participants (number of touchpoints, number of expected diary entries), they will get a full understanding of what is required of them. This helps get buy-in from participants, so they are more motivated to see the study through until the end.
Monitor and analyze participant diary entries as they are logged. This makes it easier to follow up with participants on any entries that are either unclear, lack sufficient detail, or do not correctly follow the study instructions.
Participants can often forget to record their diary entries as they go through the study, so periodically send them reminders to complete the study tasks.
Figure out the appropriate amount of touchpoints with participants in the study. In general, it’s recommended to limit the number of total touchpoints to four at the most. If participants are starting to give the same kinds of responses after several diary entries, you may want to consider adjusting the number of overall touchpoints to avoid participants quitting the study early.
During the study, set up short 10-15 minute interviews to speak with participants about their entries. Having a set time to discuss entries with the researcher makes participants feel more accountable. As a result, participants will usually put extra effort into writing out their diary entries.
Attach small incentive amounts to each diary entry (see touchpoints 2 & 3 in the table below), so participants are more likely to stay in the study. Saving the largest incentive amount for the last diary entry can help the researcher meet desired sample sizes for the study and can avoid the need to recruit additional participants over time.

Touchpoint	Time scheduled	Gratuity amount
1	45 minutes	$30 USD
2	15 minutes	$20 USD
3	15 minutes	$20 USD
4	60 minutes	$200 USD

Examples of diary studies

There are a few recent examples of diary studies completed by our GitLab team members:

Additional resources on diary studies

These resources provide a more in-depth overview on how to conduct diary studies:

Last modified November 15, 2023: Fix markdown and image issues in UX (bed95a10)

View page source - Edit this page - please contribute.