Every product manager appreciates shorter cycle times. One way to reduce cycle times is to learn from others, so five of our engineering leads share the greatest challenges their teams have experienced and offer the strategies they developed to speed up iteration.
“The impact of shorter cycle times is that users can see the result of their input quickly. Instead of contributing to the planning process and then waiting for weeks to see the feature start to take shape, they can regularly see changes, making them happy and keeping them engaged with a team. This also helps reduce the scope creep that happens when a project has been in progress for a while.” – Rachel Nienaber
What's the average cycle time for development teams?
According to the Accelerate State of DevOps Report, the average cycle time for top-performing teams is about 2 days, with the median for most teams being about 3.5 days. However, some development teams report their average cycle times as being as much as 7 days. Teams can calculate this by evaluating how long several types of fixes take from start to finish.
What are some cycle time challenges?
Every team has processes and steps that increase cycle delivery time. A shorter and faster time to market empowers teams to fulfill customer demands and exceed their expectations. Here are a few of the ones we’ve dealt with in recent past.
Getting it right the first time
When developing new features, we want to ensure that things don’t break when it gets to a user. Because of our monthly release cycle, users could be stuck with a broken feature until the following month, causing frustration and decreasing the value that GitLab brings to its users. So, it’s important that we test and ship with certainty. Marin Jankovski, Engineering Manager of the Distribution & Release Management teams, and Sean McGivern, Engineering Manager of the Plan team, note the importance of testing and shipping features.
“Finding a way to test changes faster can be challenging. With the Distribution team, we have the responsibility of ensuring that the release we ship still functions after we make our changes and that users can still install and use GitLab.” – Marin Jankovski
“Our release process is a big challenge, if you consider that the cycle ends once customers have the feature available to use. We don’t have CD for GitLab.com, but even if we did, for self-managed customers, we only have one feature release a month. So, that’s a hard limit.” – Sean McGivern
Differentiating the helpful from the unhelpful
Every workflow has components that can decrease release cycles, including code reviews, manual configuration and testing, and hand-offs. Some of these elements are necessary, like product manager meetings, but other aspects can unintentionally cause problems. Tommy Morgan, Director of Engineering of Dev Backend, highlights the essential measures that teams need to take to promote collaboration and alignment but may increase cycle times.
“Teams have all these things that are slowing down cycle times, and there could be extra steps or extra involvement that aren’t necessary or beneficial and that could unintentionally add pressure to the team to slow down. One of the biggest challenges is identifying which ones are legitimate and helpful and which ones are us giving into the natural urge to add process. Identifying across that fine line is where the real challenge comes into play for most teams.”
Working across teams
Cross-collaboration fosters innovative thinking and allows each team to specialize in a specific area to maximize contributions. While the benefits of working with multiple teams are abundant, depending on another team’s feedback or assistance slows down development, especially when there’s a blocker that can only be resolved with the help of one team. Rachel Nienaber, Engineering Manager of the Geo team, and Marin agree that working across teams can have significant impact on cycle times.
“When other teams implement a new feature that needs some additional work from the Distribution side, getting informed in time is extremely important. We need to affect the decision as early as possible, because we have certain limitations when it comes to distributing GitLab.” – Marin Jankovski
“One challenge that I see is that there are a lot of dependencies on people external to the team to ship features. Ordinarily, a quick way to shorten cycle time is to reduce those dependencies, but here at GitLab, that may reduce the amount of collaboration that happens with each feature. Collaboration is such an important value that this may have to take precedence in some cases and be more important than the gain in speed.” – Rachel Nienaber
Asynchronous communication
At GitLab, we practice asynchronous communication, so we “don’t expect an instantaneous response,” allowing us to focus on our individual workflows. The problem with working asynchronously is that projects can become delayed when working with team members in different time zones and responses don’t trickle in until the following day. Rapid movement might not be made on projects because of time zone differences. Mek Stittri, Engineering Manager of the Quality team, and Rachel acknowledge the difficulties that can come with asynchronous communication.
“My team is spread across so many projects and has someone in almost every time zone, meaning communication can be challenging.” – Mek Stittri
“This is my first role with an asynchronous method of working. I am finding that many practices that work in a synchronous team need some adjustment to be useful here.” – Rachel Nienaber
What are some solutionsb to reducing cycle times?
At GitLab, we’re fortunate to have the freedom to experiment and iterate, so we’ve been able to develop a few strategies to help us alleviate the challenges we face when meeting customer demands by reducing cycle times.
How to get it right the first time
-
Automate work as much as possible. Using CI to automatically do releases and investing time in automating other manual tasks is crucial for delivery. Manual tasks are both a huge drain on morale and prone to errors. It’s much easier to give engineers a bug to fix in an automated tool than to ask them to do the same thing multiple times.
-
Work with smaller, iterative pieces. Breaking work into smaller chunks, iterating frequently, and indicating priority more clearly within a milestone enables better predictability for what’s going to ship. Planning becomes easier, because individual issues are smaller, so it’s easy to shuffle issues around if something unexpected interrupts other work.
-
Use feature flags. Rather than using a giant merge request to make every change for a feature at once, which is harder to review, update, and keep up-to-date with the master branch, consider developing more features behind short-lived feature flags.
How to differentiate the helpful from the unhelpful
-
Measure the impact of components. Measuring impact can help determine whether a process either doesn’t help out that much in the end or helps out infrequently. In either case, the net benefit can be small, but the pain it adds (in terms of how much extra time you spend trying to ship) makes the overall impact negative. If you can’t measure impact directly, you have to be willing to experiment. Try things, see how they work, and decide if you should keep them or not. It’s important to remember that experimentation doesn’t mean process creep - the default end state for an experiment should be “let’s never do that again,” unless there’s a strong sense of value in it.
How to successfully work across teams
-
Communicate and automate where possible. Automating how others get a finished product before releasing it (e.g. create a package on click) and broadly communicating how to work with a team can result in better decisions and faster discussions.
<li>
<p>
<b>Develop a training program.</b> Creating a training program to help engineers
from other teams perform reviews can reduce cycle time for those teams
that regularly depend on the Database team. This strategy has the added
benefit of giving the Database team more time to focus on their own work.
</p>
</li>
<li>
<p>
<b>Use project management tooling.</b> Consistent <a href="/handbook/engineering/quality/project-management/">project management tooling</a>
ensures consistent board configuration that behaves the same at every level,
meaning that data rolls up to one top level board which contains a
snapshot of an entire team, ensuring that prioritization is clear and
workload is transparent.
</p>
</li>
<li>
<p>
<b>Spread triaging.</b> To spread the load of triaging across teams, use <a href="https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf8=%E2%9C%93&state=closed&label_name[]=triage-package">triage-package</a>.
Here is a <a href="https://gitlab.com/gitlab-org/gitlab-ce/issues/52024">recent example</a>
of how we used triage-package to lessen the burden on one team.
</p>
</li>
<li>
<p>
<b>Have more focused milestones.</b> Creating focused milestones can reduce
context switching, since team members can concentrate on specific aspects
of a feature.
</p>
</li>
How to make asynchronous communication work
-
Work on multiple items. Having a list of multiple items to work on during each release cycle helps team members easily transition to another task rather than remaining blocked when waiting for feedback.
Advice
Reducing cycle times to meet internal cycle time goals can be a difficult undertaking, requiring the input from product managers, engineering leads, and developers. It’s a hard task to challenge long-practiced behaviors, especially when the worst case scenario could mean features don’t make a release. Here is some advice to help your team's cycle time reduction effort.
Be thoughtful and considerate
“At GitLab, we want to iterate quickly, but we also want to keep GitLab.com fast and stable. That means that we can’t just decide to ship things faster, we need to come up with strategies to mitigate any risks to performance and availability, build tooling and processes around those strategies. This is often work that can go underappreciated, and it can be hard at times, but it’s vital to ensuring that you can safely shorten cycle times.” – Sean McGivern
Retrospectives for learning
“A successful team is a happy team. Bringing down production cycle time can help a team be more successful because they are shipping value more often, but your team might have more important things that must be addressed first. Using retrospectives will help you to figure out what success means to your team, and what needs to be done to achieve that success.” – Rachel Nienaber
Experiment
“Make yourself uncomfortable. It’s unnatural to push for shorter cycle time. It’s natural to add steps - it’s not natural to remove them. Try drastic cuts and be willing to learn from an experiment.” – Tommy Morgan
Spotlight your team
“You can’t make product managers happy, so try to make your team happy instead by giving them a chance to shine. :P” – Marin Jankovski
