Code Review Runbook
We've spoken with many engineering teams who struggle with getting their code reviews over the finish line. One of the most common sentiments we've heard is that completing thorough code reviews quickly while balancing other tasks and without sacrificing quality can be difficult. There are a lot of things that can go wrong — and it’s often hard to pinpoint exactly where the problems exists.
Our team wrote An efficient code review process has fast feedback loops, which breaks down many of the pains that teams experience during code reviews, and shares a few frameworks that teams can use to rethink how they improve them. One of the challenges with code reviews that teams experience today is that developers are often either waiting on their tools and other team members, or they’re stuck trying to find their way through complex workflows:
- Poorly scoped tickets overwhelm developers
- Oversized pull requests are hard to review
- Delays between reviews and comments lead to loss of context
- Work slips through the cracks because its not visible
If you’re interested in taking some of the chaos out of code reviews, here are the first six things you can do to get the most out of Software:
- Find your team’s bottlenecks. See how long it takes to review, merge, and approve pull requests. Dig into your data to look for bottlenecks that might be holding back your team.
- Check for stale pull requests. If you’re looking to reduce the time it takes to ship code, stale pull requests are the lowest-hanging fruit. Check if any open pull requests are currently stuck in review.
- Share your team’s open pull requests in Slack. Collaborate with your team to tackle the highest priority pull requests. Use the /pull_requests slash command to see your team’s open pull requests and their current status directly in Slack, where you can then either start a group review or view them on GitHub.
- Set up a team goal for review time. As teams grow in size, it becomes harder to stay on top of slipping work. Set a goal to make sure work doesn’t slip through the cracks (e.g. review changes in less than a day).
- Schedule your team’s daily digest. Set up daily digests in Slack to get an automated reminder about open pull requests, sorted by status, each day. It’s a great way to simplify standup meetings or nudge team members to review changes.
- Prepare for a data-driven retrospective. Celebrate your team’s wins, and take action when bottlenecks occur. See how your team is trending over time relative to your goals and historical averages.
Find your team’s bottlenecks
The Review dashboard gives a quick pulse on the health of your team’s code review process. It is automatically created when you connect your team’s GitHub organization.
The dashboard shows you how long it takes to review, approve, and merge pull requests. More specifically, Time from Open to Merge shows you the total time it takes to complete a full code review of each pull request on average.
Time to Review, Approve, and Merge let you zoom in on specific steps in the code review process. They give you a breakdown of code reviews by stage to see where you can experiment and improve.
- Time to review: time between opening a pull request and the first review
- Time to approve: time between the first review and first approval submitted
- Time to merge: time between the first approval and merging those changes to the main branch
You can also view the average pull request size (to prevent large pull requests from blocking reviews) and code review participation (to ensure the review workload is shared across team members). Pull Request Size is the average number of lines of code changed per pull request. Code Review Participation is the percent of contributors on your team participating in code reviews.
Measuring time spent reviewing code shows you where your bottlenecks happen and how you can focus your experiments on specific parts of the code review process.
Tip: Use team filters to see more relevant data to your specific team. Teams are automatically synced from GitHub (you can learn more about setting up teams in GitHub’s documentation). If you’ve already set up teams in GitHub, they will also be shown in Software without requiring any extra configuration.
Check for stale pull requests
While the Review section shows your team’s trends, the Open Pull Requests section makes it easier to stay on top of work that’s in-flight and view all of your open pull requests. You can quickly see which pull requests are waiting for reviews or approvals, and which are ready to merge.
At the top, you can see summary metrics for your team, including the total number of open pull requests and the current average wait time to review, approve, and merge pull requests.
In the table below, you can scan all your team’s open pull requests broken down by stage (awaiting review, awaiting approval, and ready to merge). From here you can quickly find and open in GitHub any pull requests that need attention.
The Open Pull Requests view is great for uncovering which pull requests might be stuck and can help your team prioritize their reviews.
From here, you can invite team members to view your dashboard to wrap up any lingering reviews. They will get access to the same breakdown of pull requests by status, so they can keep track of their work in progress.
Tip: Use the bot toggle to filter out pull requests opened by bots. While pull requests opened by bots can still be important, since they often include security or dependency updates, you can filter them out if there’s higher priority work in the queue.
Share your team’s open pull requests in Slack
Delays in code reviews sometimes happen when teams need better visibility into when pull requests are ready to be reviewed or re-reviewed.
While the Open Pull Requests page provides a detailed view of open pull requests, you can also connect Slack to collaborate on pull requests in real-time. After connecting Slack, try running the /pull_requests command to share with your team a list of all your open pull requests.
Tip: When team members connect their individual GitHub accounts, the Software bot can tag specific people in Slack messages. Run the command /connect_software to associate your GitHub and Slack usernames.
Set up a team goal for review time
Goals allow you to create ‘working agreements’ with your team. You can set goals for your code reviews, so work doesn’t slip over time. Your goals are then shown alongside your team’s open pull requests, both in the Open Pull Requests view and your Slack daily digests.
For example, you can set a goal for your team to review pull requests within a day of opening them.
Every team can set their own goals, so as a team leader you have the ability to figure out what works best for your team. Out of the box, Software automatically suggests goals for your team, but you can customize each goal to fit your needs.
Tip: When you configure a goal, you can see any pull requests that didn’t meet it. You can use these exceptions to look for specific blockers during code reviews.
Schedule your team’s daily digest
Daily digests show in Slack where in the code review process your team’s pull requests are waiting. They are an easy way to share updates with your team each day about work in progress.
Every team has a default Slack channel for sending their digests, which you can set in your team settings. Schedule your first digest by picking a channel. It will be sent every weekday at 9am in your organization’s time zone, which can be changed in the settings.
Tip: Share wins with your team. Many of our users tell us how they use their data to recognize the impact of their team’s efforts over the last cycle.
Prepare for a data-driven retrospective
Retrospectives are an opportunity to share wins with your team, revisit goals, and plan for upcoming work. You can use the filters on the Review page to dig into your team’s trends over the last 14, 30, or 90 days. It can serve as a jumping off point for conversations about your team’s review process and new tools as you think about your next sprint or cycle.
Resources
If you’re interested in reading more about code reviews and ways to experiment, check out these articles.
