Competitors - Code Review

1. You are here:
2. GitLab Direction
3. Product Stage Vision - Create
4. Category Direction - Code Review
5. Competitors - Code Review

• Competitive Analysis
  • Mature
    • Gerrit
      • Summary
      • Findings
    • Phabricator
      • Summary
      • Findings
    • Crucible
      • Summary
      • Findings
  • Not mature
    • Atlassian Bitbucket
      • Summary
      • Findings
    • Azure DevOps
      • Summary
      • Findings
    • GitHub
      • Summary
      • Findings
    • Review board
      • Summary
      • Findings
    • Reviewable
      • Summary
      • Findings

Competitive Analysis

This is a brief competitive analysis for Code Review, focused on the user experience and feature set of our competitors. Each one starts with a summary, followed by the list of findings, with the most interesting highlighted. Tools are separated into two sections: mature and not mature.

Mature

Gerrit

Homepage • Docs • Example code review

Summary

A tool specific for reviewing code, everything about the experience is focused on that. The UI is quite dense. Merge requests are called “Changes”. It focuses on only one commit, where authors can upload patch sets (revisions). “Changes” have an overview page and to see the files one must click-through into a specific “diff mode”. The overview page acts as a summary and lists the activity in chronological order, with comments unthreaded. Three interesting features are the ability to review commit messages, show the git blame inline, and auto-add reviewers based on git blame.

Findings

1. The commit message is effectively the Change’s description. (docs)
   1. Commit messages have a max-height and are then expandable/collapsible.
2. Status: Needs kind-of-review, Not Current, Ready to Submit, Merged, Abandoned.
3. Fields are organized in an area called “Change info” (docs)
4. Owner: displayed as a link to a list of the owner’s changes that have the same status as the currently viewed change. (docs)
5. Reviewers: each has a tooltip that shows on which labels the reviewer is allowed to vote.
6. Labels: our “approval rules” (docs)
7. Topics: a way to group changes (commits) related to a single topic. Could be considered our “issues” (docs)
8. File list contains auto-generated “magic files” Commit Message and Merge List (docs). It shows type of file modification, file path, number and status of comments (new, drafts, count), the size changes (additions/deletions), and a total of the size changes. Checkboxes in the file list allow files to be marked as reviewed. In the header of the file list there’s a “Diff Against” selection to compare patch sets and an “Open All” button that opens the diff view separately.
9. If a non-current patch set is viewed this is highlighted by the Not Current change state (docs)
10. The Download drop-down offers commands and links for downloading the currently viewed patch set. There is a drop-down to switch between the download schemes (SSH or HTTPS). The last chosen download scheme is remembered. (docs)
11. Uses the “star” metaphor and icon to subscribe to notifications. (docs)
12. Related changes tab: shows descendant and ascendant changes (dependencies) in a list view where the current change is in the middle (docs)
13. Same topic tab: open changes that share the topic (docs)
14. Reply button: opens a popup panel to publish inline comments, allows voting (approval) and adds a summary comment. The inline draft comments are listed to be reviewed before publishing, with links. (docs)
15. History: very compact list, similar-looking to an email inbox. New comments since the user’s last review are automatically expanded and bolded. Inline comments are directly displayed in the history with links to jump there, they are grouped per review. It has an expand/collapse all button. (docs)
16. Updates: there’s a polling system that shows a notification (similar to Gmail) when there updates. Users can Show or Ignore. (docs)
17. Checks tab (pipelines)
18. Diffs (docs):
    1. Can be marked as reviewed (can be done automatically per user preference)
    2. Scrollbar with map of diffs and comments
    3. Patch set comparisons can be changed in the file header via a numbered list.
    4. Arrows to move between files.
    5. Option to expand all comments as a user preference.
    6. Drafts are “saved” or “discarded”.
    7. Inline comments can be started by clicking on a line or selecting lines and then clicking a button.
    8. Comments can be added to the whole file.
    9. Uses “Done” instead of “Resolve” in comments.
    10. Ability to show blame in diff view.
19. List: Changes (merge requests) are classified in Size per the number of changed lines, using t-shirt sizes (e.g. XS = +1,-1)
20. Add reviewers based on the git blame computation of the changed files (docs)

Phabricator

Homepage • Docs • Example code review

Summary

The code review experience in Phabricator is similar to Gerrit’s. However, the whole review UI is one big scrollable page, instead of being sectioned into different views/pages. Three other notable differences are the support for both pre and post-merge reviews, showing older inline comments in updated diffs (ported/ghost comments), and showing test and coverage reports inline.

Findings

1. Supports two similar but separate code review workflows: "review" and "audit" (post-commit).
2. List view:
   1. Action Required is revisions you are the author of or a reviewer for, which you need to review, revise, or push.
   2. Waiting on Others is revisions you are the author of or a reviewer for, which someone else needs to review, revise, or push.
3. Detail view is a single-view, scrollable:
   1. Details: reviewers and review status, links to Trac issues, summary, test plan
   2. Actions, tags, and subscribers
   3. Diff detail: repository, branch, lint status and warnings, unit test result, build status
   4. Activity
   5. Changes
4. Separate view with audit table for inline comments with the attributes: diff no., status, reviewer, comment, created date (relative) (example)
5. Activity is shown chronologically, unthreaded
   1. When adding inline comments OR marking inline comments as done, a regular comment acts as a summary. Inline comments are listed at the bottom of the summary, indicating the file and with a link to the line.
   2. If the history is large, older items are hidden with a “Show Older Changes” button that progressively shows more and more older items.
6. Inline comments are threaded, and can be hidden/shown by the following aspects: “Done”, collapsed, older, all.
7. Multiline comments can be added by clicking one line number and dragging to a nearby line number.
8. Inline comments are initially unpublished (drafts). They are published in batch, alongside an optional normal comment and optional action (like accepting the revision).
9. Ported/ghost comments: When a revision is updated, it attempts to port inline comments forward and display them on the new diff. Ported comments have a pale, ghostly appearance and include a button that allows you to jump back to the original context where the comment appeared. Ported comments sometimes appear in unexpected locations, especially in the presence of heavy changes, but this helps prevent mistakes and makes it easier to verify feedback has been addressed.
10. Highlights when changes are pushed, when pipelines fail
11. Image macros to add frequent GIFs to comments
12. Ability to add test and coverage reporting in the specific lines of code
13. By default, approvals are not reset when new commits are pushed.
14. People can be automatically notified about a review based on rules (like content, author, files affected, etc.).

Crucible

Homepage • Docs

Summary

It’s the only code review tool we reviewed that attempts to port the IDE experience to a browser UI for code review, with a navigation sidebar to jump between sections and files, and a fixed content view. Like Phabricator, it supports pre and post-merge reviews but is more flexible about the review’s starting point: specific commits, whole branches, committed files, patch files, or attachments.

Findings

1. Roles: author, reviewer, and moderator (optional)
   1. Reviewers can be auto-suggested, by analyzing the users that have contributed to the files you've selected and also don't have a lot of open reviews.
2. Reviews can be created from commits/changesets, branches, specific files, patch files (pre-commit), or attachments. (docs)
3. Allowing anyone to join the review is an option configured per review. Else, only selected reviewers can participate.
4. Status: Draft (default on creation), Under review (after Start review is selected), Summarized (once all reviews are submitted), Closed or Abandoned (docs)
   1. Start Review sends a notification email to all participants.
5. Ability to link to issues and other reviews on creation.
6. Comments: can be added to the whole review, to a file, to a single line, to multiple lines (click and drag to select multiple lines or select text)
   1. Can be marked read/unread, marked as defect or needs resolution, saved as draft or published immediately
      1. Defects can be ranked (major, minor, etc.) and also classified: Wrong, Can be improved, Risk-prone, Security hole, Over engineered, Code duplication
   2. In the file tree, there’s a counter of total comments per file (and unread comments)
7. User switches between 3 main views: Summary, Activity, Files.
8. Summary: Details, Objectives, General comments
   1. Details: list of participants, role, time spent, no. of comments, and link to the last comment
9. Activity: All activities in chronological order, filterable by resolved and/or unresolved comments.
10. List view:
    1. Inbox: To Review, Ready to Close, In Draft, Require My Approval
    2. Outbox: Out for Review, Completed
    3. Archive: Closed, Abandoned, My Open Snippets, My Snippets

Not mature

Atlassian Bitbucket

Homepage • Docs • Example code review

Summary

In general, its pull requests and review experience are very similar to GitHub.

Findings

1. Ability to create tasks from comments and view a list of unresolved tasks. (docs)
2. Iterative reviews: Within the diff view you can select a specific commit to review, or choose to view all changes within a pull request. If you return to a pull request you previously reviewed, you'll only see the new commits added since you last reviewed the pull request.
3. Default reviewers can be added to a combination of source-target branches with the following options: Branch name, Branch pattern, Branching model, or Any branch. (docs)

Azure DevOps

Homepage • Docs

Summary

Like Bitbucket, it largely follows the GitHub’s experience, but manages to give it more structure and add many more features. Of note are their consistent locations for actions and secondary information, granular types of approval, good approach to merge confirmation and actions, and the visibility of policies (items blocking the merge) and status of the pull request.

Findings

1. List view of created and assigned pull requests (docs)
2. Allows selection of “Reviewers” and “Work Items” (related issues) in the PR creation form (docs)
3. Specific area just for related work items (issues) (docs)
4. States: Draft (WIP) > Published > Complete (merged)
5. Main actions: Complete (Merge), Set auto-complete (Merge when all branch policies pass), Mark as draft (WIP), and Abandon (Close) (docs)
   1. Completing (merging) opens a dialog to edit the merge commit message and select merge options (docs)
   2. Setting auto-complete a banner is shown, confirming that the changes will be merged as soon as the policies are satisfied. Has {: .mark} a “Cancel auto-complete” option and displays the outstanding list of policy criteria.
6. Main sections: Overview (state at a glance and discussion), Files (changes), Updates (versions/pushes), Commits
7. Uses the concept of branch policies to block pull requests from being merged. It shows an easy to read list of things remaining. (docs)
8. Ability to open file changes in full-screen mode
9. Comment states: Active (default), Resolved, Pending, Won’t Fix, Closed (docs)
10. Mixes the concept of approving with casting votes (docs)
11. Votes: Approve, Approve with suggestions, Wait for author, Reject, or Reset feedback
    1. Approve with suggestions: Agree with the pull request, but provide optional suggestions to improve the code.
    2. Wait for author: Do not approve the changes, and ask the author to review your comments. The author should let you know when you should re-review the code after they have addressed your concerns.

GitHub

Homepage • Docs • Example

Summary

GitLab’s current code review experience is largely modeled after GitHub’s, with a discussion/activity view and a changes view with all of the files stacked vertically. Some relevant differences in GitHub are resolving whole discussions (instead of comments), the concept of reviewers, submitting a summary comment with the review, and viewing only the changes done since the user’s last review. Some of its frequently mentioned issues are the difficulty in finding who is responsible for looking at the code, the lack of inline test and coverage reports, and the lack of an audit trail.

Findings

1. Uses the concept of “Reviewers” to request reviews. Code owners are automatically requested for review when someone opens a pull request that modifies code that they own. (docs)
2. When submitting a review, the reviewer can attach a summary comment to their inline comments. Besides “Approve”, a review has two additional types: “Comment” and “Request changes”. (docs)
3. Supports resolving whole discussions, instead of comment by comment. (docs)
4. Uses the term “conversations” instead of “threads” or “discussions”.
5. Allows navigating changes commit-by-commit, with convenient previous-next buttons. (example)
6. Allows commenting on multiple lines. (example)
7. Allows viewing only the changes done since the user’s last review.
8. Users can mark files as viewed to collapse them until they are changed again. (docs)

Review board

Homepage • Docs • Example code review

Summary

Even though the UI is not the most appealing or modern, Review board has some interesting features that set it apart. Specifically, the ability to filter comments by type and author, and the personal dashboard that is organized like a “code review inbox”.

Findings

1. Supports both pre and post-merge reviews.
2. Dashboard: Outgoing (All, Open); Incoming (Open, To Me, Starred)
3. Review requests list: Flag with published/unpublished comments, No. of reviews, New updates since last view, Ship it!/Issue counts (approvals, open comments), Flag to me (which directly list you in the “people” field)
4. Status: Draft, Published, Closed (“submitted” or as “discarded)
   1. Closing allows to describe the reason why it was closed (optional).
5. Roles: Owner, Reviewers (group or people)
6. Reviews consist of a header, a list of comments, and a footer. Comments can be general, to a file, and diffs.
7. In reviews, “issues” can be flagged. They can then be marked as resolved or dropped. Can be re-opened.
   1. If an issue is particularly important, you can select Require Verification when creating the comment. This will make it so the owner of the review request cannot directly mark the issue as fixed or dropped. Instead, only the reviewer (or an administrator) can then verify whether the issue should be closed or reopen it.
8. Below the description a list of open issues (comments) is filterable by: type and author.
9. Deleted file contents are not shown by default, as this code often doesn’t need to be reviewed. Instead, you’ll see a simple message stating that the file was deleted and the option to expand.
10. Multiline comments are started with click and drag through multiple line numbers.
11. Diff comments are shown next to the file with a comment counter per line. Published comments are blue, drafts are green.

Reviewable

Homepage • Docs • Example code review

Summary

It has a fresh approach to navigating a code review, with its intuitive toolbar that allows cycling between files, discussions, and drafts, and also a visual way to change which revisions are compared. There’s a big focus on the reviewer experience, especially when re-reviewing changes, by highlighting what files and discussions they need to review again or reply to. The status of the pull request is given visually with a small circular chart that shows the passed/failed checks. Other highlights include the ability to track of who re­viewed which re­vi­sion of each file, automatically entering single file mode to preserve performance, and setting merge options and commit message before merging.

Findings

1. Roles: Author, Reviewers, Mentionee (mentioned, becomes reviewer if starts a discussion or marks a file as reviewed)
2. Approval levels: Comment, Approve, Request changes
3. Merge (docs)
   1. When a review is complete, a victory graphic appears.
   2. You can set merge options and edit the merge commit message via the dropdown attached to the button.
   3. To merge the pull request, click and hold the button for a couple seconds until the arming indicator fills with yellow and starts pulsing, then release. This procedure is in place to reduce the chances of accidentally merging a pull request without requiring a separate confirmation.
4. Review toolbar:
   1. Checks: This item summarizes the current condition of GitHub's CI commit statuses and checks, mergeability, and review completion. The donut chart icon indicates the relative proportion of successful, pending, and error/fail states.
   2. Changes: This item summarizes the changes you're currently looking at.
   3. Counters: Files, discussions, and drafts. Red counters indicate that you must address the given number of items to advance the review. Grey counters indicate that other participants must address the given number of items, but you're in the clear. Click to cycle through them
   4. Main action: switches between Publish and Merge, if the user has unpublished comments.
5. Mark as reviewed: letting you track the reviewed state of each file at each revision, for each reviewer. This enables you to easily remember — and indicate to others — where you left off in the review and easily view only subsequent changes. Review marks remain in a draft state and are only visible to you until published.
6. File tree: shows which files you have reviewed (even in draft) and who else has reviewed them.
7. Single file mode: Automatically enters a single file mode to preserve performance when the number of visible files exceeds a threshold (25 files as of this writing). In this mode, only one file will appear at a time. You'll also see a summary explanation just above the first (and only) file diff, together with a button that you can click to force all files to appear anyway until you leave the page.
8. Comment status: Discussing, Satisfied, Informing, Working, or Blocking.
9. Dashboard: Waiting on me, Assigned to me
   1. Review state is visible, with counters reflect the same information as you'll see on the review page