The Import and Integrate group is a part of the Manage Stage. The group supports the product
This page covers processes and information specific to the Import and Integrate group. See also the directions pages(Importers, Integrations, Internationalization) and the features we support per category.
To get in touch with the Import and Integrate group, it's best to create an
issue in the relevant project (typically GitLab) and add the
~"group::import and integrate" label, along with any other appropriate labels. Then,
feel free to ping the relevant Product Manager and/or Engineering Manager.
For more urgent items, feel free to use the Slack Channel (internal): #g_manage_import_and_integrate.
The following people are permanent members of the group:
| Person | Role |
|---|---|
| Magdalena Frankiewicz | Product Manager, Manage:Import and Integrate |
| Martin Wortschack | Engineering Manager, Manage:Import and Integrate |
| Bojan Marjanović | Senior Backend Engineer, Manage:Import and Integrate |
| Carla Drago | Senior Backend Engineer, Manage:Import and Integrate |
| George Koltsov | Senior Backend Engineer, Manage:Import and Integrate |
| Justin Ho | Senior Frontend Engineer, Manage:Import and Integrate |
| Luke Duncalfe | Senior Backend Engineer, Manage:Import and Integrate |
| Rodrigo Tomonari | Senior Backend Engineer, Manage:Import and Integrate |
You can find our group metrics in the Manage:Import and Integrate Sisense Dashboard and Import and Integrate Group Engineering Metrics handbook page.
(Sisense↗) We also track our backlog of issues, including past due security and infradev issues, and total open System Usability Scale (SUS) impacting issues and bugs.
(Sisense↗) MR Type labels help us report what we're working on to industry analysts in a way that's consistent across the engineering department. The dashboard below shows the trend of MR Types over time and a list of merged MRs.
(Sisense↗) Flaky test are problematic for many reasons.
The Product Manager uses milestone priority labels and compiles the list of Deliverable and Stretch issues following the product prioritization process, with input from the team, Engineering Managers, and other stakeholders. The iteration cycle lasts from the 18th of one month until the 17th of the next, and is identified by the GitLab version set to be released on the 22nd.
In general, we use the standard GitLab engineering workflow.
The easiest way for Engineering Managers, Product Managers, and other stakeholders to get a high-level overview of the status of all issues in the current milestone, or all issues assigned to a specific person, is through the Current milestone board, which has columns for each of the workflow labels.
As owners of the issues assigned to them, engineers are expected to keep the workflow labels on their issues up to date, either by manually assigning the new label, or by dragging the issue from one column on the board to the next.
Once an engineer starts working an issue, they mark it with the workflow::"in
dev" label as the starting point and continue updating the issue throughout development.
The process primarily follows the guideline:
If someone starts working on an issue but it has the same workflow label for a week, the assignee has to leave a comment explaining the status of the issue. We should write at least one comment every week that the issue is not moving.
The work for the Import and Integrate group can be tracked on the following issue boards:
We use a lightweight system of issue weighting to help with capacity planning. These weights help us ensure that the amount of scheduled work in a cycle is reasonable, both for the team as a whole and for each individual. The "weight budget" for a given cycle is determined based on the team's recent output, as well as the upcoming availability of each engineer.
Since things take longer than you think, it's OK if an issue takes longer than the weight indicates. The weights are intended to be used in aggregate, and what takes one person a day might take another person a week, depending on their level of background knowledge about the issue. That's explicitly OK and expected. We should strive to be accurate, but understand that they are estimates! Change the weight if it is not accurate or if the issue becomes harder than originally expected. Leave a comment indicating why the weight was changed and tag your EM so that we can better understand weighting and continue to improve.
The weights we use are:
| Weight | Description |
|---|---|
| 1: Trivial | The problem is very well understood, no extra investigation is required, the exact solution is already known and just needs to be implemented, no surprises are expected, and no coordination with other teams or people is required. Examples are documentation updates, simple regressions, and other bugs that have already been investigated and discussed and can be fixed with a few lines of code, or technical debt that we know exactly how to address, but just haven't found time for yet. |
| 2: Small | The problem is well understood and a solution is outlined, but a little bit of extra investigation will probably still be required to realize the solution. Few surprises are expected, if any, and no coordination with other teams or people is required. Examples are simple features, like a new API endpoint to expose existing data or functionality, or regular bugs or performance issues where some investigation has already taken place. |
| 3: Medium | Features that are well understood and relatively straightforward. A solution will be outlined, and most edge cases will be considered, but some extra investigation will be required to realize the solution. Some surprises are expected, and coordination with other teams or people may be required. Bugs that are relatively poorly understood and may not yet have a suggested solution. Significant investigation will definitely be required, but the expectation is that once the problem is found, a solution should be relatively straightforward. Examples are regular features, potentially with a backend and frontend component, or most bugs or performance issues. |
| 5: Large | Features that are well understood, but known to be hard. A solution will be outlined, and major edge cases will be considered, but extra investigation will definitely be required to realize the solution. Many surprises are expected, and coordination with other teams or people is likely required. Bugs that are very poorly understood, and will not have a suggested solution. Significant investigation will be required, and once the problem is found, a solution may not be straightforward. Examples are large features with a backend and frontend component, or bugs or performance issues that have seen some initial investigation but have not yet been reproduced or otherwise "figured out". |
Anything larger than 5 should be broken down if possible.
Security issues are typically weighted one level higher than they would normally appear from the table above. This is to account for the extra rigor of the security release process. In particular, the fix usually needs more-careful consideration, and must also be backported across several releases.
Every week the engineering team completes a backlog refinement process to review upcoming issues. The goal of this effort is for all issues to have a weight so we can more accurately plan each milestone using the estimated capacity for the team and the estimated issue weights.
In addition to this backlog refinement process, engineers on the team can add weights to any issues that are straight-forward and do not need backlog refinement.
This process happens in three steps.
The engineering manager will identify issues that need to be refined. On average we will try to refine 3-6 issues per week. If there are issues that are good candidates for the backlog refinement process, please let the engineering manager know in the issue.
When picking issues to refine, we try to have themed refinements to reduce the context switching while the issues are being investigated. Here are some places to look:
Once identified, the engineering manager will apply the ready for next refinement label, which will indicate the issues are ready for
refinement.
The engineering manager will use the Refinement Bot to generate an issue with all the issues that have been identified for refinement.
Over the week, each engineer on the team will look at the list of issues selected for backlog refinement. Current backlog refinement issues.
For each issue, each team member will review the issues and provide the following information:
Some considerations:
After engineers have had a chance to provide input, the engineering manager will then:
ready for next refinement label.workflow:: label based on the outcome of the refinement. Example workflow::ready for development.For any issues that were not discussed and given a weight, the engineering manager will work with the engineers to see if we need to get more information from PM or UX.
The group has an existing threat model to assist in identifying issues that may have security implications, but there are other considerations.
An Application Security Review should be requested when the issue or MR might have security implications. These include, but aren't limited to, issues or MRs which:
CommandLineUtilFor issues which need to be announced in more detail, a release post can be automatically created using the issue. When working on an issue, either in planning, or during design and development, you can use the release post item generator to have the release post created and notify all the relevant people.
If you do not want an issue to have a release post, make sure that the issue does not have a
release notes section or do not use a release post item:: label.
We strongly believe in Iteration and delivering value in small increments. Iteration can be hard, especially when you lack product context or are working on a particularly risky/complex part of the codebase. If you are struggling to estimate an issue or determine whether it is feasible, it may be appropriate to first create a proof-of-concept MR. The goal of a proof-of-concept MR is to remove any major assumptions during planning and provide early feedback, therefore reducing risk from any future implementation.
PoC: .The need for a proof-of-concept MR may signal that parts of our codebase or product have become overly complex. It's always worth discussing the MR as part of the retrospective so we can discuss how to avoid this step in the future.
We have 1 regularly scheduled "Per Milestone" retrospective, and can have ad-hoc "Per Project" retrospectives.
The Import and Integrate group conducts milestone retrospectives in GitLab issues. These include the engineers, UX, PM, and all stable counterparts who have worked with that team during the milestone.
Participation by our team members is highly encouraged for every milestone.
These are confidential during the initial discussion, then made public in time for each month's GitLab retrospective. For more information, see group retrospectives.
If a particular issue, feature, or other sort of project turns into a particularly useful learning experience, we may hold a synchronous or asynchronous retrospective to learn from it. If you feel like something you're working on deserves a retrospective:
All feedback from the retrospective should ultimately end up in the issue for reference purposes.
When areas of the Import and Integrate codebase are changed, the reviewer roulette will recommend that the merge request is reviewed by an Import and Integrate team member. This will only happen when the merge request is authored by people outside of the Import and Integrate team. See this example of how the review recommendation looks.
The reasoning behind these special recommendations is that other groups have some ownership of certain integrations or webhooks. Reviewing changes made by non-team members allows us to act as owners of foundational code and maintain a better quality of the Import and Integrate codebase.
File paths of changes in a merge request are matched against a
list of regular expressions.
The roulette uses these hash values to recommend reviewer groups. For example, :import_integrate_be and
:import_and_integrate_fe will recommend Import and Integrate backend and frontend reviews respectively. As the regex matches
are first match wins
and not cumulative, any other relevant reviewer groups like :backend or :frontend must also be included
in each hash value.
The regex list should be updated to match integrations or webhooks code whenever needed. The list matches our commonly namespaced files, so new code in existing namespaces will always match.
To see which files in the GitLab repository produce a match, paste the following in a Rails console:
require Rails.root.join('tooling/danger/project_helper.rb')
ALL_FILES = Dir.glob('**/*');
def category_regexs(category)
matching_categories = Tooling::Danger::ProjectHelper::CATEGORIES.select do |regexs, categories|
next if regexs.is_a?(Array)
Array.wrap(categories).include?(category)
end
regexes = matching_categories.map(&:first)
Regexp.union(*regexes)
end
def print_files(category)
regex = category_regexs(category)
puts ALL_FILES.grep(regex).reject { |path| File.directory?(path) }.sort
end
puts "Backend:\n"
print_files(:import_integrate_be)
puts "Frontend:\n"
print_files(:import_integrate_fe)
This is a collection of links for monitoring our features.
queue dropdownJiraConnect::SyncMergeRequestWorker errors.JiraConnect::SyncBranchWorker errors.JiraConnect::SyncProjectWorker errors.GitLab uses error budgets to measure the availability and performance of our features. Each engineering group has its own budget spend. The current 28-day spend for the Import and Integrate team shows in this Grafana dashboard.
Error budget spend happens when either of the following exceeds a certain threshold:
To determine the highest-priority problems in our Grafana dashboard:
Fixing the top offenders will have the biggest impact on the budget spend.
Learn more about error budgets with these resources:
feature_category: :api, feature_category: :integrations, feature_category: :internationalization, feature_category: :importers, and feature_category: :webhooksgitlab-org/manage