The APM Group is responsible for providing the underlying libraries and tools to enable GitLab team-members to instrument their code. When adding new metrics, we need to consider a few facets: the impact on GitLab.com, customer deployments, and whether any default alerting rules should be provided.
Recommended process for adding new metrics:
Open an issue in the desired project outlining the new metrics desired
Label with the ~group::apm label, and ping @gl-monitoring for initial review
Impact on cardinality and performance of Prometheus
Whether any alerts should be created
Assign to an available APM Group reviewer
Milestone Planning
In planning a milestone, the engineering and product team work closely together. We do follow the Engineering Workflow and specifically the Product Development Timeline when planning a milestone. However, here is some additional information that may help in the planning process.
For APM we use the APM Planning Board to do milestone planning and to prioritize our backlog. Here is what we do to keep that board up-to-date:
Identify any issues that will not be completed in the current release and move them to the top of the next/upcoming release column.
Due Date: Last day of the month. Note: although the end of the month is only about half-way through the continuous delivery release cycle, we will hopefully have a good sense by then what we are working on and what can get done in the current release.
Responsible: Engineering Managers with input from the engineering teams
Apply the Deliverable label to the top items in the upcoming milestone that will be completed. These should be the top priority items that are almost sure to be completed by the end of the upcoming milestone.
Due Date: Fourth day of the month
Responsible: Engineering Managers with input from the engineering teams
Apply the “direction” label to the top issues for the upcoming milestone that we want to highlight in the Kickoff call.
Due Date: Before the kickoff call on the 8th of the month
Responsible: Product Manager
Move items from the upcoming release that will not be worked on into a future release to be prioritized later. Some issues that were not considered "Deliverable" could remain in, and be labeled with the Stretch label.
Due Date: Shortly after the kickoff call
Responsible: Product Manager and Engineering Manager
By using the planning board as a priority list, and by keeping it in order, then we should always be able to look at the current and upcoming milestone columns to have a prioritized list of upcoming work.
Recurring Meetings
While we try to keep our process pretty light on meetings, we do hold a Monitor APM Weekly Meeting to triage and prioritize new issues, discuss our upcoming issues, and uncover any unknowns.
Repos we own or use
Prometheus Ruby Mmap Client - The ruby Prometheus instrumentation lib we built, which we used to instrument GitLab
