Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Meltano Engineering Handbook

On this page

Code

See the active project here: gitlab.com/meltano/meltano

Read our documentation here: https://meltano.com/docs/

Triage process

The flow::Triage label is used on issues that need product/prioritization triage by the Product Manager (Danielle), or engineering/assignment triage by the Engineering Lead (Douwe). After they've been triaged, they'll have a milestone (other than Backlog), an assignee, and the flow::To Do label.

If you come across something that needs fixing:

  1. Create an issue describing the problem.
  2. If it's not obvious, justify how it relates to our persona and how it contributes to MAUI.
  3. Then:

    • If it's more urgent (has a higher impact on MAUI) than other things you've been assigned, assign it to yourself to work on later the same week:

      /milestone %<current milestone>
      /label ~"flow::To Do"
      /reassign @<yourself>
      /cc @DouweM
      
    • If it's urgent, but you're not sure who should work on it, assign it to Douwe to triage:

      /milestone %<current milestone>
      /label ~"flow::Triage"
      /reassign @DouweM
      
    • If it's not urgent or you're unsure whether it's something we should do at all, assign it to Danielle to triage:

      /milestone %“Backlog" or %<next milestone>
      /label ~"flow::Triage"
      /reassign @dmor
      

Useful issue boards

Release Process

Schedule

Meltano currently follows a weekly release schedule on Mondays.

Release Date Release Owner Speedrun Owner Shadow
2019-10-07 Ben H. Ben H.  
2019-10-14 Derek K. Derek K.  
2019-10-21 Ben H. Ben H.  
2019-10-28 Micael B. Micael B.  
2019-11-04 Derek K. Derek K.  
2019-11-11 Yannis R. Yannis R.  
2019-11-18 Douwe M. Douwe M.  
2019-11-25 Yannis R. Yannis R.  
2019-12-02 Derek K. Derek K.  
2019-12-09 Micael B. Micael B.  
2019-12-16 Ben H. Ben H.  
2019-12-23 Yannis R. Yannis R.  
2019-12-30 Ben H. Ben H.  

If you are unable to cover an assigned week, please find someone to cover for you and submit an MR to this page with the new owner.

Versioning

Meltano uses semver as its version number scheme.

Prerequisites

Ensure you have the latest master branch locally before continuing.

git fetch origin

Workflow

Meltano uses tags to create its artifacts. Pushing a new tag to the repository will publish it as docker images and a PyPI package.

  1. Meltano has a number of dependencies for the release toolchain that are required when performing a release. If you haven't already, please navigate to your meltano installation and run the following command to install all development dependencies:

    # activate your virtualenv
    source ./venv/bin/activate
    
    # pip3 install all the development dependencies
    pip3 install .[dev]
    
  2. Execute the commands below:

    # create and checkout the `release-next` branch from `origin/master`
    git checkout -B release-next origin/master
    
    # view changelog (verify changes made match changes logged)
    changelog view
    
    # after the changelog has been validated, tag the release
    make release
    
    # ensure the tag once the tag has been created, check the version we just bumped to: e.g. `0.22.0` => `0.23.0`.
    git describe --tags --abbrev=0
    
    # push the tag upstream to trigger the release pipeline
    git push origin $(git describe --tags --abbrev=0)
    
    # push the release branch to merge the new version, then create a merge request
    git push origin release-next
    

Tip: Releasing a hotfix? You can use make type=patch release to force a patch release. This is useful when we need to release hotfixes.

  1. Create a merge request from release-next targeting master and use the release template.
  2. Add the pipeline link (the one that does the actual deployment) to the merge request. Go to the commit's pipelines tab and select the one that has the publish stage.
  3. Make sure to check delete the source branch when the changes are merged.
  4. When the publish pipeline succeeds, the release is publicly available on PyPI.
  5. Follow the Digital Ocean publish process
  6. If a non-patch release, record and distribute the Speedrun Video:

Demo Day

For each demo day, we need to ensure that the following process is followed:

Demo Day: Setup

  1. Document list of features to demo
  2. Document order of people demoing
  3. Ensure every person demoing has proper display size (i.e., font sizes, zoomed in enough, etc.)
    • Font size at least 20px
    • Browser zoom at least 125%

Demo Day: Workflow

  1. Record each meeting with Zoom
  2. Generate list of timestamps for each featured demo
  3. Generate list of features (from Setup section) paired with timestamps
  4. Upload recording to YouTube
  5. Add features + timestamps to YouTube description

Speedruns

As part of Meltano's Release process, speedruns allow the team to ensure that every release is stable.

🏆 Current Record: 1:35 (Ben Hong)

Tip: Remember to leave each screen up for at least 2 seconds so users have a chance to notice that something actually happened.

Requirements

  1. Keep keystrokes to a minimum (ideally zero)
  2. You do not have to explain every step
  3. Do not need to stop and explain new features
  4. Time starts from when the webapp is loaded on the browser
  5. Make sure to pause between screens so user has a chance to register that a change is happening

Workflow

  1. Introduce Meltano to new users:
Meltano is an open source data toolkit
that makes it easy to go from data source to dashboard.

For more information, check us out at meltano.com!
  1. Check that Meltano does not exist on your machine
meltano --version
# command not found: meltano
  1. Install Meltano on your machine using distributed version
pip3 install meltano
  1. Check Meltano version matches latest release
meltano --version
  1. Create a new Meltano project
meltano init speedrun-workflow
  1. Change directory into your new project
cd speedrun-workflow
  1. Start Meltano application
meltano ui
  1. Assuming there are no conflicts on the port, you can now open your Meltano instance at http://localhost:5000.

  2. Run through tap-gitlab + target-postgres workflow as quickly as possible with some narration, but don't pause mid-action to explain something.

DigitalOcean Marketplace

Meltano is deployed and available as a DigitalOcean Marketplace 1-Click install.

Find the snapshot name

Tip: The digitalocean_marketplace job is only available on pipelines running off tags.

  1. The snapshot string should be available under meltano-<timestamp> on DigitalOcean, which you will find at the bottom of the digitalocean_marketplace job. Take note of this snapshot string as you'll use it in the next step.

Update the DigitalOcean listing

Then, head to the DigitalOcean vendor portal at https://marketplace.digitalocean.com/vendorportal to edit the Meltano listing.

Tip: Don't see the Meltano listing? You'll have to be granted access to the DigitalOcean vendor portal. Please ask your manager for access.

  1. Once inside the listing, update the following entries:
    • Version to the latest Meltano version
    • System Image to the new image (match the aforementioned snapshot string)
    • Meltano Package Version inside the Software Included Entry
  2. Submit it for review to finish the process.