Jupyter Guide

Maintained by:

Common python DS/ML libraries (pandas, scikit-learn, sci-py, etc.)
Natively connected to Snowflake using your dbt credentials. No login required!
Git functionality: push and pull to Gitlab repos natively within JupyterLab (requires ssh credentials)
Run any python file or notebook on your computer or in a Gitlab repo; the files do not have to be in the docker container
Need a feature you use but don't see? Let us know on #bt-data-science and we can add it!

Getting Started

You have two options when setting up jupyter via the data-science project. Choose from one of the following:

Full install (Recommended): Installs all libraries defined by the Pipfile, plus a complete anaconda install.
Lightweight install: installs only the libraries defined by the Pipfile and should be used if you already have a python environment on your local machine that you would like to use as the base image.

Installation Instructions

Clone the repo to your local machine git clone git@gitlab.com:gitlab-data/data-science.git
Run cd data-science
Based on which version you would like to install, run one of the following:
- For full install: run make setup-jupyter-local
- For lightweight install: run make setup-jupyter-local-no-conda
Run make jupyter-local
Jupyter should launch automatically. If it does not:
1. First make sure that Google Chrome is your default browser (go to "System Preferences", click "General" and choose Google Chrome from dropdown menu in section "Default web browser").
2. Then, in Chrome, copy paste the url and token found in terminal once the docker image creates. It should look something like http://127.0.0.1:8888/lab?token=5c7f7da79f4a0968501f087f3c79ee4dd8bd7a63e0f088a8. The token will change each time you spin up the docker container.

Validation of installation

Before you proceed further, validate if your Python and pip (package installer for Python) are in correct paths. If these are not in correct locations you may not be able to work with the most up to date packages.

To check if everything is correct open terminal and run command

which python

It lists location of your default Python. It should be in /Users/your_user_name/anaconda3/bin

And command

which pip

It lists location of your default pip. It should be in /Users/your_user_name/anaconda3/bin/pip

Connecting to Snowflake

Make sure you have setup a {User}/.dbt/profiles.yml file which does not include your password. You can use the example provide here as reference
Run through the auth_example notebook in the repo to confirm that you have configured everything successfully. The first time you run it you will get a browser redirect to authenticate your snowflake credientials via Okta. After that, if you run the notebook again you should be able to query data from Snowflake.
If you get an error then likely Snowflake is not properly configured on your machine. Please refer to the Snowflake and dbt sections of the Data Onboarding Issue. It is likely that your .dbt/profiles.yml is not setup correctly.

Mounting a local directory

By default, the local install will use the data-science folder as the root directory for jupyter. This is not terribly useful when all your code, data, and notebooks are in other locations on your computer. To change, this you will need to create and modify a jupyter notebook config file:

Run jupyter-lab --generate-config. This creates the file /Users/{user}/.jupyter/jupyter_lab_config.py
Browse to the file location and open it in an Editor
Search for the following line in the file: #c.ServerApp.root_dir = '' and replace with c.ServerApp.root_dir = '/the/path/to/other/folder/'. If unsure, set the value to your repo directory (i.e. c.ServerApp.root_dir = '/Users/{user}/repos'). Make sure you remove the # at the beginning of the line.
Make sure you use forward slashes in your path. Backslashes could be used if placed in double quotes, even if folder name contains spaces as such as \yourUserName\Any Folder\More Folders\
Rerun make jupyter-local from the data-science directory and your root directory should now be changed to what you specified above.

Increasing Docker Memory Allocation

By default, docker will allocate 2GB of memory to run containers. This is likely not enough RAM to work with jupyter and python, as data is held in-memory. It is recommended you increase the docker memory allocation to avoid out-of-memory errors.

Open Docker dashboard.
Click on the gear icon in the upper right to show settings.
Under "Resources" allocate additional memory to be used by Docker. 8GB is recommended but you may have to increase it futher if working with large datasets.
Restart Docker.

Setting Up Jupyter Extensions

The data-science repo comes with many useful Jupyter Lab extensions pre-installed, including git, variable inspector, collapsible headings, execute time, and system monitor.
To get the most out of these (and to avoid having to configure them every time you run the container), create the following file: /Users/{user}/.jupyter/lab/user-settings/@jupyterlab/notebook-extension/tracker.jupyterlab-settings
Within that file, paste the following and save:

{
    "codeCellConfig": {
        "codeFolding": true,
        "lineNumbers": true,
    },
    
    "recordTiming": true,
    
}

Interesting libraries included

Data/Model Analysis

ELI5
QuickDA

Visualisation tools:

ML libraries

SKlearn
Tensorflow
Torch
Py-earth (linear and logistic regression)
Prophet (time series)
Autots (time series)
XGBoost (powerful black-box method)

Easy concurrency

Modin
Dask (must be self-installed)

GPU speedup

PlainML