The Real-time Editing of Issue Descriptions (REID) SEG is a Single-Engineer Group within our Incubation Engineering Department.
This group is focused on Real-time collaborative editing of issue descriptions.
The goal is to have real-time collaborative editing of issue descriptions in order to turn issues into a fully real-time collaborative text-editing experience.
There are additional ideas within the Real-time collaboration Epic, however the goal of this SEG is the minimal functionality described above.
| Date | Topic | Description | Video | |
|---|---|---|---|---|
| 14 | 2022-12-21 | An ActionCable based Y.js WebSocket and BroadcastChannel provider | Editor bindings | https://www.youtube.com/watch?v=g–9JLOMXeQ |
| 13 | 2022-12-12 | Rust available in gdk and base images | Adding Rust | https://www.youtube.com/watch?v=xmj7SzYIo1o |
| 12 | 2022-11-17 | Adding Rust to GitLab Pt.2 | Adding Rust to gdk, CI, etc. | https://www.youtube.com/watch?v=K_vidq8VdAs |
| 11 | 2022-11-02 | Adding Rust to GitLab | Shipping a gem with a native extension is tricky and extends the scope of the project. | https://www.youtube.com/watch?v=IQS81adVWWg |
| 10 | 2022-10-19 | Message protocol, Editor bindings, Awareness Pt. 2 | Two busy weeks, and lots of complexity. | https://www.youtube.com/watch?v=wJnvhW4eBKQ |
| 9 | 2022-10-06 | Real-time demo (y-rb usage) | Celebrate the first table y-rb multip-platform release with a sandbox demo. |
https://www.youtube.com/watch?v=1gfYfUMh3dU |
| 8 | 2022-09-16 | Change of plans | Awareness feature not to ship to current issue sidebar and y-rb multi-platform releases. | https://www.youtube.com/watch?v=x47YnULTiro |
| 7 | 2022-08-18 | Ruby+Rust | Awareness feature about to ship, and a little intro to how Ruby+Rust bindings work. | https://www.youtube.com/watch?v=ng20jSo2TIs |
| 6 | 2022-08-05 | Progress & Roadmap | Progress on the work to ship the Awareness widget, and an outlook for what is coming next. | https://www.youtube.com/watch?v=ZSdpj_YxsCE |
| 5 | 2022-07-22 | JTBD and Progress Update | JTBD (Jobs to be done) and Iteration on the Awareness UI | https://www.youtube.com/watch?v=S0s-s45zVv8&list=PL05JrBw4t0KpPmRsaVaDOoWyIp1iKacZo&index=2 |
| 4 | 2022-07-06 | Awareness UI/UX and a CRDT database | Where and how to add the presence indicators on the issue page. A database for CRDTs is usually not necessary, but can become interesting for cases where we need a simple way to fetch the initial document state. | https://www.youtube.com/watch?v=B094U_DvL-s&list=PL05JrBw4t0KpPmRsaVaDOoWyIp1iKacZo&index=3 |
| 3 | 2022-06-22 | The Awareness feature | Technical challenges on rolling out real-time features at scale. Memory and resource concerns and how to mitigate them. | https://www.youtube.com/watch?v=ZPH9tvcPDfc&list=PL05JrBw4t0KpPmRsaVaDOoWyIp1iKacZo&index=4 |
| 2 | 2022-06-09 | Tech design & terminology | A proposal for real-time collboration at GitLab. The presented tech design aims to be a comprehensive guide and peer-reviewed document | https://www.youtube.com/watch?v=crzCYkMk5XQ&list=PL05JrBw4t0KpPmRsaVaDOoWyIp1iKacZo&index=5 |
| 1 | 2022-05-25 | First update | The y-rb Ruby gem and how we are going to use it to bring real-time collaboration to GitLab. | https://www.youtube.com/watch?v=5mTS64Y-rX0&list=PL05JrBw4t0KpPmRsaVaDOoWyIp1iKacZo |
Make real-time collaboration on GitLab work for everything.
A user should never have to leave the GitLab application for tasks like pair programming, collaborating on isssues, merge requests, tech designs and RFCs.
GitLabs software stack enables real-time collaboration via Websockets, in particular GraphQL subscriptions. Real-time comes with certain complexities when users want to collaboratively edit text, in particular rich text. In order to allow multiple parties to edit the same text concurrently, we must ensure a conflict-free replication mode between all participating clients. There are several ways to achieve this, but we utilize CRDTs. CRDTs allow us to concurrently edit state from 1, 2, … n clients and eventually end up with a consistent representation of the document on all clients.
Every participant in the editing process is a client (in comparison to Operational Transformation). The CRDT we picked (YATA) has a popular frontend implementation Y.js and we have created bindings for its Rust port y-crdt. This allows our Ruby on Rails backend to just act as one more client.
y-rb gemCurrent State: Published to RubyGems.
This Ruby gem is developed under the y-crdt umbrella (The y-crdt organization) and brings basic and complex shared types (Array, Map, Text, XML) to Ruby. The encoded updates and state vectors are a 100% compatible with Y.js and therefore state can be synced between the JavaScript frontend and the Ruby backend in a seamless way. This allows us to e.g. add, update, or remove labels via the GitLab UI but also with background jobs (some bot adding a label concurrently).
y-rb_redis gemCurrent State: Started
This Ruby gem is developed under the y-crdt umbrella (The y-crdt organization). It is a thin abstraction layer to allow persisting changesets to Redis. The updates produced by y-crdt documents are stored as easy to store and fetch elements in a list. This allows us to persist and restore documents efficiently.
y-rb_actioncable gemCurrent State: Started
The default ActionCable implementation has no delivery guarantees (we look for at-least-once), and no atomic broadcast mechanism. The order of messages sent should be the same when they are received.
Current State: Active development
Starting a collaborative editing session should be exactly the same as opening and editing an issue today. In case a second user starts to work on an issue, the backend will create a collaborative editing session users automatically and makes sure that all changes (deltas) are synced between all participating parties.
Awareness helps us to see who we are collaborating with and how. It helps us answering the following questions:
Current State: Started
To determine and replicate a change to the description field, we need to encode everything that a user does as a change event (delta). Y.js uses a binary encoded representation to optimize payloads, but essentially it is a list of operations. One example of how this looks like is the Quill Delta Format.
The GitLab UI does not rely on a uniform text editor but instead offeres a mix of textarea, tiptap/prosemirror, Web IDE, …, multiple editor modes (Raw, WYSISWYG), and post-edit transforms (Markdown parser → references). We need to build a transform layer to broadcast changes made to a text field in a conflict-free format, and reversly, apply incoming changes seamlessly in the editor.