Tracing calls through backend components
The goal of the project was to implement distributed tracing across the main backend components of Jitsi. In short, distributed tracing is a technique for tracing requests across distributed components by generating events in each component and stitching them together using identifiers. The idea was not to fully integrate distributed tracing into the whole system but to have end-to-end traces for common flows that can be used as a base for future tracing.
The following flows are now traced:
- Creating a conference
- Joining a conference, which also includes inviting JVB
- Leaving or ending a conference
- Inviting a Jibri recorder
For example, when a client joins a conference, it first joins the MUC room, then Prosody notifies Jicofo, who allocates a bridge and initiates a jingle session. Tracing this involved injecting code in the Meet client, Prosody, Jicofo, and JVB. The trace looks like this:

There were two main challenges while implementing this. The first challenge was understanding the whole system. What does each component do? How does it interact with the other components? When does it do so? As the amount of code required for this to work was small, the difficult part was actually finding exactly where to put it.
The second challenge was finding a way of propagating context. It was not enough to record the main steps of each component, but we also needed a way to stitch them together. As all communication is done through Prosody, I implemented an XMPP extension inspired by the W3C standard: https://github.com/jitsi/jitsi-xmpp-extensions/pull/146.
To enable tracing, we need a tracing backend (Tempo), an OpenTelemetry collector (Alloy), and a frontend (Grafana). An example of how to set everything up can be found in docker-jitsi-meet: https://github.com/jitsi/docker-jitsi-meet/pull/2303.
For next steps, this should be enabled in production to start gathering real data and understand which information is more valuable. There are also many other flows that could be traced but fall outside the scope of this project: inviting a transcriber, inviting SIP participants (Jigasi), changes in room metadata, etc.
Below is a list of all the PRs that were developed as part of the project.
- https://github.com/jitsi/jicofo/pull/1294
- https://github.com/jitsi/jicoco/pull/240
- https://github.com/jitsi/jitsi-videobridge/pull/2433
- https://github.com/jitsi/lib-jitsi-meet/pull/3067
- https://github.com/jitsi/jitsi-meet/pull/17621
- https://github.com/jitsi/jitsi-xmpp-extensions/pull/147
- https://github.com/jitsi/jicofo/pull/1301
- https://github.com/jitsi/jibri/pull/633
- https://github.com/jitsi/docker-jitsi-meet/pull/2303.
