GSoC 2017 Progress Report
We are already a month into the coding period of the Google Summer of Code 2017 program, and we are excited to report on our students’ progress so far. This year we are fortunate to be working with four great students, and everyone has already produced code samples (some of them merged, some of them not yet). All four projects are going according to plan, and all students have passed the first evaluation. Below is a short summary of each project, including links to each student’s own more detailed report.
Han Gyeol Oh has been working on a “micro mode” for the Jitsi Meet Electron wrapper: a separate window, which appears on top of everything else when the user switches focus to another window. After exploring a couple of approaches which turned out to be dead-ends, he has now found a viable solution and has a working prototype. You can read more about this on Han’s blog.
Nik Vaessen is working on adding speech to text functionality in Jitsi Meet, by using a modified version of jigasi. We chose to move away from his previous implementation in the JavaScript client, because of the additional flexibility that we get by having the raw audio streams in java (besides, jigasi already knows how to connect to a conference and decode all audio streams). The project will eventually provide transcribed audio in real-time in a Jitsi Meet call, as well as a final transcription of the whole conference. Nik already has a prototype which uses the Google Cloud Speech API (but we are designing this to be extensible, and planning to implement other backends too). You can read Nik’s description of the work so far on his GSoC blog.
Julian Chukwu is working on bringing in send-side bandwidth estimation based on transport-cc to libjitsi and Jitsi Videobridge, which is to eventually replace the REMB-based estimation. This is now the preferred method for bandwidth estimation in webrtc.org, and may eventually be the only way supported by Google Chrome. This is a complex project, which will be merged in multiple steps. The first step is to port the remote bitrate estimator based on abs-send-time from webrtc.org, and we have an initial PR for this. Read Julian’s report on his blog here.
Chunxu Tang has been working on adding face recognition and tracking features to Jitsi Meet. This will initially be used to remind the user to adjust their camera or position — when the system detects that a user’s face is off-center it will display an interface prompting the user to fix it. But this technology has other potential uses, too, for example automatically adjusting the crop of the rendered remote video in the thumbnails. Chunxu has a working prototype which you might be able to test here (but keep in mind that this is an active development environment and it might not always be in a working state), and he has written a blog post summarizing his progress so far.
We are really thankful for having these great students and the opportunity for participating in Google Summer of Code once again!