FOSDEM 2025 – in English
by Vasil KolevWe did FOSDEM 2025.
I think this was the event where we put the most effort into by the most people. It went very well (or, well, with less problems than expected)
Things started at the previous edition, where Martijn Braam came to see us at the NOC and said “I was bored at one point and I made an audio mixer with a relatively simple board and a Teensy 4 as a controller”…
(also from his blog, our version of the mixer and the Ethernet switch)
So a new design of the box began (unofficially version 2.5). Version 2 was a 4-port USB network card with a switch in it, a capture card based on the MS2131 chip (with built-in loop-out) and a USB hub to hook them up to. Everything else was done from a laptop that we used as an encoder, to show status and all that sort of stuff.
The plan for the new design was:
– to remove the laptop
– to put a screen on the box
– to save on any other hardware, respectively
– to build an audio mixer
– to have a way to charge the microphone batteries on the go
– (quite brazenly, bit it failed) to build a receiver for the microphones in the mixer
Accordingly, Martijn and Angel from our side started designing the boards.
(this is where I realized how much I hate hardware)
I don’t know if I can tell you everything that happened within the year. The easiest way is to look at the history of the video repo.
A few interesting points:
cursor.c, a small thing that, when preloaded, makes every use of SDL hide the cursor. Because otherwise, if you output to the HDMI port of the computer with ffmpeg, at least in Debian there is a cursor in the upper left corner (writing and testing the whole thing took less time than rebuilding ffmpeg, so that’s why it’s like this);
Assembly instructions, along with a plan for a production line/workshop for assembling many boxes by many people;
Martijn made a very nice video about the box, and with Angel they presented it in one of the FOSDEM talks, the recording is already uploaded.
For the main computer inside the box we chose a Radxa x4. Our requirements were that it be x86-64, have an Intel GPU (because they have mainline support for hardware encoding), and be affordable/available. There is a very useful site by Martijn, hackerboards.com, where you can search for any computer you need. Initially we tested with Raxda X2L, but it took up a lot of space, and there was a problem with the power supply (fixed in newer versions, but still). In the end, it turned out that for a similar amount of money we could get 110 X4s, with built-in eMMC flash and all sorts of extras (including bluetooth/wifi, which we didn’t use), and we settled on them.
I’ll spend a little more time on the history of the fans, because there were a lot of questions about why they sound like that. Without going into details that I don’t understand, the situation is as follows:
– on the power board we have a chip that can control the fans, according to the documentation either with a PWM value or based on RPM;
– the RPM-based control doesn’t work. After I wrote an implementation that followed the datasheet verbatim and didn’t do anything, I went to see the source for the same thing in the Linux kernel, and it turned out that they don’t use it based on RPM there either;
– due to some details with the wiring, the chip can support one byte for PWM, but in practice there is only a difference at 3-4, with all the others it either stalls or is at maximum speed;
– Although we bought the same, nice fans, different PWM values have different effects on them. Also, different temperature/humidity in the room, moon phase, etc. also have different effects. Which leads to the fact that sometimes the fans stop and their value has to be increased to spin, which leads to whining;
– Which whine is probably heard in a decent part of the lectures;
– Since we have a 2 mm metal plate on the bottom of the case that covers the whole case and we use it as a radiator, there was a discussion about whether to stop them altogether (because during the tests we were not able to overheat it), but we didn’t feel like taking the risk.
Unfortunately, the sound of these fans will be heard in many of the lectures. If someone comes up with a filter to remove it, please send it to us…
The metal plate on the bottom turned out to be an incredibly good idea. It all came from the fact that if we had followed the old principle of fastening, we would have had to design many different and small parts to be 3D-printed, which we can somehow attach to the few holes of the case so they can hold all the boards. The idea of making one big board was dropped very quickly, because two of the boards were not made by us at all (Radxa x4 and HDMI capture card), and it would have made the process of debugging and making new revisions very difficult. At one point we said to ourselves “why not just make one piece of iron and screw it in”, which almost as a side effect turned out to be an incredibly effective radiator (before that, the Radxa x4 would overheat and crash within 10 minutes, and the Radxa X2L would throttle its processor to 500MHz, and we were considering all sorts of cooling options).
In October and November, we had finished the design and started ordering all the parts. The good people at MinoLab provided us with a place to get all the pieces, so we could easily load and unload and assemble. We built 70 boxes out of parts we had by organizing a sweatshop at Christmas to assemble them, with an additional step shortly after the new year to install the screens.
(the original plan was to assemble at initLab, but it would have been difficult for us to fit there, and I can’t imagine how we would have carried the boxes of stuff up and down those stairs. That’s why most of the testing and various other development activities were at initLab)
I don’t know if I can explain what an incredible experience the assembly was. In the days between Christmas and New Year and then the first weekend of January, 10-15 people made 70 working, installed and tested boxes from a pile of parts and gadgets we had. We estimated that FOSDEM could happen with that many (60 for the rooms and 10 spare). If we had delayed another week, we could have built more, but the event was getting too close, and it would have been complicated to use the assembly site.
In the end, on January 7th we managed to send everything to Belgium, and it arrived there a week later.
This was very much on the verge, and we were about to activate the backup plan (which was to rent a truck and drive it there ourselves, and in extreme cases we could assemble some of the boxes on the way, in the back of the truck)
To be able to have something working, we removed the following functionalities:
– loop-out through the Radxa HDMI ports, instead of through the capture card. This way we could show something different on the screen while nothing was on. The main reason it wasn’t there was that we couldn’t find 110 pcs of short microHDMI-HDMI cables;
– To avoid one cable and carry the audio between the two boxes over the network. I ran a proof of concept with AES67, but there was no time left to make it production ready;
– The above was also stopped by the fact that due to some synchronization problem the audio mixer crashes if we use USB audio on it. There are several ideas on how to fix it, but relatively late we found that the cause was a change that makes it work at 48KHz instead of 44.1KHz, and that leads to USB desynchronization and reading the wrong memory;
– Again due to the lack of one cable and time we didn’t finish the functionality to be able to play sound from the lecturer’s presentation. Some rooms don’t have sound, so this wouldn’t be useful, but for others it would be, and more and more lecturers want to play sound these days;
– One of the USB ports on the box had to go directly to the Radxa, so we could do some interesting things. We didn’t have a suitable cable for it, but it was the better thing to do, because this is a live backdoor into the box and we haven’t figured out how to prevent someone from sticking a keyboard in and starting to make trouble…
But, we had all the functionality from previous years, along with an audio mixer that was built in (one less thing to carry), and whose levels we could monitor in real time and change when necessary (without having to send someone in the room). As an added bonus, random people couldn’t change the sound settings, because the box doesn’t have any controls :). In general, the audio mixer and things around it deserve a separate post, which Albert (who wrote most of the code) wrote :)
In the last week I managed to fix a few more things, like having a 480p stream (not only for people on crappy DSL, but for people on Wi-Fi at the university itself, who couldn’t get into the rooms).
So, we started FOSDEM 2025 with hardware that was prepared 3 weeks ago, with some problems and not-tested-enough. If we had as many problems as I expected, I would use this whole story as an example for my children of what not to do.
About 30 of us went from Bulgaria, and we were the majority of the video team. Despite all my plans to gather volunteers from other places, it didn’t work out – I managed to do an online briefing exactly one week before FOSDEM, and I couldn’t do it any earlier (due to the assembly finishing in early January, after which I got so sick that I’m still coughing). My hope is that after we assemble the rest of the boxes, we can see if various hackerspaces won’t like them (because you can do some interesting things with them) and that way we would gather more people who want to play with them and help at FOSDEM.
We had surprisingly few problems.
Our main problem came from the famous problem with Voctomix memory leaks – we had planned to migrate to Voc2mix, which was released in the summer, but we didn’t have time to test and integrate it (it would have given us more options, such as more than 2 audio channels and a way to carry backup audio). This led to holes in some lectures, and for next year there are a few ideas on how to fix it.
The other more common problem (3-4 times) was that a certain input signal in the capture card manages to screw it up and stop its loop out. It’s not clear why, we don’t have the firmware source and we can’t find documentation for the chip (Macrosilicon MS2131, if anyone has it, I won’t refuse).
(there is an assumption that certain macbook M1s do this with the capture card going crazy, eventually we will ban them)
Otherwise, a couple of boards died, the power went out in one room, nothing flooded, despite delays due to the network being deployed on Friday, we were ready at a relatively normal time, so we could go to bed on time, and the tear-down was also completed calmly enough so that everyone could eat normally.
We even managed to rewrite the way to visualize the sound levels (we separated the two channels), by moving the entire calculation of the levels to a separate machine and database (to free up processor time for the video mixers).
And I made the mistake on Saturday night of going to the staff dinner, which dragged on for a long time and, together with the fatigue from everything else, led to the fact that I had to sit down and take a nap somewhere on Sunday afternoon. Either I’ve lost my form, or I just haven’t been able to recover from my illness…
We also lent 2 boxes and a mixing laptop to FOSSASIA, there’s a chance they’ll use them soon :)
Some of the new things this year – we’ve been putting fixed phones on the VOCs and in various other places, to see if they’d help. We also hooked up some SIP mobile phones to the same PBX, for those who wanted them. The experiment was a success, they were used quite a bit (and there are already people who want them in more places). They might even replace walkie-talkies to some extent (which I still can’t get used to).
I can’t say much about FOSDEM itself – it was still very large and full of people, but as far as I know this year they’ve kept the rooms from getting crowded much better. For me, the most interesting thing was the separate “junior” track, for kids to go to workshops and be shown useful things. This is something that seems to be missing here in Bulgaria (hackconf used to be student-oriented, but it’s gone now), and it would be nice if it re-appeared.
And to end with a big thank you to all the volunteers who helped and didn’t strangle me (because the whole exercise wasn’t easy at all) – without you this wouldn’t have had a chance to happen, so thank you very much. And I’d be happy if you joined again next year :)