I’ve been developing a VNC server for wlroots based compositors such as Sway and Wayfire. It’s called wayvnc and it’s accompanied by a new VNC server library called NeatVNC. It attaches to a running Wayland session, creates virtual input devices and exposes a single display via the RFB protocol. The project has provided me with interesting problems to work on for a few months now and there are many interesting problems left to solve.

It began when I was upgrading the kiosk-mode browser that’s used for the GUI that runs on the embedded devices at work. The old browser is based on Qt4 and WebKit and it turns out that one very useful feature in Qt4 is a VNC server. Alas, Qt4 is no longer maintained and Qt5 does not support running VNC and rendering to a display at the same time1. I considered just cross-compiling X11, but we have our own in-house Linux disto and adding X11 just for running an embedded browser seemed like a rather daunting task, given all the dependencies required. That’s when I decided to try Wayland. More specifically, I decided to try running Cog on Cage2.

Wayland is fresh, so there are fewer packages that need to be compiled and maintained. Just look at this script and then compare that with the dependencies for Sway and wlroots. It is also a lot simpler than X11, so it should be easier to understand and tailor to the exact requirements. But still, one thing that Wayland has been missing is VNC, so I thought: how hard can it be?

It turns out that it can be as hard as you want it to be. I.e. there is plenty of room for optimisation and this is what makes it interesting. For my task at work I found a fledgling implementation (wvnc by Josef Gajdusek) and adapted it to my use-case so that it worked well enough. I also switched from Cage to Sway because I didn’t want to re-implement the missing protocols in Cage. I could have left things at that, but for multiple reasons I found this problem interesting enough to continue working on it in my spare time. Here are some of the reason that made me want to work on it:

  • The developers working on Sway and wlroots were very helpful and generous and I wanted to contribute more to the project.
  • Figuring out ways to make it faster and leaner is rewarding on an intellectual level.
  • Other people seemed to be interested in this.

After doing some profiling I found that what was causing most CPU load was taking screenshots in the compositor, sending them to the VNC server and then checking for differences between the frames. The VNC server didn’t really know if anything had changed, so it just requested screenshots in a loop. I introduced a change to the wlr-screencopy-unstable-v1 standard which added a method by which a user could request to be sent new frames only when they changed from the last frame sent to that user, along with information about which area changed. Getting the changes merged was a fairly painless process.

One thing that I also found missing was a good open source VNC server library. There is libVNCServer, but sadly it just didn’t meet my requirements. I neither liked the license nor the API. An LGPL would have been good, but GPL on a library is a deal-breaker for me. The API allows users to reach far into internal data structures and this is even required if you want to run it asynchronously. This kind of design (or lack thereof) makes it far more difficult to make changes while still maintaining backwards compatibility. This wasn’t really the biggest deciding factor though. Really, I just wanted to have more control so that I could optimise more aggressively; e.g. to reduce copying I would have had to make some changes to the API. Eventually, I decided to create my own VNC server library. It lacks many features that are available in other VNC solutions and the encoding algorithms may be less efficient, but a lot of those features are legacy by now anyhow. At this point it works well enough and I’ve tried to design the API in such a way that it should not be affected by future improvements and additions.

I continued modifying wvnc but I realised that I was rewriting a lot of it in order to integrate NeatVNC, reduce copying, make it properly asynchronus, etc. So, I ended up starting my own project and gave it the working title wayvnc which I haven’t changed yet. The name is probably going to stick. I’ve learned not to get bogged down with trivial things such as names or which programming language to use. Getting it done is what’s important.

The last missing piece of the puzzle came when Josef Gajdusek submitted the wlr-virtual-pointer-unstable-v1 protocol. Up to that point, wvnc had required uinput to run with a pointer, which required root privileges. At that point, wayvnc was near the MVP stage and was only missing the virtual pointer. I added Josef’s virtual pointer protocol and told the Sway maintainers about it so that they could properly test the protocol’s implementation. Things moved pretty quickly after that. The protocol was merged, the RDP backend was removed and Sway 1.4 was released.

After this I got a lot of new users because Simon and Drew were generous enough to mention the project in the release notes and point people towards it on other media. For this, I am grateful. Having users is useful because they tend to tell you when something is wrong with the software. Many bugs have been fixed by now; most of them related to keyboard handling3. Thank you all for testing! There are still many interesting issues to work on and being that I do have a day job and this is a hobby project I would be happy to receive contributions from others.

The work I’ve done on wayvnc would not have been possible without the help and support of the Sway contributors and users. Special thanks go to Simon Ser and Drew DeVault for your help and your patience, and Josef Gajdusek for pioneering work on the “VNC for Wayland” front. At last I would like to thank those who have contributed to my projects; either by contributing directly or reporting issues. Every little thing helps. Thank you.

  1. Besides, Qt WebEngine is also very buggy. Even though the bugs do get fixed, new ones are introduced in every release. Cross-compiling Qt5 is also a long and arduous process that requires many iterations. At one point I rented a 96 core VPS just to iterate faster on the build. ↩

  2. That’s not exactly how it went. I tried Weston first because it had RDP, but it was just too slow on the i.MX6 system that we have at work. I noticed that wlroots had a dmabuf protocol for sharing frames, so I decided to give that a try instead. Also, it was easier to understand and wlroots has its own standards that move much faster. It is easier to affect change in the wlroots ecosystem. ↩

  3. The RFB protocol (VNC) communicates using X Symbols which are keyboard layout agnostic. However, Wayland communicates using X Key Codes which depend on the keyboard layout, so the VNC server has to choose one layout and try to map symbols to keys. This is hard because they are not one-to-one and the mapping usually goes the other way. ↩