H.264 encoding support for TurboVNC
High-speed, 3D-friendly, TightVNC-compatible remote desktop software
Brought to you by:
dcommander
Menu
▾
▴
#10 H.264 encoding support for TurboVNC
Tasks:
(1) Low-level benchmarking using the TurboVNC Benchmark tools to ascertain whether H.264 will produce generally better compression than "Turbo" encoding and, if not, which types of workloads it works well for. Also investigate whether the performance of libx264 is sufficient to supply interactive frame rates at the resolutions that TurboVNC users need.
(2) If (1) proves fruitful, implement H.264 encoding and decoding. This will require designing a new RFB encoding for H.264 and officially registering it as part of the RFB protocol spec (probably in cooperation with TigerVNC, since they need this too.) The server and native client can be extended to use libx264 (and whatever the equivalent is on the decoder side-- libavcodec?) directly. The Java client will be somewhat trickier, as I will likely need to implement a JNI wrapper and then deploy that library along with the next-gen Linux/OS X viewer (which is a hybrid Java/native solution.) It might also be possible to use h264j, if it performs well enough.
(3) Debugging and benchmarking the end-to-end solution.
Ticket moved from /p/virtualgl/feature-requests/30/
Can't be converted:
Completed report is here:
http://www.turbovnc.org/About/H264
As predicted, for certain isolated types of workloads (games mostly), when comparing the same perceptual image quality levels between H.264 and TurboVNC, H.264 can be shown to generate better compression. However, for most workloads, TurboVNC does a much better job of compression than H.264. H.264's weakness is that it requires that a new frame be generated every time the remote framebuffer changes, even if the change is only 1 pixel. These new frames can be encoded relative to a reference frame, so not all pixels need be sent with every frame, but H.264 is definitely not as efficient as TurboVNC when encoding small inter-frame differences. Thus, it can only really work well with full-screen applications that update most of the pixels on the screen with each frame (i.e. games.)
Even though H.264 can compress better with those isolated workloads, the open source implementation (x264) is still way to slow to be useful. It is incapable of generating data fast enough to fill even a slow WAN.
I can't say what the cause might be as the write-up seems solid, but from practical experience the conclusions drawn from that analysis seem simply wrong.
Using VNC (client makes only marginal difference) is frustrating and a pain over a low-bandwidth DSL connection (approx. 1 MBit/s) while it works comfortably using Xpra (after disabling lossless refresh), which uses H.264.
It is usable (much better than VNC, though still not nice to use) even when connecting to a Cubieboard (ARM-based, Cortex-A7).
Obviously a big VNC server has other requirements, as well as users needing accuracy and thus lossless refresh, but for personal use I haven't found a solution (VNC, x2go, RDP - TeamViewer maybe) that seems even just competitive to Xpra in real-use comfort for tasks like web browsing remotely, and maybe it would be worth checking what causes the discrepency with this report.
This SF tracker item is quite old and doesn't reflect the current thinking regarding the H.264 feature. It is still under active investigation:
https://github.com/TurboVNC/turbovnc/issues/19
There are definitely some workloads that benefit from H.264. There are others that don't. The report attempted to compare performance based on some assessment (albeit probably a weak one) of image quality equivalence, and without knowing the specifics of how you performed your comparisons, I cannot speak to your findings. You might not be comparing apples to apples. Xpra is using a different protocol than we are, and whereas our RFB implementation includes extensions that provide better performance on high-latency networks, it's entirely possible that Xpra's protocol does a better job of flow control on extremely low-bandwidth links. Or it may simply be that their protocol is a better fit for your specific workload. Benchmarking is something of a dark art, and you always have to be very careful when making any kind of general assumption based on it. Really, a single benchmark cannot tell you much outside of the specific scope in which it was run.
That being said, I haven't given up on H.264. I have only given up on a software-based H.264 solution, since that is definitely too compute-intensive to be of use in TurboVNC. What I'm looking at now is using NVENC on the server, but there are a lot of problems inherent in that, as described in the GitHub issue. Ultimately, this may have to be deferred until Wayland becomes more of a thing, since Wayland would make the use of a frame-based codec like H.264 much more straightforward. Note that Xpra, by virtue of using a window-based protocol instead of a screen-based protocol like RFB, also has an easier time leveraging H.264 than any VNC solution would. VNC is, ultimately, a remote desktop solution rather than a remote application solution, and the way X11 interacts with the remote framebuffer makes frame-based codecs an awkward fit at best.