Google is using machine learning to improve the quality of Duo calls – The Verge
Google has rolled out a new technology to improve audio quality in Duo calls when the service can’t maintain a steady connection called WaveNetEQ. It’s based on technology from Google’s DeepMind division that aims to replace audio jitter with artificial noise that sounds just like human speech, generated using machine learning.
If you’ve ever made a call over the internet, chances are you’ve experienced audio jitter. It happens when packets of audio data sent as part of the call get lost along the way or otherwise arrive late or in the wrong order. Google says that 99 percent of Duo calls experience packet loss: 20 percent of these lose over 3 percent of their audio, and 10 percent lose over 8 percent. That’s a lot of audio to replace.
Every calling app has to deal with this packet loss somehow, but Google says that these packet loss concealment (PLC) processes can struggle to fill gaps of 60ms or more without sounding robotic or repetitive. WaveNetEQ’s solution is based on DeepMind’s neural network technology, and it has been trained on data from over 100 speakers in 48 different languages.
Here are a few audio samples from Google comparing WaveNetEQ against NetEQ, a commonly used PLC technology. Here’s how it sounds when it’s trying to replace 60ms of packet loss:
Here’s a comparison when a call is experiencing packet loss of 120ms:
There’s a limit to how much audio the system can replace, though. Google’s tech is designed to replace short sounds, rather than whole words. So after 120ms, it fades out and produces silence. Google says it evaluated the system to make sure it wasn’t introducing any significant new sounds. Plus, all of the processing also needs to happen on-device since Google Duo calls are end-to-end encrypted by default. Once the call’s real audio resumes, WaveNetEQ will seamlessly fade back to reality.
It’s a neat little bit of technology that should make calls that much bit easier to understand when the internet fails them. The technology is already available for Duo calls made on Pixel 4 phones, thanks to the handset’s December feature drop, and Google says it’s in the process of rolling it out to other unnamed handsets.