The next big thing in voice

Speaker A: Hello, I'm Michael Hainsworth. Taking a phone call in a crowded place like this is an exercise in frustration. Don't you just wish you could snap your fingers and make everything go silent? Dolby is doing for our phones what they did for the movie theater. Now Spatial audio isn't exactly new. Turn your head and your Apple AirPods will sound like the music still coming from the band over there. But Nokia and Dolby are working together with the Standards Group 3 GPP for an even more advanced technology called IVAAS. So everything from phone calls to video meetings sound like you're right in the room. Meet Dolby's Stefan Brun.

Speaker B: Immersive uh, voice and audio is such a big difference compared to today's mono uh experiences that are available on mobile telephony.

Speaker A: On this episode of the Network Effect, it's not just about a clear call. It's about fighting worker fatigue, improving our work from home experience and making the metaverse just as engaging as the real world. I understand why companies like Apple are big on spatial audio for music, but why do we need this technology just generally on our phones when we're having a conversation?

Speaker B: Well I think um. Well what we see in in general is uh a trend in uh consumer electronics and uh uh uh uh consumer experiences this that goes to um uh even better experiences both in terms of uh audio and uh video. Um this experience has not happened or this development has so far not really happened in telephony. So in telephony we are uh just taken, forgiven that uh the experience is mono. But of course users demand of course similar experiences even in uh uh mobile telephony as they experience for example when they go to uh a cinema experience, uh uh uh atmos, uh experiences and similar also with immersive video experiences. So what we are trying to do is really to fill a gap that we uh believe users are uh today experiencing and we see also that unless we close this gap, uh this gap will be um, growing and growing.

Speaker A: I'm fascinated by the idea that we could get a three dimensional sound out of just two stereo speakers or I suppose headphones. Can you give us a demo?

Speaker B: Yes, indeed. Um and I think what I'm uh going to demo you is uh, actually a new standard that uh we have defined in 3 gpp that standardizes uh, uh mobile uh uh systems like 4G, 5G, 6G. So there we have standardized a new codec that is called um, uh codec for immersive voice and audio services.

Speaker A: So what is a codec anyway? That's short for compression decompression. You compress the high quality audio into something that's easily transmittable and then on the other side you decompress it. We've been doing this for generations now. The big difference here of course is that IVAS has a super high quality when it's compressed and when it's decompressed you don't lose any of that quality like you do with other codecs.

Speaker B: I have a demo that uh, explains very well the uh, difference uh, between mono telephony and the promise of uh, new immersive telephony enable by IVAs. So, and to do so I can show you, uh, I have an app that uh, allows me to use an existing codec used for telephony which is called ebs. And uh, this codec offers only mona experience while there is then the new codec, the IVAS codec that offers immersive experience. And I will now uh, start the demo by putting on the um, headphones. And now I start essentially with mono voice with evs.

Speaker A: Hi everybody. Let's make this an efficient meeting. But uh, let's see.

Speaker B: First quiz, present what you will experience is that the sound is simply coming from inside your head, which is typical for mono voice if you listen to it over headphones. Well, I'm here, the audio coding team is present and the audio capture team. And now I switch to the new immersive voice, uh, and audio services codec. It's here as well. And the audio rendering team is here too. Wonderful.

Speaker A: We have everybody. Let me know if there's anything to add to the agenda that I previously.

Speaker B: So you hear a clear difference. I will once again switch it between EVS and IVAs. Well you could add a discussion point about the uh, IVAs codec evaluation. Yes, M. And uh, what do you think about adding a demo?

Speaker A: That's right. Good. That's a uh, demo of the very thing we are using now.

Speaker B: So I think this should be good. Quite impressive and uh, a good uh, illustration of what we believe is uh, immersive auspicious audio.

Speaker A: So is this going to be a technology we all just sort of take for granted in time, or will this be a differentiator for handset manufacturers and telecom providers? Are there going to be little stickers with the codecs on them so that you know who you go to? It sort of feels like the early days of the 80s and 90s of technology all over again. Or is everybody going to be using this technology?

Speaker B: Well I think at least initially it will clearly be a uh differentiator, uh ah. We uh know also that when we introduced um well 15 years ago, uh or 10 years ago, um logos uh uh, like HD voice or HD voice plus for wideband or super wideband telephony that there was um a logo program uh and still exists by gsma uh that certified uh uh certain uh uh devices and these devices uh so that they were really capable of providing their corresponding experiences. And uh. So uh. With this certification uh uh or when they um met these requirements they were allowed to have these logos and the same we expect also for um immersive voice and audio services there may be a logo that uh uh there will be of course over time one can think that this will be just um. Well also immersive voice and audio services in telephone here will be a commercial service. However um immersive voice and also audio is still something uh uh with a much greater promise. There are many uh differentiation possibilities in terms of what uh uh. Well how you can capture the immersive voice, what you capture for example if you capture into a certain direction, uh how you render, uh if you for example are able to identify certain sound objects in your audio and then allow essentially uh uh the application to and in the end also the user to select what uh the user is interested to listen at. So there is a lot of differentiation possibilities and I think uh this will be more than just a commodity service.

Speaker A: It sounds to me like this would be a great technology in a uh video conferencing environment. A lot of people in one room, maybe a whole bunch of other people remotely. And the ability to tell what corner of the room somebody's voice came from would help develop a better understanding as to who's talking and creating a sense of clarity.

Speaker B: Yes indeed. This is at least one of the applications that we are seeing. But even starting at the lower level, even for regular telephony we see already a big boost in uh user experience if this is a spatial sound. But yes, if we now consider also conferencing with the possibility to hear different uh persons from different uh directions we see definitely a large uh benefit. We are also aware of studies uh for example from call centers that uh fatigue of uh personnel working at call centers uh have much less fatigue during their workday if they are able to hear uh or discern uh spatial directions of a uh person's talking. So this helps a lot uh against fatigue. It provides better experience and it makes, contributes definitely to clarity of the conversation.

Speaker A: What about the Metaverse?

Speaker B: Well Metaverse is of course something where we see uh, uh well essentially uh converging um virtual and augmented reality space. So here we definitely see also applications or a natural place of uh immersive uh voice in the metaverse. So assume for example that you have meetings uh where some of the participants or all participants just meet in a virtual meeting room. So um, they get the impression by the audio but then um, also by a corresponding video stream that they would be in the same virtual meeting room. In practice they would be of course 1000km apart from each other but still they would uh uh have the experience as if they are in the same room. And this is uh at least one of the applications where we see um how immersive um uh audio would be part of the metaverse.

Speaker A: It's been said nobody cares about audio until they can't hear it. IVAS promises to help handset makers and others initially differentiate themselves from the competition. But with 3GPP adoption it's a new standard that one day will be ubiquitous. We'll question how we ever took a call in mono in the first place. But before we get there, collaboration is key. Without the network effect of big brains working together for a common goal. Dolby might as well hang up the phone on the whole idea. But how do you set a standard? How did Nokia and Dolby collaborate and partner on this?

Speaker B: Well uh, I can say such uh kind of uh collaboration or partnership is done on at least two levels. The one level is the regular standardization uh interaction between Dolby and um Nokia. Uh so uh, we are both uh present in the 3Gpp standardization organization that standardizes mobile uh uh Systems like uh 4G, 5G. And uh this is some collaboration where we uh strive for getting the best um for the mobile industry so that we uh define early standards that uh provide a cutting edge experience and uh, enabling uh more businesses for the mobile industry. So this is on a very general level uh in such kind of standardization of course uh we are ah working according to common rules uh which uh we sometimes also call terms of reference which means that everybody abides by such rules. And uh, well uh, there is a process that makes sure that always the best technical solution with the uh best uh merit uh would be selected. So this is the one level. Another level of the collaboration was something uh specifically for the IVAS codec, uh was something that uh is called public collaboration. And this is a process uh particularly uh designed to standardize or develop a codec as complex uh uh and complicated as the IVAS codec. Um so where essentially a number of Partners not only Dolby and Nokia but also um uh nine other uh partners were part of the um uh development. There was ah also a set of terms of reference associated to this and um uh well all partners had the same goal of uh developing the uh best possible um technology uh to be standardized uh in the UH3GPP standardization body.

Speaker A: It sounds like what you're telling me is industry collaboration plays a critical role in bringing important audio innovations to business and consumers.

Speaker B: When we are setting ah a standard there are essentially two ways to set a standard and um it depends always um so to say on the scale. There's one example where a company would essentially bring um a known standard to the market. Um so this uh is also of course a possibility. Such a standard would be called uh proprietary standard. It means that uh the company who owns the standard um uh would have uh well would be the only provider of implementations for example of such a standard. This is a valid model uh as long as there is um uh a uh free and open market where also other uh competing companies would provide um competing standards. And uh the original provider of such a proprietary standard would still be forced all the time to innovate and uh maintain really a technical lead. The other way of um standardization is to uh work like in uh mobile telephony, uh work towards uh a collaborative standard. This means that uh simply the mere size of um uh what needs to be standardized is too big for a single company. And uh as said uh there uh the companies agree on that. There's a uh uh well mutual benefit if uh the companies develop jointly technology such joint uh development does not mean that uh uh companies are not competing against each other but they are competing against a commonly agreed set of rules.

Speaker A: Tell me about the role of bringing people together create these standardizations. Everybody's got their own set of goals uh in mind. What was the biggest challenge you had to overcome when bringing all these people together from different industry groups and organizations to help create the standard that we see today. For this new uh codec there is

Speaker B: first some initial ideas what uh could be uh so to say the scope of the new standard. And here uh uh both Dolby and Nokia uh were of course had been involved long time in uh standardization of uh codecs. Uh he were the last codec also the one that I demoed uh with my demo uh before. This is the EVS codec. It was of course standardized. It was standardized 10 years ago um that one um everybody was aware that it had limitations uh like uh just mono voice and audio so there were the idea um or ah the ideas that there should uh soon be something like a stereo codec or something like this. Then of course in the standardization body there was uh, then further discussions on uh defining uh essentially um a new work item. And uh why these discussions in the standardization group uh accrue uh there was then a common understanding that uh stereo is not sufficient. So uh by that time uh uh all parties realized it has to be immersive audio. So then uh we essentially defined certain requirements for such a codec, uh for example requirements in terms of uh performance. Then also certain requirements in terms of uh design uh for example complexity, how much memory an implementation should consume on a device and so on. And after such kind of setting of uh requirements for the codec uh you had um uh technical development that would uh really also match uh these requirements. And uh such a development is essentially a completely different uh um track. Uh and uh standardization is very open so there can be anybody of course has to be member of the standardization organization but there can be anybody then uh providing a candidate for such as Donald here. In this case of the public collaboration it was um uh actually only uh a single candidate by the group uh by this public uh collaboration group.

Speaker A: So industry collaboration is critical to uh immersive audio technology. As Dolby and Nokia nailed down performance requirements and memory considerations. Three GPP and the international public collaboration bring nine partners to the table to create a single future proof technology that transcends proprietary boundaries. And in the end the entire market benefits from this open standards approach. What are some of the unexpected ways that immersive audio is shaking up society? Well this technology isn't just about fancy phone calls. It's about reimagining how we experience sound in every day dimension. So what surprising use cases are you already seeing for this technology?

Speaker B: Well I can say from, from the demos that we are doing with the um uh IVAs codec we see use cases related to uh in general immersive experience sharing. So for example you, you can think of um someone uh visiting um friends at some nice uh place um where essentially the idea is to share the complete experience. So this means that um rather than just uh uh doing a ah capture uh of voice there's actually the idea that there would be um the sharing of the complete, a complete sound ambience. In future we will see this of course uh be combined with video so meaning that we uh would see it also combined um uh with immersive uh video services. And then we are talking about very other user uh interfaces than just smart tones. For example here if we take uh such devices like this Apple Vision Pro, uh we can think of using uh, uh such services with video. On such devices we see applications of the IVAS codec on smart glasses. Meaning uh, that it uh then goes into the space of augmented reality, um in general. Then we are uh back at uh the various applications uh that we see uh so to say in the Metaverse. And there are of course huge possibilities that are possible. So anything from um, for example telemedicine to education applications using virtual reality, ah, you can think of remote uh um maintenance of machines using uh also um immersive sound and uh video experiences. So there are um, many uh opportunities for the industry. And of course this is a big share of what is uh the promise of the metaverse.

Speaker A: It's amazing technology really leapfrogs when multiple, often unrelated technologies converge to create new products. It wasn't until touchscreens, long life batteries, 3G came together that we got the iPhone. So it sounds like spatial audio is going to be used in the future in ways that you're just only beginning to understand.

Speaker B: Yes, I think that's true. What we are talking about like uh, the immersive um uh voice ah and audio service. Right. If we uh use it um on a smartphone for example, it relies on capturing immersive uh audio. And there, there is indeed uh necessity that you would capture with um, three or even more microphones in practice. We see already uh, uh on today's smartphones that many of them have three, uh, more and more uh of these smartphones have four microphones. Generally they are used for these numbers of ah microphones are used for example better noise suppression. But we can also now use them for immersive uh audio capture. So this is one of the enablers that helps us leapfrog. Uh, another enabler on the uh rendering side is that we see of course a trend that more and more people are used to wear earbuds. So um, they are not using their mobile phone any longer, just um, in handheld mode. But in practice they are using it well over stereo headphones, whether these are earbuds or real headphones, uh, um, something like that enables of course uh the uh rendition of uh immersive audio. So these uh components taken together, plus of course also the processing capabilities of uh modern smartphones make it uh, doable uh to have such um immersive services on smartphones. And like I said, we see it also in future on HMDs or on smart glasses.

Speaker A: If there was one thing you wanted the viewer to walk away from about spatial audio. In this next generation of sound, what should it be?

Speaker B: This would be the, uh, quantum leap in terms of user experience so that they really would learn that, uh, immersive, uh, voice and audio is such a big difference compared to today's mono experiences that are, uh, um, available on mobile

Speaker A: telephony, whether it be in a crowded mall, a work from home video call, or watching a rock concert. In the metaverse, the IVAs codec is coming to ease the pain of poor audio quality. Industry collaboration is key, and Dolby's work with Nokia and 3GPP will ensure that that future is crystal clear. Hi, I'm, uh, Michael Hainsworth. This has been the network effect.