3D HMI in automotive: opportunities, challenges and the future of the industry

Speaker A: We don't want to go down a path where we spend a lot of resources on making sure we have safe situation for driving and then it's not actually going to be used while driving. That's an interesting perspective to keep in

Speaker B: mind actually for the brand. It's really nice to understand who your users are, um, and what you can suggest for them.

Speaker C: Hello and welcome to another episode of Shine, a podcast by Star. And today we are joined by two very special guests. We're joined by Joaquim Hayden, who is the UX chief designer and deputy head of HMI at uh, Lynk and Co. And then our very own Alex Troitsky, who is a UF designer in the automotive practice here at Star. And so today we're asking the big question, what are the risks and opportunities of the 3D human machine interface in cars? And so we dig into the impact that AI may have on the world of the human machine interface. We understand what the spectrum is of three dimensions within the world of these interfaces, all the way from simply just having a 3D image on a 2D screen to the full Minority Report experience. So let's jump into that episode and the first voice you'll hear will be that of Alex.

Speaker B: My name is Alex. I'm working as a UX and UI designer for Star from four to five years, uh, somewhere in the middle I believe. And my general experience in design is about uh, eight years right now. And uh, last few years I'm focusing towards uh, automotive design mostly. So probably I'll be some automotive expert in the future.

Speaker C: Got it. And Joachim?

Speaker A: Yes. Hi guys, I'm Joachim Hayden. I work as a chief designer at uh, yeah, UX at Lincoln Co Design, uh, sitting in Sweden. I have basically coming from the graphic, uh, design background, but I have an engineering degree and uh, never worked as an engineer. So I split into the design area more. I've been working for 15 plus years in the industry and have gone uh, more into UX or yeah, the overall field, despite being starting in uh, the

Speaker C: graphic design area, a wealth of experience in this episode. So my first question is just to understand what we're talking about when we refer to 3D HMI or 3D human machine interfaces. Now, dare I use the word Minority Report?

Speaker A: Yeah, maybe. Maybe soon.

Speaker C: So because as I was researching for this episode, it wasn't 100% clear to me if the current use cases of 3D human machine interfaces are at that level yet. Or that is just what we're expecting in the years to come. So can either of you enlighten me on this?

Speaker A: Yeah, I think for me I see that uh, we're rapidly moving up to that point where we're going to start seeing like full 3D systems. We have seen in the last years a uh, quick growth of use of 3D. But so far it's been like you still have a 2D based system, but you have islands of 3D. Uh started out with like nice renderings of the car. You did it in 2D, you exported images, put it into the UI, but then they started adding sort of turnable 3D models so you could visualize the car from different angles, etc. And these islands of 3D inside the HDMI, uh, they've started growing more and more and I think we'll start to see the sort of turning point where you actually go full 3D. I know that there's um, a bunch of OEMs that are looking into it and have started developing it, but as far as I know there's no one who has sort of dared to take the step yet to do a full 3D HMI.

Speaker B: Yeah, basically I think the main game changer here was like game engines when they updated their last releases and they accidentally started focusing on automotive more. Uh, I don't know for which reason but uh, yeah and they bring simplicity to this process I think because previously it was a little bit uh, complicated how you export different 3D assets and etc. Uh, making some like you know, fake 3D when you render out it with a video and then put into actual design and etc. And uh, yeah, actually I think these game engines is a game changer here because they natively working with 3D and that's why it's starting developing a lot faster.

Speaker A: Last year for those companies I think it was pretty natural move since they probably got more requests from automotive companies for support. When these islands of 3D starting to become more complicated, you started looking into using these type of tools and then I guess for them it wasn't that far to stretch to see think about the continuation of that. So it's a pretty natural step for them and a super big market as well. Of course,

Speaker B: uh, I mean uh, they also bring some quality into it because uh, actually design is something that can be outdated really fast. But talking about some game engines, VFX and etcetera, uh, you can also watch some like you know, 10 years old movie which still looks good because of VFX and etc. And that's what also these technologies bring into the industry. So using 3D in this case can help you to stay like, uh, to stay inside the trends for a longer period of time I believe.

Speaker A: Yeah, and I think there might be other business perspective as well for them because we uh, just got data actually the other day here from, from last week from like if you think about the, the growth of the Chinese car industry as well, the way cars are being used in, in Asia and in China maybe in particular is quite different from how we use them here. So we got data now that stating that the average time people spend in a car during a week is around 20 hours not driving. So that's like excluded the driving parts. You actually spend 20 hours per week in the car. So this basically means that 20 hours a week you have a platform where you have to entertain people and this, I mean then it's even more logical to step in as a game engine developer. Uh, if you have your game engine there that means that you can just offer your games as well.

Speaker B: That's I believe what Tesla is doing also right now. So especially talking about like electric vehicles, uh, given that not all of the people have their home chargers and etc. So they definitely need some entertainment while charging time and etc. And I believe Tesla uh, allows to use their steering wheel as uh, a uh, game device as well. So you can participate in some virtual racing and etc. I don't know if they have actually some multiplayer here, but it will be even more cool when you can race with your friends uh, inside your car and etc. But yeah, I think entertainment actually is the main area now for 3D because integrate uh, 3D into driving process is a little bit more complicated from safety perspective. So it's really hard to balance between how not to distract the driver from driving process itself. But yeah at the same time to display some useful information in 3D and uh, during the drivers process.

Speaker C: Okay, so let's move forward and start to understand some of the design challenges that we could face as we're moving through the 3D HMI spectrum towards the full experience.

Speaker A: Yeah, I think you touched upon it Alexander. I mean it's about cluttering, it's about uh, keeping the simplicity there for the actual driving situation. But I mean it's very interesting because we also are at the same time now we're on the way of having or seeing more uh, autonomous vehicles. So um, the question is if we will sort of the industry will be, be early enough to actually get 3D, more like 3D, full 3D systems out in the actual production cars before we see the autonomous revolution or if it's something that's going to be like a short window where we have these issues with uh, uh, cluttered views and um, misunderstandings about where things are placed and everything. Or if it's. The autonomy is coming early enough to actually meet that and then it might not be as big of a problem as we think because then the whole car ride will be more about entertainment and, and things like that. So, so it's. That's sort of, for me that's an interesting perspective and something we have to think about when we are looking into. Yeah, what we're actually putting our money, spending our money on developing these type of things because yeah, we don't want to go down a path where we spend a lot of resources on making sure we have a safe situation for driving and then it's not actually going to be used while driving. That's an interesting perspective to keep in mind actually. And looking at things, um, it's hard to see. I mean we have been hearing about autonomous vehicles for I don't know how many years and they say, they said like in 2015 we will see the first level five cars. Uh, and then it was 2018 and then it was 2020 and then it was 2022 and now it's 2027 earliest at least in Europe or. Yeah, um, don't quote me on the, on the years here but just to give an example, uh, so for some reason this is getting pushed all the time. So I have a feeling that there will be a period where it can become an issue. But uh, because I think the 3D parts are going to come faster than the actual autonomy. But there will be some time in a few years that it might not be such a big problem either. So yeah, it's an for me as that's an interesting uh, thought to have in mind when looking at developing these things.

Speaker B: Yeah, like I'm speaking as uh, not a big fan of autonomous car because I'm enjoying actually a process of driving and I don't want for some machine to replace me in this process. But uh, actually yeah, if we are talking about entertainment features, it's like a lot of areas where you can introduce while having some uh, charging sessions, you can have some entertainment via 3D and some games maybe as well as we already mentioned also while some long trips you need some entertainment. And uh, regarding the entertaining features, it's a lot of topics to discuss. But regarding actual pieces, what basically uh, have we previously already made for some clients, first of all it was some assistant. So beautiful, beautiful assistant. It can be some particles or some 3D model and etc. We have been working on this mostly in 3D and some uh, small pieces like 3D map, maybe some advanced 3D map when you can see like uh, I don't know, shopping mall. And you can navigate not to shopping mall itself, but for a uh, specific uh, specific shop in this shopping mall. So extended navigation inside of it, so where you can park your car, how you can get to needed floor inside uh, the shopping mall and etc. Stuff like this. So simple things via 3D maps and. Yeah, uh, and some um, AI assistant. And basically that's it for now.

Speaker A: You're sort of touching another interesting uh, angle to it. And that's, that's like when 3D now is getting more sort of taking over the whole system. You're also seeing this break point from where you use 3D in an autonomous perspective to just make things look nicer to actually interacting in 3D. And that's uh, also like a big turning point that needs to be sort of considered because working with 3D as before in isolated islands, it's just makeup as you were talking about, like adding particles to different views and making it look nice. And of course you can make things uh, much more dynamic looking when you do it in 3D and you have the talented renderers working on it. But then you have that shift when you do everything in 3D and suddenly there's a need to sort of translate interaction patterns and the whole setup to just change the way that the developers and uh, the designers need to think about this. Because suddenly you have to think three dimensions instead of two with adding some nice polish to it. So it's. Yeah, I think that's a big challenge as well.

Speaker B: Yeah. And basically uh, given that users don't have such experience and introducing something completely new, it also can cause some problems. That's why safety first priority to it. So making this 3D advanced map for example and et cetera, how to make it useful but at the same time not to distract the user from driving process and etc. Because yes, actually the most hard I believe to implement is driving process itself because uh, idling or parking or charging. There are also a lot of uh, opportunities to look into it when like something is broken in your car so you don't see an error code and that's all. But you can rotate somehow 3D model of your car, dive into it and highlight some, you know, some piece which, uh, is broken. Yeah. While some parking process or idling, I don't know, also can be useful. But it's all not, not about driving itself, about maintenance, entertainment, et cetera. And. Yeah. So Tom, answering your question, I believe introducing 3D into driving, uh, process is the most complicated one for me.

Speaker C: So I'm sensing that you, Alex, are not a big proponent of introducing 3D into the actual driving process, uh, as well as not being a big proponent on autonomous vehicles and actually that you're or you believe the greater value can be added through 3D in entertainment.

Speaker B: Yeah, for now, I, uh, think so, but we'll see later.

Speaker C: Sure. And Joaquim, would you also agree, do you think the greatest opportunity for like improving the lives of humans through 3D HMIS in cars is through entertainment or through functionality?

Speaker A: Yeah, um, I'm not sure. I think there are some great opportunities as well, but I think that the hard part is to sort of find them and also to. I think this change of going over to thinking with 3D requires a lot from the designers and I think there will be a need for a new way of thinking around this. But if you get it right, I think there are opportunities in both areas. So. But of course you always have to. Like if you're driving yourself, for instance, there's always this issue with not distracting the driver with unnecessary stuff. But technology in general, if it's used right, is a helper. So I think the biggest issue is people not understanding how to use it properly. And yeah, I always talk about this when the phone companies started adding a lot of features to their phones, uh, that's a recurring subject for me. And when I talk with people, they just added for the sake of technology. And then it became complicated to use. So I mean you had, you had email and Internet on your old Nokia phones or um, whatever brand, but no one really knew how to use it. And it was not even possible to use even sometimes, but it was there. And I think this is the same, same thing again. There's a new technology, uh, it get pushed into products and if you use it wrong, it's completely useless. But if you start like after a while when it gets more mature, people see the real sort of use cases for it. And I think we are just sort of in the start of seeing, in the automotive industry, um, seeing the usage of this. So I think there will be good use cases for it even in driving aid. But maybe we don't really have that overview yet. It's still too sort of new and fresh. So I think yeah, it's going to be a process to get there. But it was also interesting. We did some um, I mean obviously we're looking into this area at our company as well and we have done uh, some research, uh, and we are doing tests on simple prototypes that we're building. And coming back to your question there about um, like. Or your. Yeah, maybe it was you Alexander. We're talking about like what the users are sort of used to and we had a pretty interesting finding here. That was that 50, it was like 50, 50 people, some people really appreciated being able to interact with a 3D thing while 50% of the people preferred having everything in like you know, text listed with a button next to it. So and we couldn't really see any big differences in age or things like that. It was more like some people are more prone to be visually sort of focused of course. And then you, you really appreciate having some kind of real world connection with 3D if you do it right. Uh, while some people are more sort of, you want it line by line structure. You read, you like lists. So yeah, I think it's a lot about what you sort of expect rather than the technology itself. So people are different and people prefer different things.

Speaker B: Yeah, I can definitely agree on this. That's why a lot of innovative stuff. I begin with some user research and yeah, uh, for the brand it's really nice to understand who your users are. Um, and yeah and what you can suggest for them. Uh, one more thought that you mentioned, an example with the phones and a little bit step aside. But an interesting topic that I have read some article not long time ago about uh, some game changers or life changers in human uh development actually and one of life uh changers was introduction of the Internet itself. And then there is like you know a Graph People Early 20th century when some electricity was invented and there was a game changer. Then the biggest game changer was when Internet was invented. And on this graph the next game changer is the AI uh invented. And like you know, it was a straight graph, then it was electricity, then it was Internet and we had a little bit higher. But then on this graph at this point where AI invented we are moving like you know already to the top with much more speed than ever um, previously. So yeah, this is why also maybe to predict future, maybe complicated process for now. But yeah, we are talking about only what we know but I believe there are a lot of we don't know yet and how powerful also this AI can be and how much it will be the game changer as well.

Speaker A: Yeah, uh, definitely. It's harder now to just than a year ago to actually uh, predict the future and what sort of decisions to make on a corporate level for your future roadmaps, that's for sure. And I think this goes with this topic we're discussing today as well. If we're talking about interfaces in general and moving to a 3D one, thinking about that, uh, with some retrospective. It's the 3D part of going in three in the 3D direction with the HMI is quite, or was a quite natural step. Because if you think about first when you had only like in the stone Age, you interacted directly with stuff, right? You picked up a rock, you threw it at someone if you were cross with him. But then you started inventing things and you used tools to not directly interface with something. And along uh, came the computers and interfaces and then you, someone invented the computer mouse. So that's a pretty unnatural way really to interact actually. First you only had a keyboard, you had to prompt stuff. Then you had a mouse so you could graphically interact with stuff. And then along came touch and you could actually point directly instead of using a pointer with a mouse. So the next natural step would be to like not use a flat surface to interact with, but rather interact with your hands, uh, into directly manipulating something again. So we're sort of, we started by directly manipulating something a long time ago, then we have gone sort of the full circle soon to start interacting with stuff again, but through a digital medium. So in that way it's a pretty sort of natural thing that you could predict where this was going. But with AI as well being introduced, it's super hard if, you know, if we even have to interact with stuff or how far that is going to go. I see, uh, we're taking a little bit sidetrack here with AI, but it's very interesting and it's connected to a lot of this as well. But I mean one of the first use cases now that we see of course is the ability for AI to write code, for instance. And coding is a big part of working with uh, interfaces. I think this is something that potentially can speed up, uh, right now the sort of interaction of 3D interfaces. Because coding is very hard. It takes a lot of time, uh, to do iterations that uh, sort of from a design ID to the actual done product used to be quite, quite fast, at least if you did it in 2D. And now you do 3D, you have to build the whole environment. You have to make sure that all uh, the technical aspects are there, performance, etc. And then you sort of put on the styling in the end. So before you see the end result of your IDs, it's quite a long journey and AI could probably speed up that journey pretty fast. So I think making 3D interfaces could, could become much faster pretty soon than, than what we see now. But then again, is that just like an unnecessary step because maybe we can just start talking to stuff really well soon or maybe can even, I don't know, AI ah, can help us develop stuff that you can just. Yeah, you just have to think about what to do or it can actually predict what you need to see, uh, so fast. So uh, yeah, it's really hard to predict the future right now thanks to AI. Thanks AI.

Speaker B: Yeah, we take some step aside but uh, finishing the AI, uh topic, it's really uh, really big game changer because last year before this OpenAI, ChatGPT, Dall E and et cetera was introduced. We have been talking the main trend of the year was like metaverse and etc. And everybody like, you know, you talk to a client, they want to have a Metaverse, even if they are banking, app or I don't know, it doesn't matter. Uh, car manufacturers also have been talking about introducing Metaverse a lot. But then it failed completely because now I'm not sure if Metaverse is even trend right now because there was like a lot of NFTs, other stuff. Everybody wanted to have their own Metaverse, but the main owner of Metaverse, I believe, failed with his Meta a little bit and it was um, much worse than expected. And then yeah, introducing this AI replaced Metaverse completely. Uh, because suddenly it appears a lot more interesting for people interacting with digital, with real and digital world somehow than replacing your real world with digital. Because like uh, this was about Metaverse actually. And yeah, I think they failed. It's my own opinion. I don't know if you can agree if Metaverse is completely failed, but I can't see like know so much about meta, uh, right now. Agree.

Speaker A: I think Metaverse was the big buzzword a couple of years ago and that is also a big driver why everyone wanted to start working with 3D and you had to build up this, uh, every brand needed to have their own branded universe in 3D, et cetera. But yeah, I completely agree with you. I think the, the advantages in AI, uh, and with AI will probably drive us partly away from that need. And I think what we will see instead is A much bigger integration between the real world and the sort of 3D interfaces overlaid there. So I think the area of augmented reality instead is going to be. Yeah, we're going to see much more sort of innovation in that field pretty soon as well.

Speaker B: The like, uh, maybe also controlling some real world with uh, something digital is also a case. For example, uh, I believe it was last year on CES BMW presenting their dynamic car paint. So you can change the color of your car. And there's also like, uh, the usage of 3D can be here. Like when you drive something real world with something digital, for example, you can rotate somehow your car inside, like you know, uh, HMI and uh, place some stickers, uh, on your car or play some images and etc. And then the dynamic, uh, car paint will reveal it in the real world. Uh, that might be a cool case. Like uh, it will make everybody's car a lot more individual and uh, personalized actually. Because what we have been talking with different brands about is. Yeah, like while we have been talking about Metaverse, we have been talking a lot about how you can collaborate with different brands, how you can customize your car and present it in Metaverse and et cetera. So how everybody can be individual in this Metaverse. But then if we can present something digital in real life, I think it will be much more cooler than presenting digital in digital.

Speaker C: Some m great context from both of you there. From the Metaverse hype to the AI revolution, to Joaquim's history of the ui. I want to finish off now with an uplifting note. How do we think that a 3D human machine interface in cars is going to positively impact society over the next, say, five to 10 years.

Speaker A: It's a big question. Again, I think right now we're sort of in a lot of breaking points that we have discussed during these uh, questions and then talking here. But I think there's a, at this point of time, there's a lot of these sort of breaking points that are coinciding at the same time here. So it's super hard to predict right now where we're going. But I mean, if you talk about 3D compared to 2D in general, the obvious thing for me is that, I mean the human brain is sort of made to interpret 3D directly. 2D is always like some kind of projection of how you see things and you have to sort of visualize yourself. Making interfaces in 2D has always been like making good interfaces in 2D. You always had to try to build up like A mental model for users. So I see one of the things that is a big advantage of 3D, if you can do right, is that you have to build it in 3D. You have to place stuff physically just by the medium of 3D. So in a way you sort of, when you design it, what you used to do when you did 2D, is that you tried to build, or you tried to get the users to build a mental model of how things were placed, how they were in relationship to each other. But now you have to do that for the user because you do it when you design it. So you need to place objects in the 3D world directly, which means that the model you build will be the user's mental model of the system. And uh, of course it requires a lot more from designers and uh, to realize that and be aware of it. But I think that is something that will really, if used right, it can really help people. So it's much easier to understand how things, uh, different objects or different, uh, I don't know, menus or items in general in an interface relate to each other. Because you have it there in 3D and there's no escaping of that. Uh, so you have to sort of, you're forced to really think as a designer, you're forced to think about this, what, how do you place stuff, uh, where are they in relationship to each other? And a lot of good designers did that already in 2D, but a lot of not so good designers, they don't think about that. But still, every user, they do their own interpretation of how that mental model looks, even in 2D. So even if it's like the same thing with the overall user experience, like if you want to, uh, design something for a particular experience, uh, even if you don't, the user will get an experience. And if you do it consciously, you can do a very good product. And this will be the same thing basically, that if you do it good, everything will work fine. But if you're really not good at thinking about mental models and how things are supposed to relate to each other, you will have a hard time as a designer and, um, interface designer. The coming years here before AI takes over, then, or maybe months, I should say.

Speaker B: I can see we all agree that AI will replace everything very soon because it's developing so far. But yeah, basically I can agree totally with your point that it's changing the way of thinking completely when you need to, uh, create something for the user, especially for the user who don't have similar experience previously. So, yeah, a good Point and nothing even to it from my side here.

Speaker C: I want to thank both of you. We will obviously link to Both of your LinkedIn profiles below along with the link to uh Lynk and co Joaquin. But is there anywhere else the users or users our uh listeners can contact you?

Speaker A: I have an Instagram but I'm actually quite bad at putting up like work related stuff there. I'm a vivid biker so I have a mostly like biking stuff there so I don't know if that is interesting.

Speaker B: But uh, same for me actually. I'm pretty anonymous person in the Internet. I can always share my LinkedIn.

Speaker C: Okay, we'll have Joaquin's Instagram and then Both of your LinkedIns below. Thank you so much for your time.

Speaker A: Thank you. Really fun.

Speaker B: Yeah, thank you.

Speaker C: Okay, I hope you enjoyed that interview. Massive shout out to uh, Joaquim. Massive shout out to Alex for coming on and being so generous with their wisdom. I think we got to the bottom of the opportunity opportunities particularly that I wasn't considering, which was the vast opportunity for entertainment within the automotive world that the Joaquim shared about in China. Specifically the average amount of time spent not moving in a car is 20 hours per week. And so that obviously provides a massive opportunity for automotive manufacturers to add more value to the human experience with the use of the 3D human machine machine interface. And perhaps the most impactful thing for me was the understanding that we're game shared of the life cycle of user of the user interface. For example, we used to be picking up and throwing rocks and then we've moved through this spectrum of how we interface with technology from the mouse to the keyboard to the touchscreen. And now we're actually going right back to holding things in our hand again with the 3D human machine interface. So thank you Talik, thank you to Joaquim and of course thanks. Thank you to you for listening.