Contextually aware cars – Advanced sensor fusion in a multimodal world

At Nuance, we continuously strive to integrate new intelligence into our mobility assistant platform, taking data from different aspects of the car and occupants to elevate the user experience. By doing so, we are progressing beyond the isolated integration of sensors and the information they provide – bringing intelligence and semantic fusion into the cabin and making better use of integrated technologies as they become ever more connected and automated.
By
Advanced sensor fusion in a multi-modal world

The vehicles we drive are becoming progressively more complex, with increasingly advanced technologies being added to each new generation of cars. These systems help enhance performance, efficiency and safety in our cars, but it is perhaps the convenience-focused features that are having the biggest impact, making our lives easier every day. This has led to many auto manufacturers adding an increasing number of sensors to their vehicles.

Parking sensors were first introduced nearly four decades ago. These small electromagnetic or ultrasonic sensors meant we no longer ran the risk of damaging our vehicles during slow-speed manoeuvres, because they would give an audible warning before an accident happened. This was the start of the integration of these valuable sensor-based technologies.

Today, sensor-based technologies are highly prevalent: night vision systems help us see more clearly in poor light, adaptive cruise control takes the stress out of motorway driving, blind spot monitoring prevents us from making dangerous manoeuvres when surrounded by traffic, park assist systems park our cars both in parallel and perpendicular spaces automatically, and driver drowsiness detection systems warn us when we should take a rest.

 

In-vehicle interaction

Along with these convenience and safety features, interaction inside the cabin is growing too: gesture recognition, handwriting capabilities, and voice control mean there are numerous ways to interact with the vehicle. And information can be presented in many ways, including central HMI screens, digital instrument clusters or head-up displays, and natural-sounding voice output (Text-to-Speech) making it easier to digest information.

Current, state-of-the-art systems utilise natural and varied inputs rather than stereotypical scripted commands. So, if a driver says, “it’s too hot in here,” the assistant knows to reduce the in-car temperature. Not only this, the natural language understanding recalls what a driver has said and can understand references to things that were said in the past. So, you might ask “what’s the weather like in London?” then follow up with “how about in Manchester?” The assistant has the intelligence to follow the conversation and respond to such requests.

 

Interconnectivity as key to improved UX

But, as useful as these systems are, they have always worked in a very isolated, independent and use-case-centric manner. In a changing world and in a time of greater connections, where information and data are combined and fused, technology can offer a deeper suite of functionality and provide a clearer picture of our surroundings.

At Nuance, we continuously strive to integrate new intelligence into our mobility assistant platform, taking data from different aspects of the car and occupants to elevate the user experience. By doing so, we are progressing beyond the isolated integration of sensors and the information they provide – bringing intelligence and semantic fusion into the cabin and making better use of integrated technologies as they become ever more connected and automated.

As Nuance continues to work on the future, we want to build systems that bring sensor and semantic fusion together into a multi-modal world – the ability to bring together information and resolve various references points between sensor data to provide valuable information to the driver. For example, the driver can look at shops and buildings and ask questions about them from within the car. The system is enabled to provide the user with useful information about the objects in question by combining the information in the user’s utterance with gaze and vehicle sensor information (e.g. positioning) and the 3D city.

As a result, Nuance’s Dragon Drive platform turns the vehicle into a mobility assistant, that includes Natural Language Understanding (NLU) and Natural Language Generation (NLG) as well as other modalities such as gaze, gesture and handwriting recognition, allowing drivers and passengers to interact with onboard systems and the environment intuitively – e.g. by looking at objects and asking about them.

 

Bringing the outside world on the user’s mind

Because of this, the integration of sensor and AI platforms will grow, bringing a plethora of more advanced systems that blend not only onboard sensor information, but receive and react to data from other vehicles and their surroundings. This evolutionary multi-modal and multisensory approach will merge information from the car, street, home and humans, creating a mobility assistant that goes beyond the car and can provide not only intuitive interaction between the vehicle and user but also serve as a proactive assistant, making useful offers and suggestions that fit the current driving situation. For example, considering the interaction with outside environment mentioned above, this can be expanded into a full-fledged natural form of interaction between the user and the environment including the car. The in-car mobility assistant will act as an intelligent medium in-between, that understands user’s needs and the environmental context. So, the user can point at a button in the car and ask for its functionality or look at a movie advertisement outside and ask the system to reserve tickets at the cinema close to the destination.

As autonomous vehicle technologies develop, more sensors are added to our cars, pressure is removed from the driver, and the opportunities for these systems grows yet again, opening the possibility for a myriad of new functionality.

In upcoming articles, I will take a look at how this multi-modal multisensor approach, whether it is in the car, on the street or even in the home, will continue to impact on the technologies we develop at Nuance and the functionality that it will allow us to offer.

Tags: , , , , , ,

Mohammad Mehdi Moniri

About Mohammad Mehdi Moniri

Dr.-Ing. Mohammad Mehdi Moniri is Innovation Manager for multimodal solutions at Nuance Automotive. Before joining Nuance Mehdi finished his Ph.D. at the German Research Center for Artificial Intelligence (DFKI), where he had also managed several research projects in the field of Human-Environment Interaction with applications in Automotive and Mixed-Reality scenarios. At Nuance, he is responsible for Horizon 2/3 activities in the fields of the multimodal input, output, information fusion, and fission. As product manager for these solutions, he leads R&D teams designing and implementing the horizon 2/3 interaction concepts and algorithms.