DE102024004075A1 - Methods for utilizing additional computing power and vehicle - Google Patents

Abstract

The invention relates to a method for using additional computing power for processing data by means of functions (A, B, C) which run in a data processing network (7) comprising at least one processor unit, wherein the data processing network (7) uses at least one communication interface (8) to establish a communication connection to a mobile device (2) via which computing power can be shared.
The invention is characterized in that a vehicle-internal data processing network (7) is used as the data processing network (2), which establishes a communication connection via the communication interface (8) to at least one mobile device (2) recognized in the vehicle (1) and/or attributable to a person using the vehicle (1), after which a portion of the processor capacity and/or storage capacity of the at least one mobile device (2) is used by the data processing network (7).

Description

The invention relates to a method for using additional computing power for processing data by means of functions of the type defined in more detail in the preamble of claim 1. Furthermore, the invention relates to a vehicle with an in-vehicle data processing network of the type defined in more detail in the preamble of claim 12.

Modern vehicle systems require comparatively high computing power due to complex sensor processing, navigation functions, and other features. This power is typically distributed across multiple control units within an in-vehicle data processing network, each handling the necessary functions. Functions, in this context, encompass all processes that transform input data—such as from sensors or function calls within the vehicle or its backend—into data processing, delivering relevant results for the vehicle or its user. This includes, in particular, the models used, such as machine learning models, large language models, and similar technologies. As vehicle functionality increases, especially in the infotainment sector, more and more customer-facing features are being implemented using machine learning models and artificial intelligence (AI) to offer vehicle users a more dynamic and intuitive experience than is possible with rule-based approaches. The functions can detect and process both processes inside the vehicle, such as speech recognition, gaze direction detection, fatigue monitoring, etc., and those in the vehicle's environment, such as object detection or localization for autonomous vehicle functions, emergency braking assistants, traffic sign recognition, etc.

In practice, however, this often leads to bottlenecks in available memory and/or processing power within the data processing network, especially when many, potentially concurrently applied, computing models are running. In such cases, the processes must be prioritized, which can result in long delays and/or a deterioration of the customer experience. To counteract this, it would be desirable to have larger resources available in the data processing network, at least temporarily.

The CN109769207B This describes a large-scale, dynamically operated data processing network that utilizes the computing power of mobile devices via a central computing system, employing these mobile devices as nodes. During operation of the dynamic network, the central computing system can then distribute incoming requests to the mobile devices to share their computing power. The mobile devices can also include the internal data processing systems of vehicles.

The EP 3 676 707 B1 Disclosing a method for utilizing a computing unit of an autonomously moving vehicle. For this purpose, whenever the vehicle is connected to a stationary system for charging its batteries, its computing power is made available to an external computing network.

The object of the present invention is therefore to provide an improved method for utilizing additional computing power and a suitable vehicle for this purpose.

According to the invention, this problem is solved by a method with the features in claim 1, and in particular in the characterizing part of claim 1. Advantageous embodiments of the method according to the invention are described in the dependent claims. Furthermore, a vehicle according to claim 12 solves the problem. Advantageous embodiments and further developments of this vehicle are also described in the dependent claims.

The method utilizes additional computing power for data processing via functions that run in a data processing network. This network comprises at least one processor unit, typically including multiple control units, each with at least one processor unit. Furthermore, the data processing network includes at least one communication interface, which it uses to establish a communication link with a mobile device. This makes the mobile device part of the data processing network, allowing the control units and the mobile device to share computing power.

According to the invention, an in-vehicle data processing network is used as the data processing network. This network establishes a communication connection via the communication interface to at least one mobile device detected in the vehicle and/or attributable to a user of the vehicle. Once the communication connection is established, a portion of the processor and/or memory capacity of the at least one mobile device is used by the data processing network. Results can optionally be sent back to the vehicle via the communication connection. Possible mobile devices include, in particular... Smartphones, but also other mobile devices such as smartwatches, laptops, augmented reality glasses (AR glasses), tablets, or wearables are possible. The only requirement for such a mobile device to be integrated into the data processing network is the ability to establish the necessary communication connection and to perform the necessary functions.

The data processing network can utilize various resources contained in the respective mobile device, e.g., memory modules, the CPU (Central Processing Unit), GPU (Graphics Processing Unit), or - if available - a special AI processor, i.e., a dedicated processor or part of another processor that is specifically optimized for the use of artificial intelligence.

This process enables an improved customer experience by offering more features. Furthermore, it can increase data security by reducing the use of backend services and optimizing the use of proprietary hardware. Finally, it can provide vehicle users with greater choice regarding the entities that process their data.

A highly advantageous further development of the method according to the invention can provide that a secure communication connection is established between the at least one mobile device and the data processing network via the communication interface, after which functions transmitted from the data processing network to the mobile device are processed on the mobile device in a secure area of the mobile device. Such security can be achieved, for example, by cryptographic methods, i.e., by signing and/or encrypting the exchanged data. This is particularly useful for wireless data transmission, e.g., via mobile networks (4G, 5G) or WLAN, to ensure security. The "secure area" of the mobile device also serves security purposes, but above all, it ensures that the requested computing and/or storage capacity is actually available after the communication connection has been established. This can preferably be ensured by an application pre-installed on the mobile device. Thus, an application is installed on the mobile device that acts as a placeholder and allows the vehicle's data processing network to utilize computing power. The secured area thus forms a kind of "bridgehead" of the data processing network within the mobile device's system.

The application, or in principle any other application on the mobile device, can also be used to detect the mobile device in the vehicle and/or to assign it to a person using the vehicle. Within the application, user profiles and permissions can be assigned. The application could also establish a connection with the vehicle for accessing media content or a hands-free system.

A further embodiment of the procedure may also provide for the use of additional applications installed on the mobile device from within the secure area of the mobile device. From the secure application environment, the vehicle's data processing network can thus also access the processing models already offered by the mobile device (for other purposes) and execute a parameterized call to the corresponding function.

A particularly advantageous embodiment of the invention further provides that the data processing network filters the currently running functions based on the maximum available computing power and the maximum available free storage space of the mobile device, according to their maximum required processing power and their maximum required storage capacity, so that those functions suitable for execution on at least one of the connected mobile devices are identified. After the communication has been initialized, the vehicle's data processing network must therefore decide which functions are to be offloaded to the at least one mobile device. For this purpose, the running and possible functions are first filtered according to storage requirements and maximum computing power, whereby functions that cannot be processed with the resources of the mobile device and/or the data transmission capacity of the communication interface are eliminated.

After filtering the functions based on their quantified computing power, only those functions that can realistically be executed on the given mobile device remain. These can then be stored in a list, for example.

According to a highly advantageous embodiment of the invention, the specific decision regarding which function is outsourced to the mobile device can utilize an optimization method that gradually outsources the functions based on their importance to the person using the vehicle. Therefore, it can be provided that the functions, in particular those to be executed on at least one of the connected mobile devices, are selected accordingly. Suitable functions are assigned an importance score, according to which the function with the highest importance score is executed on at least one connected mobile device. First, a score indicating importance is defined for each function. Sequential execution of the functions according to their importance is also conceivable. In this case, functions (A, B, C) are executed sequentially on the at least one connected mobile device (2) in the order of their importance scores, with the function with the highest importance score being executed first.

A beneficial further development of this approach involves initially setting the same importance score for all functions. The importance score of a function then increases with each use of that function relative to the importance scores of the other functions. The importance score can thus be defined, for example, by the vehicle manufacturer, so that all functions are equally important. As the vehicle is used, this score is then adjusted for each function based on usage. Functions used more frequently receive a higher score. This is based on the assumption that frequently used functions can be executed more reliably on a mobile device and should therefore be used preferentially there.

According to a particularly advantageous implementation, this importance indicator can be set as a value between 0 and 1 such that the sum of all importance indicators equals 1. A higher value of the importance indicator signifies greater importance and frequency of use. To guarantee the comparability of the values, the sum of a vector comprising the importance indicators should be 1. The vector can thus be viewed as a probability vector indicating how likely it is that a specific function will be called, based on past usage patterns.

Furthermore, it may also be provided that the importance rating can be changed by the person using the vehicle.

According to a favorable embodiment of the method according to the invention, the process can also be carried out iteratively for each connected mobile device. The process for integrating the mobile device and prioritizing the functions suitable for execution on the mobile device can be performed iteratively for a first mobile device in order to gradually offload several functions to the mobile device as needed and desired. Furthermore, the process can also be performed iteratively again for other mobile devices located in the vehicle.

In a further highly advantageous embodiment of the method, the vehicle's data processing network can also establish a communication channel to an external server, sharing its list of functions executable on the mobile device, including their required resources, with the external server. For this purpose, the vehicle is advantageously connected to the internet and transmits the identified list of valid functions and their computational costs to a central database and/or cloud instance of the manufacturer.

Furthermore, either as a supplement to this or as a purely alternative approach, the vehicle's data processing network can establish a communication channel to an external server. The data processing network then requests the maximum available computing power and free storage space of the mobile device from this external server for the specific vehicle-mobile device combination. Advantageously, the described process is thus extended by a preliminary step in which each vehicle's data processing network first queries a central database and/or cloud instance of the manufacturer to determine whether a list of functions for the vehicle-mobile device combination already exists. This list may originate from the same vehicle but from an earlier point in time.

Alternatively, according to a particularly advantageous embodiment of the method according to the invention, the vehicle-external server can generate information about the maximum available computing power and the maximum available free storage space of the mobile device for the respective vehicle-mobile device combination from feedback from other vehicles and/or manufacturer specifications for the respective vehicle and mobile device. The manufacturer already determines and provides information about the functions and models to the vehicles beforehand, so that during initiation, only storage capacity and computing power need to be filtered. Optionally, a list of mobile devices, including their hardware/software configurations, is also stored in the central database and/or cloud instance. Depending on this, for example in a matrix structure, it is listed which functions can be outsourced based on the configuration of the respective mobile device.

Should outsourcing fail in a particular case, a corresponding entry can be made in the material. rix is removed or replaced with a probability that creates statistics on successful and unsuccessful outsourcing.

The vehicle according to the invention is particularly suitable for carrying out the method according to the invention. It is provided that the vehicle is equipped with an in-vehicle data processing network and with a communication interface for communication with a mobile device detected in the vehicle and/or assignable to a user of the vehicle. According to the invention, the communication interface, together with an application on the mobile device, is configured to allow the data processing network to utilize at least part of the capacity of the processors and/or memory modules of the mobile device. The vehicle according to the invention thus provides a communication interface with at least one mobile device, e.g., a smartphone. The mobile device can execute predefined functions via this interface.

This also unlocks the advantages already mentioned above on the vehicle side, such as in particular the increased data security and the optimized use of proprietary hardware.

In an advantageous embodiment of the vehicle according to the invention, the communication interface can further include a power supply function for the mobile device. The mobile device can thus be charged or powered by the vehicle.

The communication interface can be wired. For example, the mobile device might be placed in a charging cradle compatible with its specific model, where it connects to the vehicle via a plug for charging and communication with the data network. However, in practice, such wired communication interfaces are not very convenient. A much more cost-effective vehicle upgrade therefore involves a wireless communication interface. For example, an inductive charging cradle could be used.

The actual communication link can be established wirelessly, e.g., via 4G, 5G, Bluetooth, or WLAN. Various communication technologies can therefore be used, as long as the channel capacity is sufficient for data transfer.

According to a particularly advantageous embodiment, the vehicle according to the invention can be configured such that the vehicle's internal data processing network comprises several control units and sensors, such as, in particular, environmental sensors and/or interior sensors. Especially in the evaluation of sensor data, machine learning models are frequently used today as functions within the meaning of this description. Particularly in such applications, it can happen that the resources of the control units are not always sufficient for processing the models or functions. Therefore, the possibility of using additional computing power to process data from a mobile device offers a particular advantage.

Further advantageous embodiments of the method and the vehicle according to the invention can also be seen from the exemplary embodiment, which is described in more detail below with reference to the figure.

• 1 eine schematische Darstellung eines Fahrzeugs zur Erläuterung der Erfindung;
• 2 eine tabellarische Darstellung der Funktionen und ihrer benötigten Ressourcen.

This shows:

• 1 a schematic representation of a vehicle to illustrate the invention;
• 2 a tabular overview of the functions and their required resources.

In the presentation of the 1 A vehicle 1 can be seen on the left. It has several sensors, which are indicated here by way of example in the form of two cameras 4 and 6 and an interior microphone 5. Together with control units, one of which is indicated here and labeled 3, they form part of a data processing network 7. The acquired sensor data is now evaluated on at least one control unit 3, along with data from other data sources, e.g., using machine learning models. These form an example of functions A, B, C (see...). 2 ) as described herein.

When processing these functions A, B, and C in data processing network 7, bottlenecks may occur in the memory and processing power of the control unit 3. This occurs primarily—but not exclusively—with beta programs and/or subsequently added software functions. To prevent this problem and to offer a user of vehicle 1 an improved experience with more and faster functionalities, at least one mobile device 2 will now be integrated into data processing network 7.

Mobile device 2 is, for example, a smartphone belonging to the person using vehicle 1. Therefore, it is not inherently part of data processing network 7. Vehicle 1. Typically, at the start of the journey, the person using Vehicle 1 transfers the device to Vehicle 3. Vehicle 1 now has a communication interface 8 to the mobile device 2 of the person using Vehicle 1, or to multiple mobile devices 2 belonging to that person, and/or to other mobile devices 2 belonging to other people in Vehicle 1. Possible mobile devices 2 include, in particular, smartphones, but other mobile devices such as smartwatches, laptops, tablets, or wearables are also possible. The requirement for such a mobile device 2 to be integrated into the data processing network 7 of Vehicle 1 is simply the ability to perform the necessary functions A, B, and C, and optionally or advantageously, the ability to charge the mobile device 2 while in use in Vehicle 1.

The feasibility of the necessary functions A, B, and C can be ensured in various ways. Either the mobile device 2 communicates a series of models already available on the mobile device 2 (e.g., speech models, object recognition models, etc.) to the vehicle 1 via the communication channel defined below, and/or the mobile device 2 communicates its available computing power and/or storage capacity. This, in turn, can be derived from the load observed on the mobile device 2 in a past time interval, such as the last few minutes. Alternatively, it can also be a guaranteed share of the computing power and/or storage capacity that the mobile device 2 can guarantee in all situations.

The communication interface 8 can be configured differently depending on the requirements and preferences of the person using vehicle 1. Since the person using vehicle 1 will, in most cases, place their mobile device 2 in a wireless charging pad, e.g., an inductive one, a wireless connection is assumed. Specifically, a wireless radio channel is established as a communication link with the mobile device 2, which, due to the proximity of the mobile device 2 and the continuity of its relative position to the transmitter, exhibits a very high data rate. Mobile devices 2 natively support standards such as 5G, which, through beamforming by the vehicle 1, can lead to at least a unidirectional acceleration of the already high data rate. However, WLAN, Bluetooth, and other communication connections are also possible. The procedure described below is independent of the specific communication technology. Therefore, we will only refer to the communication interface 8 or the communication connection in general terms, which can, in principle, also be wired.

An application is installed on mobile device 2 for the procedure. This application acts as a placeholder, allowing the data processing network 7 of vehicle 1 to utilize computing power. From this application environment, the data processing network 7 of vehicle 1 can also access the processing models already offered by mobile device 2 and execute a parameterized call to the corresponding functions A, B, and C.

After initializing communication with mobile device 2, the data processing network 7 of vehicle 1 must select which of the functions A, B, and C can be offloaded to mobile device 2. To do this, the running and potential functions A, B, and C are first filtered according to required memory and maximum runtime. Specifically, all functions that do not fit on mobile device 2 due to memory requirements or maximum runtime are eliminated.

The functions A, B, and C are then evaluated according to their importance and required computing power. 2 This is shown in an abstract table. The leftmost column, F, shows the functions A, B, and C, where A represents, for example, speech recognition, B object recognition, and C object localization. The required computing power is determined by the number of necessary calculations (FLOPs) of the function or model. These FLOPs, in turn, are derived from the number of parameters. They are listed in the second column from the left, labeled M for model. The data throughput rate D (third column from the left) also plays a role in the computing power; for example, for an optical sensor in the form of a camera, the number of images per second. These two parameters are multiplied, and for each function A, B, and C, a value is obtained that indicates the expected number of calculations per second. This value is entered in the fourth column from the left, labeled O for (necessary) calculations. This value is then added to empirically determined values regarding the necessary computing power, depending on available hardware such as memory, CPUs, GPUs, and/or AI processors for data processing and transmission, or linked via an average or maximum value. The data in the table is then filtered using information known from mobile device 2. This data, along with a renewed representation of mobile device 2, is shown in the... 2 The following is given below for the example presented here. The mobile device 2 shown in the example should therefore have 1.2 GB of free storage FR; an available computing power FC of 150 GigaFLOPs/s and, for example, a Large Language Model (LLM) as an embedded function EM.

This shows that function C (here: object localization) cannot be executed by mobile device 2, which only provides 150 gigaflops/s, due to the 4500 gigaflops per second required. For the remaining functions A and B, a usage-dependent parameter, the importance index, which indicates the importance of the functions, determines which function should be offloaded.

The specific decision regarding which of the suitable functions A, B, and C is offloaded to mobile device 2 results from an optimization process that minimizes the runtime of all functions A, B, and C, taking their importance into account. For this purpose, an importance value is first defined for each function A, B, and C. This value preferably ranges between 0 and 1 and is initially predefined by the manufacturer. During the use of vehicle 1, this importance value is adjusted for each function A, B, and C based on usage. Functions A, B, and C that are used more frequently are weighted more highly. Thus, functions A, B, and C that are used often are defined as important. This is based on the assumption that function A, B, or C can be executed more reliably on mobile device 2 and should therefore ideally be executed there. A higher importance value signifies greater importance and higher usage.

To guarantee the comparability of the importance metric, it is also stipulated that the sum of the vector must equal 1. It can therefore also be viewed as a probability vector indicating the likelihood of a specific function being called, based on past usage patterns. However, this value can also be modified by the function provider (PF) to offer greater configuration flexibility and personalization.

Since all functions still listed, namely functions A and B, are technically executable on mobile device 2, only the function with the highest importance score needs to be selected and transferred to mobile device 2. Speech recognition function A is used more frequently, which is why it has the highest importance score of 0.2 on the far right of the table. It is therefore transferred to mobile device 2. All data that would normally be transformed by this function is now sent to mobile device 2 via communication interface 8 and processed there using the transferred model. The results are then sent back to the corresponding control unit 3.

As already mentioned, the management of the list could also be done via a backend of the vehicle manufacturer in order to optimize the filtering based on their own data, fleet data and/or on the basis of combined technical data of the respective combination of vehicle 1 and mobile device 2.

QUOTES CONTAINED IN THE DESCRIPTION

This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• CN 109769207B [0004]
• EP 3 676 707 B1 [0005]

Claims (16)

Method for using additional computing power for processing data by means of functions (A, B, C) which run in a data processing network (7) comprising at least one processor unit, wherein the data processing network (7) uses at least one communication interface (8) to establish a communication connection to a mobile device (2) via which computing power can be shared, characterized in that the data processing network (2) is an in-vehicle data processing network (7) which establishes a communication connection via the communication interface (8) to at least one mobile device (2) recognized in the vehicle (1) and/or attributable to a person using the vehicle (1), whereby a portion of the processor capacity and/or storage capacity of the at least one mobile device (2) is used by the data processing network (7). Procedure according to Claim 1 , characterized in that a secure communication connection is established between the at least one mobile device (2) and the data processing network (7) via the communication interface (8), whereby functions transmitted from the data processing network (7) to the mobile device (2) are processed on the mobile device (2) in a secure area of the mobile device (2), wherein the secure area of the mobile device (2) and the secure communication connection are made available via an application previously installed on the mobile device (2). Procedure according to Claim 2 , characterized in that further applications installed on the mobile device (2) are used from within the secured area of the mobile device (2). Procedure according to one of the Claims 1 until 3 , characterized in that the data processing network (7) filters the currently running functions (A, B, C) and functions (A, B, C) expected to run in the near future based on the maximum available computing power and the maximum available free storage space of the mobile device (2) with respect to their maximum required computing power and their maximum required storage space, in order to identify those functions (A, B, C) which are suitable for execution on at least one of the connected mobile devices (2). Procedure according to one of the Claims 1 until 4 , characterized in that the functions (A, B, C), in particular the functions (A, B, C) suitable for execution on at least one of the connected mobile devices (2), are provided with an importance value, according to which the function with the highest importance value is executed on the at least one connected mobile device (2) or the functions (A, B, C) are executed sequentially on the at least one connected mobile device (2) in the order of their importance values, wherein the function with the highest importance value is executed first. Procedure according to Claim 5 , characterized in that the importance index is initially set the same for all functions (A, B, C), whereby the importance index of one function (A, B, C) is increased with each use of this function (A, B, C) relative to the importance indexes of the other functions (A, B, C). Procedure according to Claim 5 or 6 , characterized in that the importance value of the function (A, B, C) is set as a value between 0 and 1, such that the sum of all importance values is 1. Procedure according to one of the Claims 4 until 7 , characterized in that the process is performed iteratively for each connected mobile device (2). Procedure according to one of the Claims 4 until 8 , characterized in that the data processing network (7) of the vehicle (1) establishes a communication channel to a vehicle-external server, wherein the data processing network (7) shares its list of functions (A, B, C) executable on the mobile device (2) including their required resources with the vehicle-external server. Procedure according to one of the Claims 4 until 9 , characterized in that the data processing network (7) of the vehicle (1) establishes a communication channel to a vehicle-external server, wherein the data processing network (7) requests the maximum available computing power and the maximum available free storage space of the mobile device (2) for the respective combination of vehicle (1) and mobile device (2) from the vehicle-external server. Procedure according to Claim 10 , characterized in that the vehicle-external server obtains information about the maximum available computing power and the maximum available free storage space of the mobile device (2) for the respective combination of vehicle (1) and mobile device (2) from feedback from other vehicles and/or generated from the manufacturer's specifications for the respective vehicle (1) and mobile device (2). Vehicle with an in-vehicle data processing network and a communication interface for communication with a mobile device detected in the vehicle and/or attributable to a user of the vehicle, characterized in that the communication interface, together with an application on the mobile device, is set up to enable the data processing network to use at least part of the capacity of the processors and/or memory module of the mobile device. Vehicle after Claim 12 characterized in that the communication interface further includes a power supply function for the mobile device. Vehicle after Claim 12 or 13 characterized in that the communication interface is designed as a wireless communication interface. Vehicle after Claim 12 , 13 or 14 , characterized in that the provided capacity of the processors includes capacity of a CPU, a GPU and/or an AI processor of the mobile device (2). vehicle after one of the Claims 12 until 15 , characterized in that the vehicle-internal data processing network (7) comprises several control units (3) and sensors (4, 5, 6), such as in particular environmental sensors and/or interior sensors.