WO2025182009A1 - Generation device, generation method, and generation program - Google Patents

Abstract

A generation device 40 comprises a generation unit 41 that uses the arrival time of data from a data stream group determined in advance as a basis to generate a data arrangement plan for arranging, in a continuous storage area, data from a plurality of data streams having the same data cycle or similar data characteristics among the data stream group.

Description

Generation device, generation method, and generation program

This disclosure relates to a generation device, a generation method, and a generation program.

Communications infrastructure, with its strengths of ultra-low latency and massive connectivity, is expected to promote smart industrial processes such as remote factory control and distributed energy resource control. To achieve this, computing infrastructure must also be able to transfer and process data in deterministic time with ultra-low latency and support massive connectivity.

For example, the time it takes to collect data from a real-world data source and complete processing such as AI (Artificial Intelligence) analysis must be kept deterministically below a specified value. Even when collecting data from multiple data sources at the same time and performing multi-dimensional analysis, the time it takes must be kept deterministically below a specified value.

In this regard, one technique for improving the efficiency of packet sending and receiving is to divide large amounts of data received by the kernel or NIC (Network Interface Card) into smaller, more easily processed sizes, or to combine smaller pieces of processed data into smaller, more easily transmitted sizes (Non-Patent Document 1).

“Segmentation Offloads”, “Generic Segmentation Offload”, “Generic Receive Offload”, The Linux Kernel documentation, [online], [Retrieved December 12, 2020], <URL: https://docs.kernel.org/networking/segmentation-offloads.html>

However, it is not possible to dynamically optimize data partitioning and data integration depending on the situation. Furthermore, when partitioning data, huge amounts of data must be processed separately, which increases the number of I/O operations to the storage area where the data is temporarily stored.

This disclosure has been made in light of the above circumstances, and its purpose is to provide technology that can improve the time required to process data in a storage area.

A generation device according to one aspect of the present disclosure includes a generation unit that generates a data placement plan for placing data from multiple data streams with the same data period or similar data characteristics within a data stream group in contiguous storage areas, based on the arrival times of data from the data stream group that have been determined in advance.

In one aspect of the generation method disclosed herein, a generation device generates a data placement plan that places data from multiple data streams with the same data period or similar data characteristics within a data stream group in contiguous storage areas based on the arrival times of data from the data stream group that have been determined in advance.

A generation program according to one aspect of the present disclosure causes a computer to function as the generation device.

This disclosure provides technology that can reduce the time required to process data in a storage area.

FIG. 1 is a diagram showing an example of the overall configuration of a system according to this embodiment. FIG. 2 is a diagram illustrating an example of the configuration of a video analysis system. FIG. 3 is a diagram illustrating an example of a method for generating a data allocation plan and a data readout plan. FIG. 4 is a diagram showing an example (first example) of processor arrangement. FIG. 5 is a diagram showing an example (first example) of a data allocation plan and a data read plan. FIG. 6 is a diagram showing an example (second example) of processor arrangement. FIG. 7 is a diagram showing an example (second example) of a data allocation plan and a data read plan. FIG. 8 is a diagram illustrating an example (third example) of processor arrangement. FIG. 9 is a diagram showing an example (third example) of a data allocation plan and a data read plan. FIG. 10 is a diagram illustrating an example of a method for generating a data transfer plan. FIG. 11 is a diagram showing a process flow for generating each plan. FIG. 12 is a diagram showing a processing flow of a data stream. FIG. 13 is a diagram illustrating a hardware configuration of the generation device.

Embodiments of the present disclosure will be described below with reference to the drawings. In the drawings, identical parts will be designated by the same reference numerals and their description will be omitted.

[Summary of the Disclosure]
In the case of general-purpose operating systems (OS) currently widely used in cloud computing, scheduling is performed with an emphasis on fairness in processor usage time among tasks, so the wait time until a processor is assigned to a task is constantly fluctuating. In addition, interrupt processing is given top priority regardless of the task priority, so processing time is also constantly fluctuating.

This disclosure therefore attempts to build a real-time system that guarantees no fluctuations in waiting time or processing time. In particular, optimization is performed to reduce processing waiting time and processor idle time, taking into account the I/O processing load on the storage area. However, this is premised on the assumption that the data arrival time (including the data arrival time pattern) is known in advance. Communication networks in which the data arrival time is known in advance (communication means with determinism regarding data arrival) are existing technology.

Specifically, based on this premise, a data placement plan is generated that places data from multiple data streams (groups of data that flow intermittently over time) with the same data cycle or similar data characteristics in a contiguous storage area, and a data reading plan is generated in advance that bulks the data from the multiple data streams from that contiguous storage area and reads it all at once.

This reduces the number of I/O operations to the storage area, thereby reducing the time required to process data in the storage area.

[System configuration example]
In this embodiment, an example will be described in which video analysis is performed by aggregating video from a plurality of cameras.

FIG. 1 is a diagram showing an example of the overall configuration of a system 1 according to this embodiment. The system 1 includes multiple cameras 10, multiple processing devices 20, a user terminal 30, and a generating device 40. These are connected so as to be able to communicate with each other via a communication network 50.

Figure 2 shows an example configuration of a video analysis system. A group of video data streams output from multiple cameras 10 are distributed to a user terminal 30 via one or more processing devices 20 in series and/or in parallel.

The processing device 20 comprises a NIC 21 that receives multiple data streams from multiple cameras 10, a memory 22 that stores the multiple data streams, and one or more processors 23 that process the multiple data streams. The multiple processors 23 are connected in multiple stages. Video data is processed sequentially by a first processor 23a, a second processor 23b, and a third processor 23c.

In the case of Figure 2(b), the first processing device 20a is, for example, a video termination device. The first processor 23a performs color correction and contour enhancement on frames of the data stream. The second processor 23b changes the resolution of frames of the data stream to a resolution that can be processed by a video analysis device. The second processing device 20b is, for example, a video analysis device, and analysis processing is performed by the third processor 23c.

However, the processing device 20 may be a processing device that executes any processing. The processor 23 may also be a processor that executes any processing. The number of processing devices 20 and the number of processors 23 are also arbitrary. The processor may be, for example, a CPU, GPU, FPGA, etc. The memory 22 may be a shared memory. A sensor may be used instead of the camera 10.

[Configuration of the generation device]
1, the video analysis system of this embodiment includes a generating device 40. The generating device 40 includes a generating unit 41, a transmitting unit 42, and a storage unit 43.

The generation unit 41 has the function of allocating a memory area in advance to temporarily store each piece of data in the data stream group, generating a plan to write each piece of data into the allocated memory area when it arrives, and generating a plan to bulk out each piece of data from the memory area and read it all at once.

Specifically, the generation unit 41 has the function of generating a data placement plan that places, based on the pre-determined arrival times (including data arrival time patterns) of data from a data stream group, data from multiple data streams related to the same or similar tasks, that is, data from multiple data streams with the same data period or similar data characteristics, in contiguous memory areas.

The generation unit 41 has the function of generating a data reading plan for reading the data of the multiple data streams from the continuous memory area all at once.

The generation unit 41 has the function of generating a data transfer plan in which, after a first-stage processor among the multi-stage first processor 23a to third processor 23c has processed all of the data from the multiple data streams, the data from the multiple data streams is transferred all at once to a subsequent-stage processor.

The generation unit 41 has the function of generating a software program for referencing the data placement plan, the data readout plan, and the data transfer plan.

The generation unit 41 can also generate the data placement plan, data readout plan, and data transfer plan as a single plan.

The transmission unit 42 has the function of transmitting the software program to one or more processing devices 20.

The memory unit 43 has the function of storing the above software programs.

The storage unit 43 has the function of storing the data necessary to generate the data placement plan, data readout plan, and data transfer plan. The necessary data includes, for example, processor time slot design information (maximum processor execution time, maximum communication time between processors, etc.), and placement information (processing task content (data characteristics, etc.), input data volume, processor placement). This data can be obtained, for example, from actual machines in a test environment.

[Method for generating data placement plan and data readout plan]
FIG. 3 is a diagram showing an image of generating a data allocation plan and a data readout plan, and an image of processing data based on these plans.

Even if multiple frames are generated at the same time by multiple different cameras 10, there will be a phase shift due to fluctuations in processing within the cameras 10 and differences in the line length to the first processing device 20a for each camera 10, and the frames will not necessarily arrive at the same time.

Assuming that the data arrival time is known in advance, a data placement plan is generated in advance that bulks I/O processing of data streams with the same timing or similar characteristics and allocates them to contiguous memory areas, and a data reading plan is generated in advance that reads data from those contiguous memory areas all at once, thereby reducing the number of I/Os to memory and optimizing data processing wait times and processor idle time.

(Method for generating a data placement plan)
3, it is known in advance that data streams A, C, D, and E are 30 fps, and data streams B and F are 15 fps. It is also known in advance that the frames of the data streams will arrive in the following order at a given time: A → C → D → B → E → F.

In this case, the generation unit 41 generates a data placement plan that places 30 fps frames A1, C1, D1, and E1 in consecutive memory areas R11 to R14, in that order, and 15 fps frames B1 and F1 in consecutive memory areas R21 to R22, in that order, based on the arrival times of the data in the data stream group that have been determined in advance.

Similarly, for the subsequent 30 fps frames A2, C2, D2, and E2, the generation unit 41 generates a data placement plan for allocating them in consecutive memory areas in that order.Similarly, for the subsequent 15 fps frames B2 and F2, the generation unit 41 generates a data placement plan for allocating them in consecutive memory areas in that order.

(How to generate a data readout plan)
In the above example, the generation unit 41 generates a data reading plan that reads out 30 fps frames A1, C1, D1, and E1 arranged in consecutive memory areas R11 to R14 in that order, all at once, and reads out 15 fps frames B1 and F1 arranged in consecutive memory areas R21 to R22 in that order, all at once, and reads out 15 fps frames B1 and F1 arranged in consecutive memory areas R21 to R22 in that order, all at once.

Similarly, for the subsequent 30 fps frames A2, C2, D2, and E2, the generation unit 41 generates a data readout plan to read them all at once from consecutive memory areas in that order. Similarly, for the subsequent 15 fps frames B2 and F2, the generation unit 41 generates a data readout plan to read them all at once from consecutive memory areas in that order.

(Specific examples of data placement plans and data reading plans)
(First example)
As shown in FIG. 4, consider a case where multiple processors are arranged in multiple stages (horizontally on the page) and in parallel (vertically on the page), and the processing of three 30 fps data streams is consolidated into one data pipeline.

In this case, the generation unit 41 generates a data placement plan that places frames A1, B1, and C1 input to the first data pipeline in consecutive memory areas in that order. The generation unit 41 also generates a data readout plan that reads frames A1, B1, and C1 in that order all at once from consecutive memory areas.

Figure 5 shows a data placement plan and a data read plan corresponding to the configuration of the first data pipeline in Figure 4. At time 0, the first processor 23a obtains all of the data in a single read, since frames A1, B1, and C1 are arranged in a contiguous area. Next, at time 20, the second processor 23b obtains the processing results of frames A1, B1, and C1 from the first processor 23a in a single read. The same is true for data acquisition by the third processor 23c. Finally, at time 65, the third processor 23c writes the processing results of frames A1, B1, and C1 to contiguous shared memory in a single output.

Note that the "●" in the diagram indicates the processor's available status.

(Second Example)
As shown in Fig. 6, it is also possible to allocate the processing of data in a data stream in units of frames, which are smaller processing units. In this case, the generation unit 41 generates a data placement plan and a data read plan as shown in Fig. 7. At time 0 and time 75, frames A1 to J1 are I/O processed in consecutive memory areas.

(Third Example)
As shown in Fig. 8, it is also possible to reduce the number of first processors 23a and third processors 23c to two each, divide 300 fps (equivalent to 10 data streams) into 150 fps (equivalent to 5 data streams), and allocate this to the two first processors 23a. In this case, the generation unit 41 generates a data placement plan and a data readout plan as shown in Fig. 9. At time 0 and time 75, frames A1 to J1 are I/O processed in consecutive memory areas.

(supplement)
If the memory 22 is a shared memory that can be commonly accessed by multiple processors 23, it is preferable to plan for multiple data write operations to be performed between data read operations by each processor 23. This can further reduce the time required to process data in the memory area.

[How to generate a data transfer plan]
10 is a diagram showing an example of how a data transfer plan is generated and how data is processed based on that plan. The control unit 14 generates a data transfer plan for transferring, for example, four 30 fps frames A1, C1, D1, and E1 all at once to a downstream processor. Since multiple frames are transferred all at once, a delay occurs while the data is stored in a buffer, but the delay can be planned to be within a predetermined delay requirement.

The nth (n is a natural number) processor 23 receives four frames A1, C1, D1, and E1 all at once from the n-1th processor, according to the data transfer plan, processes them all, and then transfers the four frames A1, C1, D1, and E1 all at once to the n+1th processor.

As a result, as shown in Figure 10(a), the number of data input/output operations for the nth processor for the four frames A1, C1, D1, and E1 is a total of two. In contrast, in the conventional case, as shown in Figure 10(b), the number of data input/output operations is a total of eight. The number of data input/output operations between processors is reduced to one-fourth of the conventional number, enabling a reduction in the number of data input/output operations between processors.

The data placement plan, data reading plan, and data transfer plan are generated by the generation device 4. The generation device 4 may generate each plan as is based on the plan input by the user. The generation device 4 may also generate each plan autonomously using the data required to generate each plan. The generation device 4 may also generate each plan collectively at once. The generation device 4 may utilize machine learning, etc., when generating the task allocation plan.

[Plan generation operation]
FIG. 11 is a diagram showing a process flow for generating each plan.

Step S11:
The generator 41 generates a data allocation plan for allocating data of a plurality of data streams having the same data period in consecutive memory areas based on the arrival times of data of the data stream group that have been determined in advance.

Step S12:
The generator 41 generates a data readout plan for reading the data of the plurality of data streams from the continuous memory area all at once.

Step S13:
The generation unit 41 generates a data transfer plan in which, after a first-stage processor among multiple multi-stage processors has processed all of the data of the multiple data streams, the data of the multiple data streams is transferred all at once to a second-stage processor.

Step S14:
The generating unit 41 generates software programs relating to the generated data placement plan, data read plan, and data transfer plan. The transmitting unit 42 transmits the software programs to one or more processing devices 20.

[Data Stream Processing Operation]
FIG. 12 is a diagram showing a processing flow of a data stream.

Step S21:
The NIC 21 of the processing device 20 executes the above software program, and when data from the data stream group arrives, it places each frame of multiple data streams with the same data period in a consecutive memory area based on the data placement plan used by the execution.

Step S22:
Each processor 23 of the processing device 20 executes the software program and, based on the data reading plan used by the execution, reads out each frame (each frame of multiple data streams with the same data period) from consecutive memory areas all at once.

Step S23:
Each processor 23 of the processing device 20 executes the software program, processes all of the frames based on the data transfer plan used by the execution, and then transfers all of the frames together at once to the subsequent processor 23. Thereafter, the last processor 23 of the processing device 20 allocates the processed frames in a continuous memory area based on the data allocation plan.

[Application example]
(Application example 1)
For example, the present invention can be applied to assisting safe driving of a vehicle.

This use case prevents accidents by conducting a multi-dimensional assessment and prediction of the risk of collisions between vehicles and between vehicles and pedestrians at intersections through integrated analysis of multiple cameras, and providing real-time feedback to vehicles and pedestrians (via traffic lights, signs, speakers, etc.).

To realize such use cases, it is necessary to process everything from collecting camera footage to assessing and predicting the risk of accidents and issuing warnings to drivers and pedestrians with low and deterministic latency.

By applying this embodiment, it is possible to prevent accidents such as right-turn collisions and head-on collisions, which occur frequently at intersections. Specifically, it is possible to reduce collisions with oncoming vehicles caused by drivers misjudging the road or poor visibility when turning right, or ignoring traffic lights. It is also possible to reduce accidents caused by driver inattention or poor visibility at intersections with no traffic lights and stop signs.

(Application example 2)
The technology of this embodiment is highly effective in cases where high-speed, low-latency processing is required. The following use cases are possible examples of application.

One use case is short-term power supply and demand adjustment. Power consumption data for factories, data centers, homes, etc., as well as power supply data for energy conservation and automobile batteries, are collected and analyzed (sensing and risk assessment) with high frequency and low latency, and a stable power supply is achieved by adjusting demand and supply in response to adjustment requests from the grid, etc. based on the results of near-future power supply and demand forecasts.

One use case is low-latency trading. In automated trading systems in financial markets (systems used by market participants that receive data from the market, analyze the received data, automatically generate buy and sell orders, and send the generated orders to the exchange's system), real-time analysis is achieved using a variety of constantly changing market data in order to improve operational performance.

[effect]
According to this embodiment, the generation unit 41 of the generation device 40 generates a data placement plan for placing data from multiple data streams with the same data period or similar data characteristics within the arriving data stream group in consecutive memory areas based on the arrival times of the data from the data stream group that have been determined in advance, thereby reducing the number of times data is input into the memory 22 and the time required to process the data in the memory area.

Furthermore, according to this embodiment, the generation unit 41 of the generation device 40 generates a data reading plan that reads the data of the above multiple data streams from consecutive memory areas all at once, thereby reducing the number of times data is output from the memory 22 and further reducing the time required to process the data in the memory area.

Furthermore, according to this embodiment, the generation unit 41 of the generation device 40 generates a data transfer plan in which, after a first-stage processor among multiple multi-stage processors has processed all of the data in the multiple data streams, the data in the multiple data streams is transferred all at once to a second-stage processor. This reduces the number of data transfers and shortens the data transfer time between processors.

[others]
The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the present disclosure.

The generation device 40 of this embodiment described above can be realized, for example, as shown in FIG. 13, using a general-purpose computer system equipped with a CPU 901, memory 902, storage 903, communication device 904, input device 905, and output device 906. The memory 902 and storage 903 are storage devices. In this computer system, the CPU 901 executes a predetermined program loaded onto the memory 902, thereby realizing each function of the generation device 40.

The generating device 40 may be implemented as a single computer. The generating device 40 may be implemented as multiple computers. The generating device 40 may be a virtual machine implemented on a computer.

The program for the generation device 40 can be stored on a computer-readable recording medium such as a HDD, SSD, USB memory, CD, or DVD. The computer-readable recording medium is, for example, a non-transitory recording medium. The program for the generation device 40 can also be distributed via a communications network.

1 System 10 Camera 20 Processing device 21 NIC
22 Memory 23a First processor 23b Second processor 23c Third processor 30 User terminal 40 Generation device 41 Generation unit 42 Transmission unit 43 Storage unit 50 Communication network 901 CPU
902 Memory 903 Storage 904 Communication device 905 Input device 906 Output device

Claims (5)

a generation unit that generates a data allocation plan for allocating data of a plurality of data streams having the same data period or similar data characteristics within a data stream group into consecutive storage areas based on arrival times of data of the data stream group that have been determined in advance;
A generating device comprising: The generation unit
The generating device according to claim 1 , further comprising: a data readout plan for reading the data of the plurality of data streams from the continuous storage area all at once; The generation unit
The generation device according to claim 1 generates a data transfer plan in which, after a first-stage processor among a plurality of multi-stage processors processes all of the data of the plurality of data streams, the data of the plurality of data streams is transferred all at once to a second-stage processor. In a generation method performed by a generation device,
generating a data allocation plan for allocating data of a plurality of data streams having the same data period or similar data characteristics within the data stream group into consecutive storage areas based on the arrival times of data of the data stream group that are determined in advance;
Generation method. A generation program that causes a computer to function as the generation device described in any one of claims 1 to 3.