JP2003051964A - Image processing device - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To enable temporal synchronization between drawing processing and AV processing. SOLUTION: An operation unit 33a receives first stream data having a first time stamp and a plurality of object data to be drawn in a display frame, performs a first type operation process, and executes a result of the operation process. Is set in the memory area 4. The control unit 35A receives the display output synchronization signal, generates an n-th time indicating a time interval for displaying the n-th display frame, and inputs the first stream data. Then, the n time is compared with the first time stamp, and the setting in the memory area 4 is synchronized with the n time based on the comparison result.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to drawing processing and A
The present invention relates to an image processing device that performs V (AUDIO VIDEO) processing in synchronization.

[0002]

2. Description of the Related Art A conventional image processing system for performing drawing processing and AV processing will be described with reference to FIGS.

FIG. 8 is a block diagram showing a configuration example of a conventional image processing system. In the image processing system in FIG. 8, the CPU 1, the main memory 2, and the LSI 30 that performs drawing processing and AV decoding processing on the system bus 6.
Are connected. The local memory 4 is an LSI
30 is a memory area used exclusively. Furthermore, LS
I30 is a CPU interface 31, a memory interface 32, a drawing processing unit 33B, a video decoding unit 3
4, a control unit 35B, and a display output unit 36. Then, the control unit 35B controls the time stamp detection unit 3
5a, a frame stamp detector 35e, and a time counter 35d are built in. Time stamp detector 3
5a detects a time stamp in the decoded video stream data. Frame stamp detector 35e
Is rendering stream data (hereinafter, referred to as “DL (DISP
LAY LIST) ". ) Inside the frame stamp is detected. The time counter 35d counts up the time in synchronization with the display output synchronization signal indicating the period for displaying the display frame output from the display output unit 36. The output result from the display output unit 36 is displayed on the monitor 5.

FIG. 9 shows a main memory 2 and a local memory 4.
3 is a schematic diagram showing an array of image processing data stored in FIG. In FIG. 9, 21 is video stream data such as MPEG, 22 is DL, 21A is a time stamp added to the video stream data 21, and 22C is a frame stamp added to the DL 22. The video stream data 21 includes a time stamp 21A as a field, and thereafter, a plurality of video stream data to be decoded are consecutive within the display period indicated by the display output synchronization signal. On the other hand, the DL 22 includes a frame stamp 22C as a field, and thereafter has a configuration in which a plurality of drawing commands to be drawn are continuous. Further, the DL 22 manages a plurality of drawing commands from one frame stamp to the next frame stamp as drawing data for one frame (one display screen). The video stream data 21 and DL 22
Are linearly arranged with respect to the address of the held memory.

The CPU 1 transfers the video stream data 21 and DL 22 from the main memory 2 to the LSI 3. These data are temporarily held in the local memory 4 by the CPU interface 31 and the memory interface 32, but are linearly held at the same address as the main memory 2. Then, the video stream data 21 and the DL 22 are sequentially read by the LSI 30.
The time stamp 21A and the frame stamp 22C included in the read video stream data 21 and DL 22 are detected by the time stamp detecting unit 35a and the frame stamp detecting unit 35e incorporated in the control unit 35B. The control unit 35B controls the drawing processing unit 33B, the video decoding unit 34, and the display output unit 36 together with the display output synchronization signal used for the monitor 5.

FIG. 10 is a diagram showing an example of combining display frames in conventional image processing. In FIG. 10, (a)
Shows the time elapse of the drawing result in the drawing frame memory, (b) shows the time elapse of the decoding result in the decode frame memory, and (c) shows the time elapse of the combining result in the display frame. ing. Further, display output synchronization signals 101 to 103, 111 to 117, and 121 to 1
27 is a drawing processing unit 33 and a video decoding unit 3 respectively.
4 and the display output synchronization signal referred to by the control unit 35B when the frame memory of the display output unit 36 is switched.

The outline of the control in the conventional image processing system will be specifically described below with reference to FIGS. 8 to 10. Here, a case where the LSI 30 performs the drawing process and the video decoding process in parallel will be described.

The CPU 1 writes the DL 22 generated as data for drawing processing written in the main memory 2 into the local memory 4 together with the video stream data 21. At the same time, the CPU 1 instructs the control unit 35B to start processing. The control unit 35B controls the drawing processing unit 33B and the video decoding processing unit 34. DL22
Is processed in synchronization with the display frame period in the drawing processing unit 33B, but is not necessarily updated for each display frame due to the progress of the drawing process. That is,
In FIG. 10A, switching of the drawing frame memory is performed based on the display output synchronization signals 101 to 103. On the other hand, the video stream data 21 is processed in the video decoding processing unit 34 in synchronization with the display frame period, and updated for each display frame. That is, in FIG. 10B, the switching of the decode frame memory is performed based on the display output synchronization signals 111 to 117. Therefore, the switching of the decode frame memory is performed in accordance with the display output synchronization signals 121 to 127.

The control of the drawing frame memory, the decoding frame memory, and the display frame will be specifically described below with reference to FIG.

<Control of Drawing Frame Memory> First, the control of the drawing frame memory will be described with reference to FIG.

As a premise, the control unit 35B prepares a first drawing frame memory and a second drawing frame memory in the local memory 4.

First, it is assumed that the drawing processing unit 33 has already completed drawing of the second drawing frame at the time when the display output synchronizing signal 101 is input. At this time, the control unit 35
B designates the first drawing frame memory as a destination to which the drawing processing unit 33B outputs the calculation result. At the same time,
The control unit 35B designates the second drawing frame memory as a drawing frame memory to be referred to when the display output unit 36 performs composite display. The control unit 35B controls the drawing processing unit 33B.
After starting drawing to the first drawing frame memory,
The display output synchronization signal is monitored, and the drawing frame memory displayed on the display output unit 36 is not switched until the drawing in the first drawing frame memory is completed.

Specifically, the control unit 35B instructs the drawing processing unit 33B to start the calculation for the nth DL 22, for example. Then, the drawing processing unit 33B outputs all the objects in the DL 22 to the first drawing frame memory, and when the drawing is completed, notifies the control unit 35B of the end of the calculation. Then, the control unit 35B controls the n-th DL2.
The end of the operation for 2 is detected. After that, when the display output synchronization signal 102 is input, the drawing frame memory referred to by the display output unit 36 is changed to the first drawing frame memory, and the drawing processing unit 33B performs the second drawing as the drawing frame memory for drawing. Specify the frame memory.

Then, the control unit 35B instructs the drawing processing unit 33B to start the calculation for the (n + 1) th DL 22 for the next drawing frame. The control unit 35B inputs the display output synchronization signal indicating the display frame period, and when the drawing processing unit 33B finishes drawing, selects the drawing frame memory to be displayed and notifies the display output unit 36 of the selected drawing frame memory. As described above, the control of the drawing frame memory is performed on the assumption that all the processes corresponding to the plurality of objects in the DL are performed.

<Control of Decode Frame Memory> Next,
Regarding the control of the decode frame memory, FIG.
Will be described with reference to.

As a premise, the control unit 35B prepares the first video frame memory and the second video frame memory in the local memory 4.

First, it is assumed that the video decoding unit 34 has already completed the decoding process of the second video frame memory when the display output synchronizing signal 111 is input. At this time, the control unit 35B designates the first video frame memory as the destination to which the video decoding unit 34 outputs the decoding result. At the same time, the second video frame memory is designated as the video frame memory to be referred to when the display output unit 36 performs the composite display. Control unit 35B
Monitors the display output synchronization signal after the video decoding unit 34 starts decoding to the first video frame memory, and the video frame memory displayed on the display output unit 36 is switched together with the display output synchronization signal.

Specifically, the control unit 35B instructs the video decoding unit 34 to start decoding of the nth video stream data 21, for example. The video decoding unit 34 sequentially decodes the video stream data 21 starting from the time stamp 21A and outputs the video stream data 21 to the first decoded frame memory. When this processing is performed within the input interval time of the display output synchronization signal, the display output synchronization signals 111 to 117 and the display output synchronization signals 121 to 127 for updating the decode frame memory and the display frame respectively have the same display. Since the output synchronizing signal can be used, the output to the monitor 5 is updated every display output synchronizing signal.

Further, the control section 35B inputs the display output synchronizing signal indicating the display frame period and increments the built-in time counter 35d. The time counter 35d is compared with the time stamp 21A included in the video stream data 21 being video-decoded, and the video frame memory having the time stamp 21A matching the time indicated by the time counter 35d is selected. The display output unit 36 is notified.

Then, the control unit 35B detects the end of video decoding for the nth video stream data 21. After that, when the display output synchronization signal 112 is input, the video frame memory referred to by the display output unit 36 is changed to the first video frame memory, and the video decoding unit 34 serves as the second video frame memory for performing video decoding. Specify the video frame memory. Then, the control unit 35B instructs the video decoding unit 34 to start the calculation on the (n + 1) th video stream data 21 for the next video frame memory.

Further, the control unit 35B inputs the display output synchronizing signal, and when the video decoding by the video decoding unit 34 is completed, it selects the video frame memory to be output and notifies the display output unit 36. If the video decoding by the video decoding unit 34 is not completed,
Correspondence such as continuing to display the (n-1) th video frame memory is performed. However, since the display frame is determined based on the display output synchronization signal and the time stamp 21A, the video stream data is processed in real time.

Note that the video decoding process may not be completed in time within the display frame period due to the calculation capability of the video decoding unit 34 and the like. In this case, the current video decoding process may be stopped, and in this case, the control unit 35B is notified of the stop of the video decoding process.

<Display Frame Control> Finally, display frame control will be described with reference to FIG.

The display output unit 17 has a function of synthesizing the drawing frame memory shown in FIG. 10A and the decoding frame memory shown in FIG. 10B and outputting it as a display frame. Therefore, the display output unit 36 superimposes the drawing frame memory and the decoding frame memory designated by the control unit 35B, and outputs them to the monitor 5.

By the control as described above, the combined display frame output to the monitor 5 is not updated in the drawing result for the three-frame synchronization period. On the other hand, the video decoding result is updated every frame.

[0026]

However, the above-mentioned conventional techniques have the following problems.

First, there is a problem that it is difficult to synchronize the video display screen superimposed with the drawing of the drawing object in terms of time.

This problem occurs when image processing is performed in which it is necessary to display a specific drawing object at a specific time. As described above, the control unit 35B sets the video stream data 21 to the display screen for the video decoding unit 34 based on the comparison result between the time counter 35d and the time stamp 21A included in the video stream data. Order. On the other hand, also for drawing, the control unit 35B instructs the drawing processing unit 33B to draw on the display screen based on the comparison result.
Therefore, it is affected by the rendering performance and video decoding performance of the image processing device, which causes an error in display frame synchronization. Therefore, it becomes difficult to synchronize the drawing and video display screens in units of display output synchronization signals.

Second, there is a problem that it is difficult to process data with the maximum performance of the image processing apparatus.

This problem occurs when the drawing processing and the video decoding processing are performed by the same arithmetic unit in the image processing apparatus, or when the drawing processing unit 33B and the video decoding unit 34 perform processing on the unified memory. That is, this is a problem caused by performing drawing processing and video decoding processing by using the memory in a time division manner. When the real-time property is required and the video decoding process is processed with the highest priority, the video decoding process must be completed within the input interval of the display output synchronization signal. Therefore, the drawing process is performed asynchronously with the time stamp 21A added to the video stream data 21.

Therefore, the drawing process is not started after the drawing of one drawing frame is completed and until the next display output synchronizing signal is input. Therefore, even though the video decoding process and the drawing process have finished,
A period in which the drawing process must be waited until the next display output synchronization signal is input occurs without fail at a frequency of once every several frames. For this reason, the CPU 1 cannot generate the DL 22 based on the maximum performance of the image processing apparatus, which makes it difficult to perform data processing with the maximum performance of the image processing apparatus.

In view of the above problems, the first object of the present invention is to provide an image processing apparatus capable of temporally synchronizing drawing processing and AV processing. A second object is to provide an image processing apparatus that enables data processing with maximum performance.

[0033]

Means for Solving the Problems In order to solve the above problems, the means of the invention according to claim 1 has a first time stamp and a plurality of object data to be drawn in a display frame. For stream data,
An arithmetic unit that performs the first-type arithmetic processing and draws the result in a memory area, and an n-th time indicating a time interval for displaying the n-th display frame are generated based on the display output synchronization signal. And a control unit for comparing the time with the first time stamp of the first stream data, and synchronizing the drawing in the memory area with the nth time based on the comparison result. .

According to the first aspect of the present invention, since the control unit inputs the first stream data having the time stamp, the time management for the object data can be performed by the software program on the user side. Then, by comparing the nth time and the first time stamp, the first stream data can be drawn in synchronization with the nth time which is the time axis of display.

According to a second aspect of the present invention, in the first aspect of the invention, the object data is added with a priority indicating the degree of necessity of drawing in the memory area, and the arithmetic unit is configured to perform the priority processing. Based on the above, the first type arithmetic processing is performed.

According to the second aspect of the present invention, by adding the priority indicating the necessity degree for drawing to the object data, the software program on the user side allows
It is possible to construct a drawing processing system capable of coping with image processing apparatuses having different drawing processing performance.

According to a third aspect of the present invention, in the image processing apparatus according to the first aspect, the arithmetic unit further includes a second
The second stream data having the time stamp of 1 and the data to be decoded in the display frame is subjected to the second type arithmetic processing, and the result is set in the memory area. n time and the second
The second time stamp of the stream data is compared with the second time stamp, and the setting is synchronized with the nth time based on the comparison result.

According to a fourth aspect of the invention, in the image processing apparatus according to the second aspect, the arithmetic unit further comprises a second
The second stream data having the time stamp of 1 and the data to be decoded in the display frame is subjected to the second type arithmetic processing, and the result is set in the memory area. n time and the second
The second time stamp of the stream data is compared with the second time stamp, and the setting is synchronized with the nth time based on the comparison result.

According to the invention of claim 3 or claim 4, the control unit further inputs the second stream data having the second time stamp to input the nth time, the first time stamp and the second time stamp. By comparing with the time stamp, the drawing of the first stream data and the setting of the second stream data can be synchronized.

According to a fifth aspect of the present invention, in the image processing apparatus according to the third aspect, the control unit is configured to perform the first time interval from the nth time to the (n + 1) th time. The arithmetic unit is controlled so as to complete the second-type arithmetic processing of the second stream data to be processed in n times, and the arithmetic unit processes from the first time interval to the nth time. The first type arithmetic processing of the first stream data starts at a third time interval after subtracting the second time interval required for the second type arithmetic processing of the power second stream data, and the control unit If the first type arithmetic processing is not completed within the third time interval, the first type arithmetic processing of the first stream data to be processed at the time after the (n + 1) th time is skipped, First The continuation of the first type processing the first stream data to be processed in the time assumed to command to the arithmetic unit.

According to the invention of claim 5, when the first type arithmetic processing is not completed within the third time interval, the first type arithmetic processing of the first stream data to be processed in the (n + 1) th time is skipped. By continuing the first type arithmetic processing of the first stream data to be processed at the nth time, the drawing of the first stream data and the setting of the second stream data can be synchronized.

According to a sixth aspect of the present invention, in the image processing apparatus according to the third aspect, the control unit is configured to perform the first time interval from the nth time to the (n + 1) th time. The arithmetic unit is controlled so as to complete the second-type arithmetic processing of the second stream data to be processed in n times, and the arithmetic unit processes from the first time interval to the nth time. The first type arithmetic processing of the first stream data starts at a third time interval after subtracting the second time interval required for the second type arithmetic processing of the power second stream data, and the control unit If the first-type arithmetic processing is completed within the third time interval, the n + th time is calculated from the completion of the first-type arithmetic processing.
Until the time is incremented by 1 time, the nth +
It is assumed that the arithmetic unit is instructed to stop the first-type arithmetic processing of the first stream data to be processed in one time.

According to the sixth aspect of the present invention, when the first type arithmetic processing is completed within the third time interval, the first type arithmetic processing is stopped until the third time interval ends, whereby the first type arithmetic processing is completed. It is possible to synchronize the drawing of the stream data and the setting of the second stream data.

According to a seventh aspect of the present invention, in the image processing apparatus according to the fourth aspect, the control section is configured to perform the first time interval from the n-th time to the (n + 1) -th time increment. The arithmetic unit is controlled so as to complete the second-type arithmetic processing of the second stream data to be processed in n times, and the arithmetic unit stores a priority register that holds a value of a predetermined priority. And a priority register for holding the value of the priority for which the first-type arithmetic processing has been completed, the second stream data to be processed in the n-th time from the first time interval. Two
At the third time interval after subtracting the second time interval required for the seed operation processing, the first seed operation processing of the first stream data is started, and when the first time is incremented to the (n + 1) th time, the first The first type arithmetic processing of the first stream data to be processed after the (n + 1) th time is updated by updating the value of the priority register to the priority value of the object data for which the seed arithmetic processing is completed. Object data with a priority value greater than or equal to the value held by the priority register is operated, and object data with a value lower than the priority value held by the priority register is not operated And

According to the invention of claim 7, when the first type arithmetic processing is not completed within the third time interval, the priority value held by the priority register is referred to, and the priority value is more than the priority value. The first-class arithmetic processing of low data is not performed. This allows
The drawing of the first stream data and the setting of the second stream data can be synchronized. In addition, it is possible to perform the first type arithmetic processing utilizing the maximum performance of the image processing apparatus.

Further, in the invention of claim 8, in the image processing apparatus of claim 4, the control section is configured to perform the first time interval from the n-th time to the (n + 1) -th time increment. controlling the arithmetic unit so as to complete the second-type arithmetic processing of the second stream data to be processed in n times, and the arithmetic unit comprises a priority register for holding a value of a predetermined priority; At the third time interval obtained by subtracting the second time interval required for the second type arithmetic processing of the second stream data to be processed at the n-th time from the first time interval, the first stream data Starting the first type arithmetic processing of, the first type arithmetic processing calculates the object data to which the value of the priority higher than the value held by the priority register is added,
The first type arithmetic processing of the object data corresponding to the priority held in the priority register is completed at the (n-1) th time until the first type arithmetic processing is incremented at the (n + 1) th time. In the case of
Until the time is incremented, the n-1 th
Considering the priority value held by the priority register before the time, it corresponds to the priority having a value lower than the priority value held by the priority register at the (n-1) th time. It is assumed that the object data is additionally calculated, and the value of the priority register is updated to the priority value of the object data for which the first-type operation processing is completed when incremented in the (n + 1) th time.

According to the eighth aspect of the present invention, when the first type operation processing is completed within the third time interval, the priority value held by the priority register is lowered, so that more object data is drawn. Since it can be performed, the drawing of the first stream and the setting of the second stream data can be synchronized. In addition, it is possible to perform the first type arithmetic processing utilizing the maximum performance of the image processing apparatus.

According to a ninth aspect of the present invention, in the image processing apparatus according to the eighth aspect, the control unit sets the value of the priority register to the value when the value held by the priority register is updated. The stream data output unit that generates and outputs the first stream data is notified, and the stream data output unit selects, from the first stream data,
It is assumed that only the object data to which the priority value greater than or equal to the updated value held by the priority register is added is selected and output.

According to the ninth aspect of the invention, when the value of the priority register is updated in the process of the first type arithmetic processing inside the image processing apparatus, the CPU or the like which generates the first stream data on the system bus. The stream data output unit can be notified to adjust the generation amount of the first stream data. Therefore, the transfer of the first stream data on the system bus can be minimized. Therefore, it is possible to prevent the reduction of the data processing capability of the image processing device involved in the data transfer on the system bus.

According to the tenth aspect of the present invention, in the image processing apparatus according to the eighth aspect, the first stream data is input, and the priority value equal to or higher than the updated value held by the priority register is set. It is further provided with an interface unit that selects only the added object data, reconstructs the first stream data, and holds the reconstructed first stream data in a buffer.

According to the tenth aspect of the present invention, the interface section selects only the object data of the priority level that can be drawn by the image processing apparatus from the first stream data corresponding to the objects having different drawing priorities. Internal data transfer. Therefore, it is possible to minimize the data transfer amount in the case of data processing in which a drawing frame and a display frame are mapped and used on the unified memory. Therefore, it is possible to prevent the reduction of the data processing capability of the image processing device involved in the internal transfer.

[0052]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Each embodiment of the present invention will be described with reference to the drawings.

(First Embodiment) A first embodiment of the present invention will be described with reference to FIGS.

FIG. 1 is a block diagram showing an example of the arrangement of an image processing system according to the first embodiment of the present invention. In the image processing apparatus system shown in FIG. 1, a CPU 1, a main memory 2, a drawing process (corresponding to a first type arithmetic process) and an AV decoding process (corresponding to a second type arithmetic process) are arranged on a system bus 6. The LSI 3 to be executed is connected.
In addition, the local memory 4 (corresponding to the memory area) is
This is a memory area used exclusively by the LSI 3. further,
The LSI 3 includes a CPU interface 31, a memory interface 32, a drawing processing unit 33A (corresponding to a calculation unit), a video decoding unit 34 (corresponding to a calculation unit), a control unit 35A, and a display output unit 36.

The control section 35A has a time stamp detecting section 35a, a time stamp detecting section 35b, a priority detecting section 35c, and a time counter 35d built therein. The time stamp detection unit 35a detects a time stamp (corresponding to the second time stamp) 21A in the video stream data (corresponding to the second stream data) 21 shown in FIG. 2 described later. The time stamp detection unit 35b detects a time stamp (corresponding to the first time stamp) 22A in the DL (corresponding to the first stream data) 22 having a plurality of object data to be drawn in the display frame. Priority detection unit 35
c detects the priority indicating the priority of object data to be drawn. The time counter 35d counts up in synchronization with a display output synchronization signal indicating a period for displaying the display frame output from the display output unit 36, and generates a time interval for displaying the display frame. The time counter 35d generates, for example, the nth time indicating the time interval for displaying the nth display frame. The output result from the display output unit 36 is displayed on the monitor 5. The drawing processing unit 33A also includes a priority register 33a that holds a priority value.

FIG. 2 is a schematic diagram showing an array of image processing data in this embodiment. In FIG. 2, 21
Is a video stream data such as MPEG, 22 is a DL, 21A is a time stamp added to the video stream data 21, and 22A is a time stamp added to the DL 22. The video stream data 21 is
It includes the time stamp 21A as a field,
A plurality of video stream data to be decoded are consecutive within the display period indicated by the display output synchronization signal. On the other hand, the DL 22 includes a time stamp 22A as a field, and thereafter, a plurality of drawing commands 2 to be drawn
2B is continuous. Further, the DL 22 is added with a priority indicating the necessity of an object to be drawn with respect to the drawing command 22B. Then, the drawing processing unit 33A sequentially inputs the DL 22 to which the priority is added, and draws each object in consideration of the necessity of drawing. That is, the drawing processing unit 33A draws objects having priorities equal to or higher than the determined value in the order in which they appear regardless of the priority. The video stream data 21 and the DL 22 are arranged linearly with respect to the address of the held memory.

The CPU 1 transfers the video stream data 21 and DL 22 from the main memory 2 to the LSI 3. These data are temporarily held in the local memory 4 by the CPU interface 31 and the memory interface 32, but are linearly held at the same address as the main memory 2. Then, the video stream data 21 and the DL 22 are sequentially read by the LSI 3. Read video stream data 21 and DL 22
The time stamps 21A and 22A included in are detected by the time stamp detection units 35a and 35b incorporated in the control unit 35A. The control unit 35A uses the monitor 5
Together with the display output synchronization signal used for
3A, the video decoding unit 34, and the display output unit 36 are controlled.

The control unit 35A receives the DL 22 to which the time stamp 22A shown in FIG. 2 is added. Therefore, the control unit 35A causes the drawing processing unit 33A to draw the drawing command 22B that is being drawn in the drawing frame memory currently being drawn.
However, it can be controlled on the basis of when the output from the CPU 1 is made to be displayed on the monitor 5. Specifically, the control unit 35
A inputs the display output synchronizing signal indicating the display frame period and increments the built-in time counter 35d. Here, for example, the n-th time indicating the time interval for displaying the n-th display frame is generated. Then, the n-th time is compared with the time stamp 22A included in the DL 22 on which the drawing process is performed, and the DL 22 having the time stamp 21A matching the n-th time indicated by the time counter 35d is selected. At this time, the selected DL 22 is notified to the display output unit 36, and the display output unit 36 is notified.
Displays the selected DL 22 on the display screen.

In this way, it becomes possible to synchronize the drawing of the drawing object and the video display screen superimposed on each other in time.

Also, the synthesis of display frames in the image processing apparatus using the DL 22 to which the priority is added will be described with reference to FIG.

FIG. 3 is a diagram showing an example of combining display frames in the image processing apparatus according to this embodiment. In FIG. 3, (a) is an image processing device with low drawing performance, which is DL2.
FIG. 6 is a diagram showing the elapsed time of a drawing frame memory that holds the result of calculating 2; FIG. 6B is a drawing frame memory that holds the result of calculating DL 22 in an image processing device with high drawing performance. Further, (c) is a decode frame memory that holds the result of video decoding processing of the video stream data 21. Further, (d) is a display frame of an image processing apparatus having low drawing performance, and (e) is a display frame of an image processing apparatus having high drawing performance.

For example, a DL including priority levels 1 to 4
22 is generated, the DL 22 becomes the display frame shown in (d) when it is output to the image processing apparatus with low drawing performance, and (e) when it is output to the image processing apparatus with high drawing performance. It becomes the display frame shown in. That is, the display frame shown in (d) is a combined display of the drawing frame memory shown in (a) in which only the drawing object of priority level 1 is drawn and the decoding frame memory shown in (c). Further, the display frame shown in (e) is such that the drawing frame memory in which the drawing objects of priority levels 1 to 4 shown in (b) are drawn and the decoding frame memory shown in (c) are combined and displayed. Become.

However, priority level 1 (level 1
Means that the value has a high priority) contains essential information for a plurality of users having image processing devices having different drawing performances. Therefore, by displaying a drawing object at this level, a plurality of users can perform processing without causing a system problem. On the other hand, a user of an image processing apparatus having high drawing performance capable of drawing all drawing objects from priority levels 1 to 4 (meaning that the priority values are sequentially decreased from level 2 onward) has more detailed information. Can be obtained. For example, in an application in which the user instructs the CPU 1 to input an input for moving the drawing command 22B based on the drawing and display results of the image processing device, and the CPU 1 generates the DL 22 based on the change of the drawing command 22B, A case can be assumed in which the drawing object instructing the movement is designated as priority 1 and the background object is designated as priority 2.

As described above, according to the present embodiment, by inputting the DL 22 with the time stamp 22A added to the control unit 35A, the control unit 35A generates the nth time and the time stamp 22A generated from the display output synchronization signal. It is possible to temporally synchronize the drawing of the drawing object with the display of the video display screen by comparing. Therefore, this control is indispensable for an application that requires drawing in time units. Also,
By inputting the DL 22A to which the priority is added to the drawing processing unit 33A, due to the drawing processing performance and other reasons,
Even when the drawing process is not in time within the time specified by the time stamp 22A, the drawing processing unit 33A enables the drawing object to be selected according to its priority. Therefore, when the user decides the operation to be input at the next time based on the display result, the minimum necessary drawing command 22B can be specified, so that the drawing processing performance required for the image processing apparatus has a wide range. Thus, data processing can be performed with the maximum performance of the image processing apparatus.

Although the drawing processing unit 33A and the video decoding processing unit 34 are shown separately in FIG.
The present invention can be similarly applied to the case where the time division processing is performed using the same arithmetic unit. Also, AV-CG synchronization processing using the same arithmetic unit will be described in a second embodiment described later.

(Second Embodiment) A second embodiment of the present invention will be described by taking a case of a car navigation system as an example with reference to FIGS. 1, 2, and 4 to 7. It should be noted that in the figure, the same reference numerals are given to constituent elements common to the first embodiment.

<Outline of Car Navigation System> First, FIG. 4 shows an outline of the car navigation system.
Will be described with reference to.

FIG. 4 is a diagram showing a configuration example of a car navigation system using the building drawing method according to the present embodiment. The car navigation system shown in FIG. 4 includes a car navigation device 41, a liquid crystal screen device 42, a vehicle sensor 43, a GPS antenna 44, a VICS antenna 45, a disk 46, a remote control device 47,
A light receiving device 48 and a digital AV device 49 are provided.

The car navigation device 41 has a function of selecting an optimum route from the current position of the vehicle to the destination, drawing the route along with map information, and decoding the digital AV stream. The liquid crystal screen device 42 displays the drawing result by the car navigation device 41.
The own vehicle sensor 43 includes a speed sensor for knowing the speed of the own vehicle and a gyro sensor for knowing acceleration / deceleration and turning. The GPS antenna 44 is an antenna for accurately knowing the vehicle position. The VICS antenna 45 is
It is an antenna for knowing road traffic information such as traffic jams.
The disk 46 is a recording medium such as a DVD-ROM or a CD-ROM in which map data and AV data are recorded. The remote control device 47 is a device for the user to set a destination, set conditions for route selection, and control a digital AV stream for the car navigation device 41. The light receiving device 48 is a device for the remote control device 47. The digital AV device 49 is a digital A device such as a terrestrial digital broadcast or a DVD-Video 0.
A device for reading a V stream.

FIG. 5 is a block diagram showing an outline of a configuration example of the car navigation system 41. The car navigation device 41 in FIG.
CPU 1, main memory 2, display output unit 36, disk reading unit 52, GPS receiving unit 53, VICS receiving unit 5
4, a vehicle information analysis unit 55, and a remote control signal input unit 56
A digital AV input section 57 and a drawing section 60. The output destination of the display output unit 36 in FIG. 5 is the liquid crystal display device 42 also shown in FIG. Also in FIG.
Although not shown in the figure, the GPS antenna 44 is connected to the GPS receiving section 53, and the VICS receiving section 54 is connected to the VICS.
The antenna 45 is connected, the own vehicle sensor 43 is connected to the own vehicle information analysis unit 55, and the digital AV device 49 is connected to the digital AV input unit 57. Further, the light receiving device 48 is connected to the remote control signal input unit 56 via the liquid crystal screen device 42, and a signal indicating the user's instruction content received by the light receiving device 48 is input from the remote control device 47.

Here, the CPU 1 performs various processes for controlling the entire car navigation device 41 according to the program stored in the main memory 2. As the processing performed by the CPU 1, for example, the drawing command 22B shown in FIG. 2 including the calculation of the rendering parameter for the drawing unit 60 is issued, and the digital A for the drawing unit 60 is issued.
It also includes issuing V-streams.

<Specific Operation of Drawing Unit 60> Next, a specific operation of the drawing unit 60 will be described with reference to FIG.

First, the drawing unit 60 performs a process corresponding to the process performed by the arithmetic unit and the control unit 35A shown in FIG.
That is, the drawing unit 60 performs the drawing process and the digital AV.
The stream decoding process and the drawing and decoding synchronization process are performed.

The drawing unit 60 includes a drawing control unit 50, a rendering unit 51, a DL / AV memory 4a, and a display memory 4b (corresponding to a memory area). The DL and AV memory 4a and the display memory 4b correspond to the local memory 4 in FIG.

Drawing command 22 issued from CPU 1
B is stored in the DL and AV memory 4a via the system bus 6. The drawing control unit 50 uses DL and A
The drawing commands 22B are sequentially fetched from the V memory 4a, and the drawing commands 22B are decomposed into primitive drawing commands and rendering parameters that can be interpreted by the rendering unit 51. Then, after the modeling coordinates given as the parameters of the drawing command 22B are converted into the drawing coordinates, the drawing command 22B
A rendering command reflecting the attribute of is issued to the rendering unit 51. The rendering unit 51 uses the drawing control unit 50.
Drawing is performed in the display memory 4b in accordance with the drawing command and the rendering parameter received from.

The digital AV stream issued from the CPU 1 is stored in the DL and AV memory 4a via the system bus 6. The drawing control unit 50
Digital AV streams are sequentially taken out from the L and AV memories 4a, and decoded. The drawing control unit 50 also stores the decoding result in the display memory 4b. The display output unit 36 video-outputs the contents drawn in the display memory 4b to the liquid crystal screen device 42.

The DL and AV memory 4a is used for drawing processing or digital AV stream decoding processing.
By setting the display memory 4b in the drawing unit 60, the drawing unit 60 performs either one or both of the drawing process and the decoding process. That is, in the same vertical blanking period, the drawing control unit 50 temporally switches the drawing process and the decoding process of the digital AV stream. Further, the DL and AV memories 4a and the display memory 4b simultaneously hold areas for drawing processing and digital AV stream decoding processing, so that drawing and digital AV can be simultaneously displayed on the display screen in synchronization. It will be possible.

In this embodiment, the drawing control unit 50
Together with the drawing process, the decoding process of the digital AV stream is performed as so-called soft decoding. However, the present invention can be similarly implemented even when the decoding process of the digital AV stream is performed by a decoder engine other than the drawing control unit 50 as so-called hard decoding.

<Synchronization Processing> Next, the synchronization processing will be specifically described.

First, the digital AV stream decoding and drawing synchronization processing is the same as that described in the first embodiment, with the digital AV stream (corresponding to the second stream data) and the DL 22 (first stream data). Corresponding to the time stamp (first
And corresponding to the second time stamp). That is, the drawing control unit 50 generates, for example, the n-th time corresponding to the n-th display frame from the display output synchronization signal and compares it with the respective time stamps to synchronize the decoding and drawing of the digital AV stream. Do. It should be noted that, here, the case where the second stream data is data of video (video) information and audio (audio) information has been described, but even if the second stream data is additional information such as character information, the present embodiment It can be similarly implemented.

Further, in the decoding process of the digital AV stream, the audio decoding and the video decoding are synchronized by the time stamps added to the respective streams. The digital AV stream is stored in the DL and AV memory 4a from the digital AV input section 57 via the CPU 1, the main memory 2, and the system bus 6.

A concrete description will be given with reference to FIG. FIG. 6 is a diagram for explaining a digital AV decoding and drawing synchronization process.

The digital AV stream data 61 is
The access unit 61A can be disassembled, and each access unit 61A has time stamps 73a (audio) and 73v (video). The digital AV stream is stored for each object indicated by each access unit 61A in the decode buffer 64 set for each object in the DL and AV memory 4a. Then, the drawing control unit 50 performs a decoding process for each decode buffer 64.

At this time, if the time stamp 73v in the video stream decoding is too advanced with reference to the time stamp 73a in the audio stream decoding, the video stream is decoded by 1V.
A freeze process is performed in which the OP (Video Ojeck Plane) (VOP is each screen that constitutes a VO (Video Ojeck) included in a video stream) is stopped for the time and the same VOP is displayed.

On the other hand, when the time stamp 73v in the video stream decoding is too late, the video stream is skipped to the next I picture, and the skip processing for continuing the video stream decoding is performed.
In this way, the decoding of the audio stream is synchronized with the decoding of the audio stream by delaying the video decoding that is too advanced and advancing the video decoding that is too late based on the decoding of the audio stream.

Next, the DL 22 synchronous processing will be described with reference to FIG.

The drawing process is also synchronized by the time stamp. The DL 22 can be decomposed into a plurality of drawing commands 22B, and a drawing command 2 that sets a time stamp 73d at the beginning of the DL 22 to be drawn in a frame within the same vertical blanking period (corresponding to the first time interval)
Put 2B. That is, it is expected that the DL 22 following the drawing command 22B placed at the head will be drawn and displayed at the same time.

As described above, in the same vertical blanking period, the drawing control unit 50 temporally switches the drawing process and the decoding process of the digital AV stream. Therefore, the drawing process takes time (second second) from the same vertical blanking period to video decoding.
It is desired that the drawing process be started and completed within the time (corresponding to the third time interval) until this vertical blanking period is subtracted by subtracting (corresponding to the time interval). However, due to drawing performance and other factors,
The drawing process may not be completed within the third time interval, or conversely, may be completed by the end of the vertical blanking period within the third time interval. In this case, the drawing process is synchronized by using the speed adjustment type system by the drawing freeze process or the skip process and the object adjustment type system that adjusts the amount of the drawing object based on the priority of the drawing object. Be done.

The reason why both types are used together is that there are two cases for adjusting the drawing when the drawing processing performance of the image processing apparatus is insufficient. In other words, even if all the desired objects in one frame need to be drawn even if the dropped frames occur, on the contrary, it is necessary that the dropped frames do not occur even if all the desired objects cannot be drawn. There are cases. In this embodiment, a method for enabling adjustment in both cases will be described below.

The reason why the audio decoding is used as a reference for the synchronization is that the change in the audio reproduction speed is more noticeable to the user than the video, and therefore the audio reproduction speed needs to be as constant as possible. . Therefore, of the processing in the drawing processing unit 50, the audio decoding processing has priority over the video decoding processing and the drawing processing. In addition, the time stamp 7 in the decoding of the audio stream that should be the reference for synchronization
If there is no 3a, the CPU 1 manages the time and generates the time stamp 73a, so that the same synchronization process can be performed.

(1) Rendering speed adjustment type synchronous processing First, the speed adjustment type synchronization processing of drawing will be described.

As described above, this processing is a method of adjusting the synchronous processing of drawing when it is necessary to draw all desired objects in one frame even if a frame drop occurs in drawing. When the time stamp 73d in the drawing process is too advanced compared to the audio time stamp 73a, for example, the DL corresponding to the nth time
If the drawing process of 22 is completed within the third time interval,
The drawing process of the DL 22 corresponding to the nth time is stopped and the freeze process for displaying the same frame is performed.

Further, the time stamp 7 in the drawing process
3d is too late compared to the audio time stamp 73a, that is, DL2 corresponding to the nth time.
If the drawing process of 2 is not completed within the third time interval,
In the drawing processing at the (n + 1) th time, the DL 22 corresponding to the (n + 1) th time is skipped and the DL corresponding to the nth time is skipped.
The drawing process of 22 is continued, and the skip process of continuing the drawing process is performed.

As described above, the audio decoding process and the drawing process can be synchronized by delaying the drawing process that is too advanced and advancing the drawing process that is too late based on the audio decoding.

(2) Drawing Object Adjustment Type Synchronous Processing Next, the drawing object adjustment type synchronous processing will be described.

As described above, this processing is a method of adjusting the synchronous processing of drawing even when all desired drawing objects cannot be drawn and it is necessary to prevent dropping of frames. In this method, the priority specified for each drawing object is the same as the drawing priority (here, the drawing priority means the priority indicating an object that can be drawn by the image processing apparatus) or the drawing priority. Only objects that are higher than the drawing priority are drawn, and drawing objects that are lower than the drawing priority are not drawn. As a result, within the third time interval corresponding to the nth time, the priority register 3
If all the objects having the same or higher priority than the drawing priority corresponding to the (n-1) th time held by 3a have been drawn, lower the drawing priority. The number of objects to be drawn in the third time interval corresponding to the next (n + 1) th time is increased by. At this time, considering the priority value held by the priority register 33a before the (n-1) th time, for example, statistics of the value of the priority register 33a are taken so that the display screen is not blurred. , Until the vertical blanking period corresponding to the nth time ends, it is possible to additionally calculate objects having lower priority values.

On the other hand, within the third time interval corresponding to the nth time, all of the objects having the same priority as the drawing priority held by the priority register 33a or higher than the drawing priority have not been drawn. If it is completed, the skip processing in the speed adjustment type synchronization processing method described above is performed. At the same time, the drawing priority is increased to reduce the objects to be drawn in the third time interval corresponding to the next (n + 1) th time. By thus performing the drawing process in consideration of the drawing priority, it is possible to prevent dropping of frames and synchronize the drawing and the video display screen with the maximum performance of the image processing apparatus.

(3) AV and CG synchronization processing Finally, the above description is repeated.
The synchronization process of V and CG will be described with reference to FIG. 7. FIG. 7 is a flowchart for explaining the AV and CG synchronization processing.

First, in step S1, it is determined whether or not the image processing system has audio data. If there is audio data (YES in process S1), the process proceeds to process S2. In step S2, the audio data is decoded to extract the time stamp 73a in order to perform synchronization based on the audio data, and the process proceeds to step S4. On the other hand, when there is no audio data (NO in step S1)
In step S3, the CPU 1 generates a time stamp 73a which serves as a synchronization reference. And process S4
Proceed to.

Next, in step S4, it is determined whether or not the image processing system has video data. When there is no video data (NO in step S4), the process proceeds to step S9. on the other hand,
If there is video data (YES in process S4), the process proceeds to process S5. In process S5, the video data is decoded to extract the time stamp 73v, and the process proceeds to process S6. In step S6, the time stamp 73a is compared with the time stamp 73v to determine whether the video decoding is ahead or behind the reference time stamp 73a. If video decoding is in progress (YES in process S6), the process proceeds to process S7, and freeze process of video decoding is performed. Then, the process proceeds to step S9. On the other hand, when the video decoding is delayed (NO in step S6), the process proceeds to step S8, and the video decoding skip process is performed. Then, the process proceeds to step S9.

Next, in step S9, it is determined whether or not the image processing system has drawing data. If there is no drawing data (NO in step S9), the process in one display frame period ends. On the other hand, if there is drawing data (process S9
If YES), the process advances to step S10 to draw an object having the same priority as the drawing priority or a priority higher than the drawing priority, and does not draw a drawing object having a priority lower than the drawing priority. Then, the process proceeds to step S11. In step S11, it is determined whether or not the desired drawing process has been completed by the end of the vertical blanking period. When the desired drawing process is completed (YES in process S11)
In step S12, the drawing priority is lowered to increase the amount of objects to be drawn in the subsequent drawing, and the process ends. On the other hand, if the desired drawing process has not been completed (NO in step S12), the process proceeds to step S13, the drawing skip process is performed to display the previous drawing result again, and the drawing priority is increased. The processing is ended by reducing the amount of objects to be drawn in the drawing.

As described above, according to the present embodiment, the drawing process that is too advanced is delayed and the drawing process that is too delayed is advanced based on the audio decoding.
The AV decoding process and the drawing process can be synchronized. Further, since the drawing process and the AV decoding process are synchronized in consideration of the drawing priority, it is possible to allow the data processing at the maximum performance of the image processing device by giving a range to the drawing processing performance required of the image processing device. .

Here, a modification of the embodiment of the present invention will be described.

Conventionally, a plurality of multimedia terminals having various drawing performances and video decoding performances on a network, such as when a plurality of image processing apparatuses are connected to the network and playing a network battle type game, have been used. However, in the case of a process requiring real-time processing between these terminals, there is a problem in that it is difficult to synchronize the video display screen superimposed with the drawing of the drawing object in time. This problem has largely hindered the realization of multimedia applications, which makes sense to synchronize the drawing of a drawing object with the display of a video display screen.

In such a case, the input from each terminal is collected by the server on the network, and thereafter the drawing stream data and the video stream data are distributed at every time interval set by the server. Then, it is necessary for each terminal side to judge the drawing result and the result of the video decoding process and perform the output operation to the next network. Therefore, when the drawing stream data and the video stream data are delivered, the drawing process and the video decoding process must be performed at the same time.

However, according to the present invention described above,
Even when an application in which a plurality of image processing apparatuses are connected on a network and which requires real-time processing is assumed, it is possible to synchronize drawing and display in a display frame.

Another modification of the embodiment of the present invention will be described with reference to FIG. 1 again.

In the image processing apparatus of FIG. 1, the control unit 35
The priority detection unit 35c incorporated in A detects the priority that can be drawn by the image processing apparatus. Therefore, when the priority is updated, the control unit 35A notifies the CPU 1 of the updated priority by using a means such as the interrupt signal 1S. Based on this updated priority, for example, C
PU1 (corresponding to the stream data output unit) is DL2
In the case of generating 2, the image processing apparatus selects only a drawing object having a priority capable of drawing processing and outputs it to the LSI 3.

In this way, since the amount of DL generated can be adjusted, the amount of data transferred on the system bus 6 can be minimized. Therefore, it is possible to prevent the reduction of the data processing capability of the image processing apparatus related to the data transfer on the system bus 6.

Still another modification will be described with reference to FIG.

In the image processing apparatus of FIG. 1, first, CP
U1 creates DL22 with all priorities to create an LSI
Output to 3. In this case, the CPU interface 31
(Corresponding to the interface unit) receives the priority notification signal 31S including the drawing priority information which is the currently drawable priority from the control unit 35A, and partially decodes the DL 22 input via the system bus 6. Compare with. Then, the DL is reconfigured using only the drawing object having the currently drawable priority, and the reconfigured DL is temporarily held in the local memory 4 via the memory interface 32. Thereafter, the drawing processing unit 33A inputs the reconstructed DL 22, performs drawing processing, and outputs the drawing frame memory.

In this way, the drawing process is executed for all DLs 22 as in the conventional drawing process.
From the DL 22 input to the LSI 3, only object data having a drawable priority is selected and reconstructed. Then, since the reconfigured DL 22 is internally transferred, it is possible to prevent a reduction in the data processing capability of the image processing apparatus related to the internal transfer.

[0113]

As described above, according to the present invention, drawing stream data (D
A time stamp is added to L) and the first stream data is input. Therefore, it becomes possible to manage the time of the drawing process for the drawing object. Then, the nth time, which is the time axis of the display, is generated, and the nth time and the first time stamp are compared. Based on the comparison result, the first stream data can be drawn in synchronization with the display time axis.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration example of an image processing system according to first and second embodiments of the present invention.

FIG. 2 is a schematic diagram showing an array of image processing data according to the first and second embodiments of the present invention.

FIG. 3 is a diagram showing an example of combining display frames in the image processing apparatus according to the first embodiment of the present invention.

FIG. 4 is a diagram showing a configuration example of a car navigation system according to a second embodiment of the present invention.

FIG. 5 is a block diagram showing a configuration example of a car navigation device according to a second embodiment of the present invention.

FIG. 6 shows a digital A according to a second embodiment of the present invention.
It is a figure for demonstrating the synchronous process of V decoding and drawing.

FIG. 7 is a flowchart diagram for explaining a synchronization process of drawing and AV decoding according to the second embodiment of the present invention.

FIG. 8 is a block diagram showing a configuration example of a conventional image processing system.

FIG. 9 is a schematic diagram showing a data array of conventional image processing data.

FIG. 10 is a diagram showing an example of combining display frames in a conventional image processing apparatus.

[Explanation of symbols]

1 CPU (stream output unit) 2 main memory 3 LSI 4 local memory (corresponding to memory area) 5 monitor 6 system bus 31 CPU interface (corresponding to interface unit) 32 memory interface 33A drawing processing unit (corresponding to arithmetic unit) ) 33a Priority register 34 Video decoding unit (corresponding to arithmetic unit) 35A Control unit 35a Time stamp detection unit 35b Time stamp detection unit 35c Priority detection unit 35d Time counter 36 Display output unit 60 Drawing unit 50 Drawing control unit (Calculation) 51 rendering unit 4a DL and AV memory 4b display memory (corresponding to memory area) 60 drawing unit 61 AV stream data (second stream data) 61A access unit 64 decoding buffer 73a audio time stamp (second time stamp) 73v video time stamp (second time stamp) 73d drawing timestamp (first time stamp) 22 DL (first stream data) 22B drawing commands

   ─────────────────────────────────────────────────── ─── Continued front page    F-term (reference) 5B050 AA10 BA10 CA06 CA09 EA10                       EA24 FA02                 5C021 PA78 PA82 YB01 YB03 ZA03                 5C082 CB01 DA22 MM08

Claims (10)

[Claims] 1. A first-kind arithmetic processing is performed on first stream data having a first time stamp and a plurality of object data to be drawn in a display frame,
A calculation unit that draws the result in the memory area and an n-th time interval indicating the time interval for displaying the n-th display frame
A time is generated based on a display output synchronization signal,
A control unit that compares time with the first time stamp of the first stream data, and synchronizes drawing in the memory area with the nth time based on the comparison result. Image processing device. 2. The image processing apparatus according to claim 1, wherein the object data is added with a priority indicating the necessity of drawing in the memory area, and the arithmetic unit is configured to perform the operation based on the priority. An image processing apparatus, characterized by performing a first-type arithmetic processing. 3. The image processing apparatus according to claim 1, wherein the arithmetic unit further includes a second type of second stream data having a second time stamp and data to be decoded in a display frame. Arithmetic processing is performed, the result is set in the memory area, and the control unit further compares the n-th time with the second time stamp of the second stream data, and also compares the comparison result. In addition, the image processing apparatus, wherein the setting is synchronized with the n-th time. 4. The image processing apparatus according to claim 2, wherein the arithmetic unit is of a second type for the second stream data having a second time stamp and data to be decoded in the display frame. Arithmetic processing is performed, the result is set in the memory area, and the control unit further compares the n-th time with the second time stamp of the second stream data, and also compares the comparison result. In addition, the image processing apparatus, wherein the setting is synchronized with the n-th time. 5. The image processing apparatus according to claim 3, wherein the control unit performs processing at the nth time in the first time interval from the nth time to the (n + 1) th time. The arithmetic unit is controlled so as to complete the second type arithmetic processing of the second stream data, and the arithmetic unit performs the second stream data of the second stream data to be processed at the n-th time from the first time interval. Second required for second-type arithmetic processing
In a third time interval after subtracting the time interval of, the first type arithmetic processing of the first stream data is started, and the control unit is configured to perform the first type arithmetic processing within the third time interval. If not completed, the first type arithmetic processing of the first stream data to be processed at the time of the (n + 1) th time and thereafter is skipped, and the first stream data of the first stream data to be processed at the nth time is skipped.
An image processing apparatus, characterized in that the operation unit is instructed to continue the seed operation process. 6. The image processing apparatus according to claim 3, wherein the controller is configured to perform processing at the nth time in a first time interval from the nth time to the (n + 1) th time. The arithmetic unit is controlled so as to complete the second type arithmetic processing of the second stream data, and the arithmetic unit performs the second stream data of the second stream data to be processed at the n-th time from the first time interval. Second required for second-type arithmetic processing
In a third time interval after subtracting the time interval of, the first type arithmetic processing of the first stream data is started, and the control unit is configured to perform the first type arithmetic processing within the third time interval. When completed, the first type arithmetic processing of the first stream data to be processed at the (n + 1) th time is stopped from the completion of the first type arithmetic processing to the increment to the (n + 1) th time. An image processing apparatus for instructing the arithmetic unit. 7. The image processing apparatus according to claim 4, wherein the controller is configured to perform processing at the nth time in a first time interval from the nth time to the (n + 1) th time. The arithmetic unit is controlled so as to complete the second type arithmetic processing of the stream data of No. 2, and the arithmetic unit includes a priority register that holds a value of a predetermined priority, and from the first time interval. , The second required for the second type arithmetic processing of the second stream data to be processed at the n-th time.
In the third time interval after subtracting the time interval of, the first type arithmetic processing of the first stream data is started, and when the first type arithmetic processing is incremented to the (n + 1) th time, the first type arithmetic processing is completed. The value of the priority register is updated to the value of the priority of the existing object data, and the first-class arithmetic processing of the first stream data to be processed after the (n + 1) th time is the value held by the priority register. An image processing apparatus, which calculates object data to which the above priority value is added, and does not calculate the object data to which a value lower than the priority value held in the priority register is added. . 8. The image processing apparatus according to claim 4, wherein the controller is configured to perform processing at the nth time in a first time interval from the nth time to the (n + 1) th time. The arithmetic unit is controlled so as to complete the second type arithmetic processing of the stream data of No. 2, and the arithmetic unit includes a priority register that holds a value of a predetermined priority, and from the first time interval. , The second required for the second type arithmetic processing of the second stream data to be processed at the n-th time.
At a third time interval from which the first time interval is subtracted, the first type arithmetic processing of the first stream data is started, and the first type arithmetic processing has a priority equal to or higher than a value held by the priority register. The object data to which the value of is added is calculated, and the first type calculation processing is performed on the priority held by the priority register at the (n-1) th time until it is incremented at the (n + 1) th time. When completing the first type operation processing of the corresponding object data, until the n + 1 time is incremented,
In consideration of the priority value held by the priority register before the (n-1) th time, the priority register has a value lower than the priority value held by the priority register at the (n-1) th time. The object data corresponding to the priority is additionally calculated, and the value of the priority register is updated to the priority value of the object data for which the first type calculation processing is completed when incremented in the (n + 1) th time. An image processing device characterized by: 9. The image processing apparatus according to claim 8, wherein the control unit generates a value of the priority register to generate the first stream data when a value held by the priority register is updated. The stream data output unit to be output is notified, and the stream data output unit adds, from the first stream data, the priority value greater than or equal to the updated value held by the priority register. An image processing apparatus which selects and outputs object data. 10. The image processing apparatus according to claim 8, wherein the first stream data is input, and object data to which a priority value greater than or equal to an updated value held by the priority register is added is selected. The image processing apparatus further includes an interface unit that reconstructs the first stream data and holds the reconstructed first stream data in a buffer.