TWI889394B - Image processing device and image processing method - Google Patents

Abstract

An image processing device is provided. The image processing device is coupled to an external memory and includes a memory, a first processor, and a second processor. The first processor is configured to perform the following steps: loading a virtual machine monitor to the external memory; loading a first virtual machine to the external memory; waking up the second processor; and loading a second virtual machine to the external memory. The second processor is configured to perform the following steps: executing the virtual machine monitor; and executing the first virtual machine. The first processor first loads the first virtual machine and then loads the second virtual machine, and the loading time of the first virtual machine is less than the loading time of the second virtual machine.

Description

Image processing device and image processing method

The present invention relates to an image processing device, and more particularly to an image processing device and method for executing multiple virtual machines.

The boot speed of an electronic device is one of the indicators of the performance of the electronic device. In order to speed up the boot speed, conventional electronic devices often adopt the following methods: (1) replace a large and relatively complete operating system (hereinafter referred to as a rich OS, such as Linux, UNIX, Android, Windows, etc.) with a lightweight operating system (hereinafter referred to as a simple OS, such as a real-time operating system (RTOS) or a non-OS) (the complexity of a rich OS is greater than that of a simple OS); or (2) simplify a complex OS.

The disadvantage of method (1) is that the application porting is difficult due to the lack of software functions and hardware support of the simple operating system. The disadvantage of method (2) is that the simplification results are limited by the architectural design of the complex operating system, and the startup time that can be shortened is limited.

In view of the shortcomings of the prior art, one object of the present invention is to provide an image processing device and an image processing method to improve the shortcomings of the prior art.

An embodiment of the present invention provides an image processing device. The image processing device is coupled to an external memory and includes: a memory, a first processor and a second processor. The first processor is used to execute the following steps: loading a virtual machine monitor to the external memory; loading a first virtual machine to the external memory; waking up the second processor; and, loading a second virtual machine to the external memory. The second processor is used to execute the following steps: executing the virtual machine monitor; and, executing the first virtual machine. The first processor first loads the first virtual machine and then loads the second virtual machine, and the loading time of the first virtual machine is shorter than the loading time of the second virtual machine.

Another embodiment of the present invention provides an image processing device. The image processing device is coupled to an external memory and includes: a memory and a processor. The processor is used to execute the following steps: loading a virtual machine monitor to the external memory; loading a first virtual machine to the external memory; executing the virtual machine monitor; executing the first virtual machine; and loading a second virtual machine to the external memory. The processor first loads the first virtual machine and then loads the second virtual machine, and the loading time of the first virtual machine is less than the loading time of the second virtual machine.

Another embodiment of the present invention provides an image processing method. The image processing method is applied to an image processing device, which is coupled to an external memory, and the method includes: (A) loading a virtual machine monitor to the external memory; (B) loading a first virtual machine to the external memory; (C) executing the virtual machine monitor; (D) executing the first virtual machine; and (E) loading a second virtual machine to the external memory. Step (B) is earlier than step (E), and the loading time of the first virtual machine is less than the loading time of the second virtual machine.

The technical means embodied in the embodiments of the present invention can improve at least one of the shortcomings of the prior art. Therefore, compared with the prior art, the present invention can speed up the startup speed of the electronic device and the electronic device can execute a complete complex operating system.

The features, implementation and effects of the present invention are described in detail below with reference to the accompanying drawings.

The technical terms used in the following descriptions refer to the customary terms in this technical field. If this manual explains or defines some of the terms, the interpretation of those terms shall be based on the explanation or definition in this manual.

The disclosure of the present invention includes an image processing device and an image processing method. Since some of the components included in the image processing device of the present invention may be known components individually, the details of the known components will be omitted in the following description without affecting the full disclosure and feasibility of the device invention. In addition, part or all of the process of the image processing method of the present invention may be in the form of software and/or firmware, and can be executed by the image processing device of the present invention or its equivalent device. Without affecting the full disclosure and feasibility of the method invention, the following description of the method invention will focus on the step content rather than the hardware.

Please refer to FIG. 1, which is a functional block diagram of an embodiment of the electronic device of the present invention. The electronic device 100 includes an image processing device 101, an external memory 102, an image sensor 103, a microphone 104, and a transmission circuit 105. The image processing device 101 includes a processor 110, a processor 120, an image signal processor 130, an audio signal processor 140, a codec 150, and a memory 160. The processor 110 and the processor 120 can be general-purpose processors or special-purpose processors, etc., and the special-purpose processor can be a neural network processor or an intelligent processor, etc. The external memory 102 may be a dynamic random access memory (DRAM), and the memory 160 may be a static random access memory (SRAM).

The image sensor 103 and the microphone 104 are used to capture images and audio, respectively, and generate image data DV0 and audio data DA0, respectively. The image signal processor 130 is coupled to the image sensor 103, and is used to process the image data DV0 (including but not limited to calculating the brightness of the image data DV0 and controlling the exposure length of the image sensor 103 according to the brightness) to generate processed image data DV1. The audio signal processor 140 is coupled to the microphone 104, and is used to process the audio data DA0 (including but not limited to noise reduction and amplification) to generate processed audio data DA1. The encoder 150 is coupled to the image signal processor 130, the audio signal processor 140, the memory 160 and the external memory 102, and is used to encode the processed audio data DA1 and the processed image data DV1 to generate multimedia data Dout, and store the multimedia data Dout in the external memory 102. Then, the image processing device 101 (more specifically, the processor 110 or the processor 120) controls the transmission circuit 105 to read the multimedia data Dout from the external memory 102, and transmit the multimedia data Dout to a network or other electronic device (including but not limited to a storage device). The multimedia data Dout includes at least one frame of image.

The memory 160 can be used to temporarily store the intermediate data generated by the encoder 150 during the encoding process.

The processor 110 and the processor 120 may be two independent processors, two cores of the same processor, or two cores belonging to different processors. The processor 110 and the processor 120 both execute a virtual machine monitor VMM (virtual machine monitor, also known as Hypervisor). The virtual machine monitor VMM manages the virtual machines VM0 and VM1 executed by the processor 110 and the processor 120. More specifically, the virtual machine monitor VMM allocates hardware resources (e.g., processors) to the virtual machine VM0 or the virtual machine VM1 according to preset rules (e.g., virtual machine priority). The operating principle of the virtual machine monitor VMM is well known to those skilled in the art, and will not be described in detail.

In the following description, the priority of the virtual machine VM0 is higher than the priority of the virtual machine VM1, and the virtual machine VM0 and the virtual machine VM1 respectively execute the simple operating system and the complex operating system. The image sensor 103, the microphone 104, the image signal processor 130, the sound signal processor 140 and the encoder 150 are controlled by the virtual machine VM0, and the transmission circuit 105 is controlled by the virtual machine VM1.

Please refer to FIG. 2 , which is a flow chart of an embodiment of the image processing method of the present invention, comprising the following steps.

Step S210: The processor 110 loads the virtual machine monitor VMM to the external memory 102 (for example, loads the image file of the virtual machine monitor VMM). In some embodiments, the processor 110 completes step S210 by executing a bootloader.

Step S220: The processor 110 loads the virtual machine VM0 to the external memory 102 (eg, loads the image file of the virtual machine VM0). The processor 110 may complete step S220 by executing a startup program.

After step S220 is completed, the processor 110 wakes up the processor 120 to execute steps S250-S290, and continues to execute steps S230-S240 and S270-S290.

Step S230: The processor 110 loads the virtual machine VM1 to the external memory 102 (eg, loads the image file of the virtual machine VM1). In some embodiments, the processor 110 completes step S230 by executing a startup program.

It should be noted that because the complexity of the complex operating system is greater than that of the simple operating system (ie, the image file of the virtual machine VM1 is larger than the image file of the virtual machine VM0), the loading time of the virtual machine VM1 will be greater than the loading time of the virtual machine VM0.

Step S240: The processor 110 jumps to the virtual machine monitor VMM, that is, executes the virtual machine monitor VMM.

It should be noted that the processor 110 executes the virtual machine monitor VMM (step S240) after loading the virtual machine VM0 and the virtual machine VM1 (steps S220 and S230). In other words, the virtual machine monitor VMM of this case does not have the function of loading the virtual machine VM0 and the virtual machine VM1. Such a design can simplify the virtual machine monitor VMM to shorten the execution time of step S210 (i.e., further speed up the startup speed of the electronic device 100).

Step S250: The processor 120 jumps to the virtual machine monitor VMM.

Step S260: The processor 120 executes the virtual machine VM0. More specifically, the processor 120 executes three sub-steps, namely, steps S262, S264, and S266 in step S260.

Step S262: The processor 120 initializes the sound module (including the microphone 104 and the sound signal processor 140) and the image module (including the image sensor 103 and the image signal processor 130).

Step S264: The processor 120 controls the sound module to capture and process the sound data DA0, and controls the image module to capture and process the image data DV0.

Step S266: The processor 120 controls the encoder 150 to encode the processed audio data DA1 and the processed image data DV1 to generate an image IMG0 (a part of the multimedia data Dout), and stores the image IMG0 in the external memory 102. The image IMG0 is the first frame of the image generated by the electronic device 100 after startup.

After the processor 110 loads the virtual machine VM1 (step S230 ) and jumps to the virtual machine monitor VMM (step S240 ), the processor 110 and the processor 120 execute the virtual machine monitor VMM simultaneously. The virtual machine monitor VMM allocates the use rights of the processor 110 and/or the processor 120 to the virtual machine VM0 (i.e., executes the VM0 execution thread (step S270)) or the virtual machine VM1 (i.e., executes the VM1 execution thread (step S280)) according to the priorities of the virtual machine VM0 and the virtual machine VM1, or controls the processor 110 and the processor 120 to enter an idle state (i.e., executes an idle thread (step S290)).

For example, after the image IMG0 is generated, the virtual machine monitor VMM continues to allocate the hardware resources of the processor 120 to the virtual machine VM0 to generate more frames of images (step S270), and at the same time allocates the hardware resources of the processor 110 to the virtual machine VM1 to control the transmission circuit 105 to output the image IMG0 and subsequent images (step S280).

It should be noted that processor 120 is not exclusive to virtual machine VM0, and processor 110 is not exclusive to virtual machine VM1. For example, in certain cases (e.g., when there is no image to be transmitted in external memory 102), virtual machine monitor VMM can assign processor 110 and processor 120 to virtual machine VM0 at the same time. Similarly, virtual machine monitor VMM can assign processor 110 and processor 120 to virtual machine VM1 at the same time. In other words, both simple operating systems and complex operating systems can obtain computing power of more than one processor, which is more beneficial to resource scheduling of the processor.

As mentioned above, since the priority of the virtual machine VM0 is higher than that of the virtual machine VM1, when the virtual machine VM0 and the virtual machine VM1 simultaneously request hardware resources from the virtual machine monitor VMM, the virtual machine monitor VMM preferentially allocates the hardware resources to the virtual machine VM0.

When the virtual machine VM0 and the virtual machine VM1 do not need hardware resources, the virtual machine monitor VMM controls the processor 110 and the processor 120 to execute the idle thread (step S290).

Please refer to FIG. 3 , which is a sequence diagram corresponding to FIG. 2 . The processor 110 wakes up the processor 120 at time t1 . Between time t1 and time t2 , when the processor 120 is executing steps S250 and S260 , the processor 110 executes step S230 . Since the complex operating system is relatively large (i.e., the loading time of the virtual machine VM1 is relatively long), when the processor 110 finishes loading the virtual machine VM1 (step S230 ends), the processor 120 has already generated the image IMG0 (time t2 ). After generating the image IMG0, the processor 120 executes the virtual machine VM0 to generate more frames of images (step S270). After loading the virtual machine VM1, the processor 110 executes the virtual machine monitor VMM (step S240), and then executes the virtual machine VM1 (step S280). In step S280, the processor 110 controls the transmission circuit 105 to read the image IMG0 from the external memory 102 and transmit the image IMG0.

As shown in FIG3 , the electronic device 100 of the present invention has generated at least one image (image IMG0) before executing the complex operating system (ie, virtual machine VM1). Compared with the prior art, the present invention greatly increases the startup speed of the electronic device 100 and improves the user experience.

Please refer to FIG4 , which is a functional block diagram of another embodiment of the electronic device of the present invention. The electronic device 400 is similar to the electronic device 100 , except that, in the image processing device 401 , the number of processors is 1 (ie, processor 410 ).

Please refer to FIG. 5A-FIG. 5B, which are flowcharts of another embodiment of the image processing method of the present invention. The process of FIG. 5A-FIG. 5B is executed by the processor 410, wherein steps S510, S520, and S530 are respectively the same as steps S210, S220, and S250, and the sub-steps of step S540 are shown in FIG. 5B. Please refer to FIG. 5B, step S540 includes sub-steps S542, S544, S546, and S548, and steps S542, S544, S546, and S548 are respectively the same as steps S262, S264, S266, and S230. More specifically, while the processor 410 is generating the image IMG0 (steps S542 to S546), the processor 410 simultaneously loads the virtual machine VM1 in a time division multiplexing (TDM) manner (step S548). In other words, the processor 410 substantially executes step S548 and steps S542 to S546 simultaneously.

It should be noted that, since the complex operating system is relatively large (i.e., the loading time of the virtual machine VM1 is relatively long), when the processor 410 has finished loading the virtual machine VM1, the processor 410 has already generated the image IMG0. In other words, the end time of step S546 is earlier than the end time of step S548.

Although the above-mentioned embodiments are based on an image processing device, this is not a limitation of the present invention. Those skilled in the art can appropriately apply the present invention to other types of electronic components based on the disclosure of the present invention.

Although the embodiments of the present invention are described above, these embodiments are not intended to limit the present invention. A person having ordinary knowledge in the technical field may modify the technical features of the present invention according to the explicit or implicit contents of the present invention. All such modifications may fall within the scope of patent protection sought by the present invention. In other words, the scope of patent protection of the present invention shall be subject to the scope of the patent application defined in this specification.

100,400: electronic device 101,401: image processing device 102: external memory 103: image sensor 104: microphone 105: transmission circuit 110,120,410: processor 130: image signal processor 140: sound signal processor 150: encoder 160: memory DA0: sound data DA1: processed sound data Dout: multimedia data DV0: image data DV1: processed image data IMG0: image VM0,VM1: virtual machine VMM: virtual machine monitor S210, S220, S230, S240, S250, S260, S262, S264, S266, S270, S280, S290, S510, S520, S530, S540, S542, S544, S546, S548: Steps t1, t2: Time points

FIG. 1 is a functional block diagram of an embodiment of the electronic device of the present invention; FIG. 2 is a flow chart of an embodiment of the image processing method of the present invention; FIG. 3 is a timing diagram corresponding to FIG. 2; FIG. 4 is a functional block diagram of another embodiment of the electronic device of the present invention; FIG. 5A to FIG. 5B are flow charts of another embodiment of the image processing method of the present invention.

100: Electronic devices

101: Image processing device

102: External memory

103: Image sensor

104: Microphone

105: Transmission circuit

110,120:Processor

130: Image signal processor

140: Sound signal processor

150: Encoder

160:Memory

DA0: Sound data

DA1: Processed sound data

Dout: Multimedia data

DV0: Image data

DV1: processed image data

IMG0: Image

VM0,VM1:virtual machine

VMM: Virtual Machine Monitor

Claims (20)

An image processing device coupled to an external memory comprises: a memory; a first processor; and a second processor; wherein the first processor is used to execute the following steps: loading a virtual machine monitor to the external memory; loading a first virtual machine to the external memory; waking up the second processor; and loading a second virtual machine to the external memory; wherein the second processor is used to execute the following steps: executing the virtual machine monitor; and executing the first virtual machine; The first processor first loads the first virtual machine and then loads the second virtual machine, and the loading time of the first virtual machine is less than the loading time of the second virtual machine. The image processing device of claim 1, wherein the first virtual machine runs a first operating system, the second virtual machine runs a second operating system, and the complexity of the first operating system is less than that of the second operating system. The image processing device of claim 1, wherein the first virtual machine has a higher priority than the second virtual machine. The image processing device of claim 1 further comprises: an image signal processor for processing an image data to generate a processed image data; an audio signal processor for processing an audio data to generate a processed audio data; and an encoder coupled to the image signal processor and the audio signal processor to encode the processed image data and the processed audio data to generate an image; wherein the second processor executes the first virtual machine to control the image signal processor, the audio signal processor and the encoder, and when the second processor executes the first virtual machine, the first processor is loading the second virtual machine. An image processing device as claimed in claim 4, wherein the image is generated before the second virtual machine is loaded. The image processing device of claim 4 further comprises: a transmission circuit for transmitting the image; wherein the first processor executes the second virtual machine to control the transmission circuit. The image processing device of claim 1, wherein the first processor executes a startup program to load the second virtual machine, and the first processor executes the virtual machine monitor only after loading the second virtual machine. An image processing device is coupled to an external memory, comprising: a memory; and a processor for executing the following steps: loading a virtual machine monitor to the external memory; loading a first virtual machine to the external memory; executing the virtual machine monitor; executing the first virtual machine; and loading a second virtual machine to the external memory; wherein the processor first loads the first virtual machine and then loads the second virtual machine, and the loading time of the first virtual machine is less than the loading time of the second virtual machine. The image processing device of claim 8, wherein the first virtual machine executes a first operating system, the second virtual machine executes a second operating system, and the complexity of the first operating system is less than that of the second operating system. An image processing device as claimed in claim 8, wherein the first virtual machine has a higher priority than the second virtual machine. The image processing device of claim 8 further comprises: an image signal processor for processing an image data to generate a processed image data; an audio signal processor for processing an audio data to generate a processed audio data; and an encoder coupled to the image signal processor and the audio signal processor to encode the processed image data and the processed audio data to generate an image; wherein the processor executes the first virtual machine to control the image signal processor, the audio signal processor and the encoder, and the processor generates the image and loads the second virtual machine in a time-division multiplexing manner. An image processing device as claimed in claim 11, wherein the image is generated before the second virtual machine is loaded. The image processing device of claim 11 further comprises: a transmission circuit for transmitting the image; wherein the processor executes the second virtual machine to control the transmission circuit. An image processing device as claimed in claim 8, wherein the second virtual machine is loaded into the external memory when the processor executes the first virtual machine. An image processing method is applied to an image processing device, which is coupled to an external memory, and the method includes: (A) loading a virtual machine monitor to the external memory; (B) loading a first virtual machine to the external memory; (C) executing the virtual machine monitor; (D) executing the first virtual machine; and (E) loading a second virtual machine to the external memory; wherein step (B) is earlier than step (E), and the loading time of the first virtual machine is less than the loading time of the second virtual machine. The image processing method of claim 15, wherein the first virtual machine executes a first operating system, the second virtual machine executes a second operating system, and the complexity of the first operating system is less than that of the second operating system. The image processing method of claim 15, wherein the first virtual machine has a higher priority than the second virtual machine. The image processing method of claim 15 further comprises: (F) processing an image data to generate a processed image data; (G) processing an audio data to generate a processed audio data; and (H) encoding the processed image data and the processed audio data to generate an image; wherein step (F), step (G) and step (H) are substantially performed simultaneously with step (E). An image processing method as claimed in claim 18, wherein the image is generated before the second virtual machine is loaded. An image processing method as claimed in claim 15, wherein step (E) is a sub-step of step (D).

Images