JP2013046319A - Image processing apparatus and image processing method - Google Patents

Abstract

【Task】
In the videophone device, once the connection is completed, video data is transferred regardless of whether the user is looking at the screen or not, and thus the network bandwidth, which is a finite resource, may be wasted.
[Solution]
In a videophone device, if the viewer's terminal analyzes the other party's video and determines that the other party is watching the screen, the compression rate of the video data to be sent is lowered (the bit rate is increased). When it is determined that the other party is not looking at the screen, the video data to be transmitted is increased in the compression rate (the bit rate is decreased), and the network bandwidth is appropriately controlled so as not to be consumed wastefully.
[Selection] Figure 1

Description

  The present technology relates to a videophone device that is a communication means using video and audio, and particularly to a control technology of video and audio data.

  In recent years, with the widespread use of networks and wider bandwidth, and the development of digital video and audio technologies including digital broadcasting, videophone devices, which are communication means using video and audio, are expected to spread to general households.

  In connection with this, an image compression communication method has been proposed in which the line-of-sight position including the region of interest with respect to the display screen of the participant is detected in real time and the cost of communication is reduced without making the participant aware of the deterioration of the image quality. ing.

JP 2006-54830 A Image compression communication method and apparatus

  However, in the above prior art, the line-of-sight information is received from the communication partner, and based on the received line-of-sight information, the video compression unit compresses the video compression rate so that the video compression rate becomes a low value (that is, a high data rate). Is output. Therefore, there is a problem that both systems that perform communication need to support the above-described method and that extra data transfer occurs because the line-of-sight information is mutually exchanged.

  The videophone device according to the present technology is characterized in that it has means for detecting line-of-sight information of a communication partner from a received video and controlling a bit rate of a video transmitted to the communication partner from the detected line-of-sight information.

  As a result, it is not necessary to acquire the line-of-sight information of the communication partner, and extra data transfer of the line-of-sight information does not occur. In addition, even when connected to a conventional videophone device that does not have a bit rate control function, the video bit rate control function does not work from the communication partner, but compatibility is ensured for system connectivity. Therefore, convenience for the user can be improved.

3 is an example of a functional block diagram of the image processing apparatus in Embodiment 1. FIG. 6 is an example of a flowchart of a line-of-sight detection process in the first embodiment. FIG. 6 is a diagram illustrating a first operation example according to the first embodiment. FIG. 10 is a diagram illustrating a second operation example according to the first embodiment. FIG. 10 is a diagram illustrating a third operation example according to the first embodiment. FIG. 10 is an example of a functional block diagram of an image processing apparatus according to a second embodiment. 12 is an example of a flowchart of voice detection processing in the second embodiment. 12 is an example of a functional block diagram of an image processing apparatus in Embodiment 3. FIG. 12 is an example of a flowchart of a line-of-sight detection process in Embodiment 3. FIG. 10 is an example of a functional block diagram of an image processing apparatus according to a fourth embodiment. 12 is an example of a flowchart of a line-of-sight detection process in Embodiment 4. It is an example of the rate setting in Example 4. FIG. 10 is a diagram illustrating an operation example in the fourth embodiment. 10 is an example of a functional block diagram of an image processing apparatus in Embodiment 5. FIG. 10 is an example of a flowchart of a line-of-sight detection process in the fifth embodiment. It is an example of the rate setting in Example 5. FIG. 10 is a diagram illustrating an operation example in the fifth embodiment. It is an example of the hardware block diagram of an image processing apparatus.

FIG. 18 is an example of a hardware configuration diagram of the image processing apparatus according to the present embodiment.
The image processing apparatus 300 includes a storage unit 1801, an external device control unit 1802, a communication unit 1803, a memory 1804, and a control unit 1805, which are connected by a hub 1806.

  The external device control unit 1802 is connected to an external device such as a microphone, a camera, a display, and a speaker, and controls the operation of the external device by communicating with the external device.

  A communication unit 1803 is connected to the network 109 and communicates with other image processing apparatuses, servers, and the like via the network. The image processing apparatus according to the present embodiment is connected to another image processing apparatus via the communication unit 1803, transmits video / audio to the other image processing apparatus, and receives video / audio from the other image processing apparatus. .

  The control unit 1805 can express each function of the image processing apparatus 300 by reading the software program stored in the storage unit 1801, developing the read software program in the memory 1804, and executing the software program. .

  The control unit 1805 executes various programs such as the audio signal processing program 103, the video signal processing program 104, and the audio encoding processing program described in FIG. 1, for example, thereby performing audio signal processing, video signal processing, audio encoding processing, and the like. Various processes are performed.

  However, in the following embodiments, the various processes are not limited to the control unit 1805, and each of the processes described in the embodiments such as the audio signal processing unit 103, the video signal processing unit 104, the audio encoding processing unit 105, and the like. It can also be realized by hardware such as an integrated circuit as a processing unit for performing each processing performed by the processing unit. It is also possible to integrate a plurality of processes or some processes performed by each processing unit into an integrated circuit. Therefore, the hardware configuration of the image processing apparatus in the following embodiments may be different from the example of FIG.

  In the following, for simplification of description, each processing realized by the control unit 1805 executing various programs will be described as being realized mainly by each processing unit. When each processing unit is realized by hardware, each processing unit mainly performs each process.

  Hereinafter, embodiments will be described in detail with reference to the drawings.

  First, the image processing apparatus of this embodiment will be described.

  FIG. 1 is a functional block diagram of the image processing apparatus according to the first embodiment. The image processing apparatus 300 includes an audio signal processing unit 103, a video signal processing unit 104, an audio encoding processing unit 105, a video encoding processing unit 106, an AV packet synthesis processing unit 107, a network processing unit 108, an AV packet separation processing unit 110, a video A decoding processing unit 111, an audio decoding processing unit 112, a display driver 113, an audio driver 114, a line-of-sight detection processing unit 117, and a bit rate setting processing unit 118 are connected to the network 109 via a communication unit 1803 and are connected to an external device control unit A microphone 101, a camera 102, a display 115, and a speaker 116 are connected via a 1802.

  The audio signal processing unit 103 quantizes the audio input from the microphone 101 and outputs it.

  The audio encoding processing unit 105 compresses and encodes the audio signal input from the audio signal processing unit 103.

  The video signal processing unit 104 quantizes the video input from the camera 102 and outputs it.

  The video encoding processing unit 106 compresses and encodes the video signal input from the video signal processing unit 104.

  The AV packet synthesis processing unit 107 multiplexes the compressed and encoded audio signal from the audio encoding processing unit 105 and the compressed video signal from the video encoding processing unit 106 and outputs the multiplexed AV signal as an AV packet.

  The network processing unit 108 outputs the AV packet from the AV packet synthesis processing unit 107 to the network 109 and also outputs the AV packet to the AV packet separation processing 110 from the network 109.

  The AV packet separation processing unit 110 separates the AV packet from the network processing unit 108 and outputs the compression-coded video signal to the video decoding 111 and the compression-coded audio signal to the audio decoding processing 112.

  The video decoding processing unit 111 decodes the compression-coded video signal from the AV packet separation 110 and outputs the decoded video signal to the display driver 113 and the line-of-sight detection processing 117.

  The audio decoding processing unit 112 decodes the compression-encoded audio signal from the AV packet separation 110 and outputs it to the audio driver 114.

  The display driver 113 and the audio driver 114 output the decoded video signal from the video decoding process 111 to the display 115 and the decoded audio signal from the audio decoding process 112 to the speaker 116, respectively.

  The line-of-sight detection processing unit 117 performs line-of-sight measurement processing, normal vision determination processing, bit rate setting processing, weight processing, and end determination processing. The line-of-sight measurement process is a process of measuring the range of the vertical, left, and right angles of the line of sight of the person in the output video signal of the video decoding process 111. The normal vision determination process is a process for determining whether or not the display or the camera is looking straight from the angle information obtained by the visual line measurement process. The bit rate setting process is a process of setting a bit rate value to the bit rate setting processing unit 118 according to the determination result obtained by the normal vision determination process. The wait process is a process waiting for execution for a certain period. The end determination process is a process for determining whether to return to the line-of-sight measurement process again or to end without returning to the line-of-sight measurement process.

  The bit rate setting processing unit 118 sets encoding parameters related to bit rate control of the video encoding processing 105 according to the result of the line-of-sight detection processing unit 117.

  First, the processing procedure for transmitting video / audio data from the image processing apparatus 300 will be described. The sound collected by the microphone 101 is converted into a digital signal by the sound signal processing unit 103, encoded by the sound encoding processing unit 105, and used as sound output data.

  The video imaged by the camera 102 is also converted into a digital signal by the video signal processing unit 104 and encoded by the video encoding processing unit 106 as video output data. The audio output data and the video output data are multiplexed by the AV packet synthesis processing unit 107, and the multiplexed video / audio data is transmitted to the image processing apparatus used by the communication partner via the network 109.

  Next, processing for outputting video / audio data received by the image processing apparatus 300 will be described. When the image processing apparatus 300 receives the video / audio data via the network 109, the video / audio data is separated into video output data and audio output data by the AV packet separation processing unit 110. The video output data is decoded by the video decoding process 111, converted into an output video signal by the display driver 113, and output to the display 115 as a video. The audio output data is also decoded by the audio decoding process 112, converted into an output audio signal by the audio driver 114, and output as audio to the speaker 116.

  At this time, the line-of-sight information is created by the line-of-sight detection processing unit 117 analyzing the image included in the video output from the video decoding processing unit 111. The bit rate setting processing unit 118 acquires the line-of-sight information created by the line-of-sight detection unit 117, and controls the encoding parameters of the video encoding processing unit 106 on the viewer side based on the acquired line-of-sight information.

  Next, the operation of the line-of-sight detection processing unit 117 in the first embodiment will be described using a flowchart. In the flowchart, “S” means “step”.

FIG. 2 is an example of a flowchart of line-of-sight detection processing in the first embodiment.
2, the line-of-sight detection processing unit 117 in FIG. 1 performs the line-of-sight detection processing with the video output data received from the video decoding processing unit 111 as an input, and the bit rate setting processing unit 118 performs viewing based on the result of the line-of-sight detection. 4 is a flowchart showing a method for controlling the bit rate of video output data as a parameter of video encoding processing of the video processing device 300 used by a person. This processing is realized by the visual line detection processing unit 117 performing analysis and determination using the output from the video decoding processing 111 as an input, and outputting the input of the bit rate setting processing 118.

  When the line-of-sight detection processing unit 117 is activated in synchronization with other functional blocks of the image processing apparatus 300, the line-of-sight detection processing unit 117 starts a line-of-sight detection process (S501), and sets an initial value bit rate to the bit rate setting processing unit 118. Processing is performed (S502).

  Next, the line-of-sight detection processing unit 117 analyzes the video output data received from the video decoding processing unit 111, performs a line-of-sight measurement process (S503), and executes a normal vision determination process from the measurement result (S504). If it is determined, the high bit rate setting process (S505) for setting a high bit rate value is performed, and if it is not determined to be normal, the low bit rate setting process (S506) for setting a low bit rate value is performed. This is performed for the setting processing unit 118.

  Further, wait processing for a certain time is executed (S507), execution program end determination processing is performed (S508), and if not ended, the process returns to the line-of-sight measurement process (S503) again. The detection process 117 is terminated (S509).

  FIG. 3 is a diagram illustrating a first operation example according to the first embodiment. It is assumed that an image processing apparatus 300A and an image processing apparatus 300B are connected via a network 109, and a display and a speaker are connected to each.

  In the diagram of the first operation example, both viewers of the image processing device 300A and the image processing device 300B are watching each other on the display screen.

  The operation of the line-of-sight detection processing unit 117 of the image processing apparatus 300A in FIG. 3 is as follows based on FIG. When the line-of-sight detection processing unit 117 of the image processing apparatus 300A is activated in synchronization with other functional blocks of the image processing apparatus 300A (S501), the line-of-sight detection processing unit 117 performs initial value bit rate setting processing on the bit rate setting processing unit 118. (S502). Next, the video signal data received from the video decoding processing unit 111 is analyzed, a line-of-sight measurement process is performed (S503), and a normal vision determination process is executed from the measurement result (S504).

  As a result of the normal vision determination process, when it is determined to be normal vision, a high bit rate setting process (S505) is performed on the bit rate setting process 118. Further, a wait process for a predetermined time is executed (S507), and an end determination process is performed (S508). In this example, since it is not completed, the process returns to the line-of-sight measurement process (S503) again. In this example, the image processing apparatus 300B performs exactly the same processing.

  As described above, in the case of FIG. 3, since both are paying attention to the display screens, the realism is emphasized, and the compression rate is controlled to be a low value (that is, the data rate is high). The

  FIG. 4 is a diagram illustrating a second operation example according to the first embodiment. Similarly to FIG. 3, it is assumed that an image processing apparatus 300A and an image processing apparatus 300B are connected via a network 109, and a display and a speaker are connected to each.

  In the diagram of the second operation example, the viewer of the image processing apparatus 300A is gazing at the display screen, but the viewer of the image processing apparatus 300B is watching the material at hand and viewing the display screen. This is an example of not paying attention.

  The operation of the line-of-sight detection processing unit 117 of the image processing apparatus 300A in FIG. 4 is as follows based on FIG. When the line-of-sight detection processing unit 117 of the image processing apparatus 300A is activated in synchronization with other functional blocks of the image processing apparatus 300A (S501), the line-of-sight detection processing unit 117 performs initial value bit rate setting processing on the bit rate setting processing unit 118. (S502). Next, the video signal data received from the video decoding processing unit 111 is analyzed, a line-of-sight measurement process is performed (S503), and a normal vision determination process is executed from the measurement result (S504).

  The line-of-sight detection processing unit 117 of the image processing apparatus 300A determines that the viewer of the image processing apparatus 300B is not viewing the display as a result of the normal vision determination process, and performs the low bit rate setting process (S506) as the bit rate setting process. To the unit 118. Further, a wait process for a predetermined time is executed (S507), and an end determination process is performed (S508). In this example, since the execution program has not ended, the process returns to the line-of-sight measurement process (S503) again.

  On the other hand, when the line-of-sight detection processing unit 117 of the image processing apparatus 300B is activated in synchronization with other functional blocks of the image processing apparatus 300B (S501), the initial value bit rate setting is performed with respect to the bit rate setting processing unit 118. Processing is performed (S502). Next, the video signal data received from the video decoding processing unit 111 is analyzed, a line-of-sight measurement process is performed (S503), and a normal vision determination process is executed from the measurement result (S504).

  The line-of-sight detection processing unit 117 of the image processing apparatus 300B determines that the viewer of the image processing apparatus 300A is looking at the display as a result of the normal vision determination process, and performs the high bit rate setting process (S505). This is performed on the processing unit 118. Further, a wait process for a predetermined time is executed (S507), an end determination process is performed (S508), and since the process is not ended, the process returns to the line-of-sight measurement process (S503) again.

  In the case of FIG. 4, the viewer of the image processing device 300 </ b> A is watching the display screen, but the viewer of the image processing device 300 </ b> B is not watching the display screen. Therefore, since there is no need for realism from the image processing device 300A to the image processing device 300B, the data transfer efficiency is emphasized, and control is performed so that the compression rate is a high value (that is, the data rate is low). Is done. On the contrary, the image processing apparatus 300B controls the image processing apparatus 300A so that the sense of reality is emphasized and the compression rate is a low value (that is, the data rate is high).

  FIG. 5 is a diagram of a third operation example according to the first embodiment. Similarly to FIG. 3, it is assumed that an image processing apparatus 300A and an image processing apparatus 300B are connected via a network 109, and a display and a speaker are connected to each.

  In the diagram of the third operation example, both viewers of the image processing device 300A and the image processing device 300B are not gazing at the display screen.

  The operation of the line-of-sight detection processing unit 117 of the image processing apparatus 300A in FIG. 5 is as follows based on FIG. When the line-of-sight detection processing unit 117 of the image processing apparatus 300A is activated in synchronization with other functional blocks of the image processing apparatus 300A (S501), the line-of-sight detection processing unit 117 performs initial value bit rate setting processing on the bit rate setting processing unit 118. (S502). Next, the video signal data received from the video decoding processing unit 111 is analyzed, a line-of-sight measurement process is performed (S503), and a normal vision determination process is executed from the measurement result (S504).

  The line-of-sight detection processing unit 117 of the image processing apparatus 300A determines that the viewer of the image processing apparatus 300B is not looking at the display as a result of the normal vision determination process, and performs the low bit rate setting process (S506). This is performed on the processing unit 118. Further, a wait process for a predetermined time is executed (S507), and an end determination process is performed (S508). In this example, since the execution program has not ended, the process returns to the line-of-sight measurement process (S503) again. In this example, the image processing apparatus 300B performs exactly the same processing.

  In the case of FIG. 5, since both viewers are not gazing at each other's display screen, there is no need for a sense of reality. Therefore, emphasis is placed on data transfer efficiency, and a value with a high compression rate (that is, a value with a low data rate). It is controlled to become.

  According to the present embodiment, the image processing apparatus 300 detects the line-of-sight state of the communication partner and controls the encoding parameter of the video encoding process to determine that the communication partner is not gazing at the display. Is a network line that is finite by operating to send video data having a high compression rate (ie, a low data rate) from one image processing device 300 to the other image processing device 300. Bandwidth can be used effectively.

  In this embodiment, the line-of-sight state of the communication partner is detected and the video encoding parameter is controlled inside the image processing apparatus, so that the image processing apparatus used by the communication partner controls the encoding parameter as in this embodiment. Even in the case of an image processing apparatus having no function, interoperability is ensured.

  However, when both the image processing apparatuses 300 have a function of controlling the video encoding parameter, if one of the image processing apparatuses 300 has an extremely high compression rate (that is, a value with a low data rate), the image quality is improved. The detection accuracy of the line-of-sight state of the communication partner decreases because the video frame rate becomes very low or the video frame rate becomes very low, and the compression rate is high (that is, the data rate is low) in spite of normal vision. Since the video signal is transmitted, it is necessary to set the compression rate so that the accuracy of the line-of-sight detection is equal to or higher than a predetermined accuracy.

  Another image processing apparatus of this embodiment will be described. In addition, about the function, process, etc. equivalent to Example 1, the same number is attached | subjected in drawing, and description is abbreviate | omitted.

  FIG. 6 is an example of a functional block diagram of the image processing apparatus according to the second embodiment. The image processing apparatus 301 according to the present exemplary embodiment includes a calibration control unit 119 and a memory 120 in addition to the image processing apparatus 300 according to the first exemplary embodiment.

  The calibration control unit 119 is a control unit for taking in the reference video as the comparison data of the line-of-sight detection processing in the line-of-sight detection processing unit 117 using the output video of the video decoding processing unit 111 in a state of looking forward in advance.

  The memory 120 holds the reference image by the calibration control unit 119.

  In addition, the line-of-sight detection processing unit 117 according to the present embodiment further performs line-of-sight comparison processing for comparing and determining the output video of the video decoding processing 111 and the reference video of the memory 120.

  Also in the present embodiment, as in the first embodiment, the line-of-sight detection processing unit 117 analyzes the video output data received from the video decoding processing unit 111 to acquire the line-of-sight information, and based on the acquired line-of-sight information, The encoding parameters of the video encoding processing unit 106 on the viewer side are controlled by the bit rate setting processing unit 118, but it is determined whether or not the display of the line-of-sight detection processing unit 117 is correctly viewed according to the installation location of the camera 102. An error may occur in Therefore, in the present embodiment, the calibration control unit 119 stores the output video of the video decoding processing unit 111 in a state of looking straight ahead in the memory 120 as comparison data for the visual line detection processing in the visual line detection processing unit 117. .

  Next, the operation of the line-of-sight detection process 117 in the second embodiment will be described using a flowchart. In the flowchart, “S” means “step”.

FIG. 7 is an example of a flowchart of line-of-sight detection processing in the second embodiment.
In FIG. 7, the line-of-sight detection processing unit 117 in FIG. 6 performs line-of-sight detection processing using the video output data received from the video decoding processing unit 111 as an input, and based on the result of the line-of-sight detection processing, the bit rate setting processing unit 118 It is a flowchart which shows the method which controls the process which sets the bit rate in the video encoding process of the video encoding process part 111, The said process performs analysis and determination by using the output from the video decoding process 111 as an input, Bit rate setting process This is realized by outputting 118 inputs.

  When the line-of-sight detection processing unit 117 is activated in synchronization with other functional blocks of the image processing apparatus 301, the line-of-sight detection processing unit 117 starts the line-of-sight detection processing (S501), and sets an initial value bit rate to the bit rate setting processing unit 118. Processing is performed (S502).

  Next, the line-of-sight detection processing unit 117 analyzes the video output data received from the video decoding processing unit 111, performs line-of-sight measurement processing (S503), and performs the measurement result and the line-of-sight detection processing stored in the memory 120 stored in advance. A line-of-sight comparison process is executed from the comparison data (S516). As a result of the line-of-sight comparison process, when it is determined that the line-of-sight of the comparison data and the video output data match, the high bit rate setting process (S505) is performed. When the line of sight is determined not to match, the low bit rate setting process is performed. (S506) is performed on the bit rate setting processing unit 118.

  Further, wait processing for a certain time is executed (S507), execution program end determination processing is performed (S508), and if not ended, the process returns to the line-of-sight measurement process (S503) again. The detection process 117 is terminated (S509).

  According to the present embodiment, in addition to the effects of the first embodiment, the image processing apparatus 301 has an error in the normal vision determination due to the installation location of the image processing apparatus 301 (particularly the position of the camera 102). By performing a calibration that records an image of the normal viewing state as a reference in advance, erroneous determination of the normal viewing state can be reduced.

  Another image processing apparatus of this embodiment will be described. In addition, about the function, process, etc. equivalent to Example 1, the same number is attached | subjected in drawing, and description is abbreviate | omitted.

  FIG. 8 is an example of a functional block diagram of the image processing apparatus according to the third embodiment. The image processing apparatus 302 according to the present exemplary embodiment includes a voice detection processing unit 121 in addition to the image processing apparatus 300 according to the first exemplary embodiment.

  The sound detection processing unit 121 detects the input level from the sound signal processing 103 and controls the bit rate setting processing 118.

  Also, the bit rate setting process 118 in the present embodiment controls the video encoding process 106 by the input from the line-of-sight detection process 117 and the input from the audio detection process 121.

  Also in the present embodiment, as in the first embodiment, the line-of-sight detection processing unit 117 analyzes the video output data received from the video decoding processing unit 111 to acquire the line-of-sight information, and based on the acquired line-of-sight information, The encoding parameter of the video encoding processing unit 106 on the viewer side is controlled by the bit rate setting processing unit 118. In this embodiment, the audio output data output from the audio signal processing unit 103 is added to the first embodiment. The bit rate setting processing unit 118 controls the encoding parameters of the video encoding processing unit 106 on the viewer side based on the audio information collected by the microphone 101 by the analysis by the audio detection processing unit 121.

  Since the operation of the line-of-sight detection processing unit 117 in the third embodiment is the same as that in the first embodiment, description thereof is omitted.

  Next, the operation of the voice detection processing unit 121 according to the third embodiment will be described with reference to a flowchart. In the flowchart, “S” means “step”.

FIG. 9 is an example of a flowchart of voice detection processing according to the third embodiment.
In FIG. 9, the audio detection processing unit 121 in FIG. 8 receives the audio transmitted from the video processing device used by the viewer as an input, and outputs the video as the parameter of the video encoding processing of the video processing device 300 used by the viewer. 6 is a flowchart illustrating a method for controlling the bit rate of data. The processing is realized by the voice detection processing unit 121 performing analysis and determination using the output from the voice signal processing unit 103 as an input and outputting it as one of the inputs of the bit rate setting processing unit 118.

  When the voice detection processing unit 121 is activated in synchronization with other functional blocks of the image processing apparatus 302, the voice detection processing unit 121 starts the voice detection processing (S510), and the level of the output voice output data input from the voice signal processing unit 103 Is measured (S511), and the level determination process of the measured audio data is performed (S512). When the level of the measured audio data exceeds the determination value, a high bit rate setting process (S513) is performed, and when the level does not exceed the determination value, nothing is done.

  Further, a wait process for a predetermined time is executed (S507), an execution program end determination process is executed (S514), and if it is not ended, the process returns to the sound measurement process (S511) again. The detection process 121 ends (S515).

  According to the present embodiment, in addition to the effects of the first embodiment, the image processing apparatus 302 receives the viewer's voice and controls the bit rate setting process as a parameter of the viewer's video encoding process. When the viewer starts a conversation, the user can immediately control the bit rate setting process, and the communication partner places a gaze on the monitor faster than the other party's eyes. In other words, a video with a high data rate) can be displayed.

  Further, according to the present embodiment, the image processing apparatuses 302A and 302B according to the present embodiment transmit images at a low bit rate, and it is difficult to detect the line of sight by the line-of-sight detection processing unit 117 of the image processing apparatuses 302A and 302B. Even in some cases, the viewer of the image processing device 302A can transmit a high bit rate video to the image processing device 302B by producing sound. The line-of-sight detection processing unit 117 of the image processing apparatus 302B can detect that the viewer of the image processing apparatus 302A is looking directly at the display by analyzing the received high bit rate video. In contrast, it is possible to control to transmit a high bit rate video.

  As described above, the image processing apparatus 302 according to the present exemplary embodiment can appropriately switch the bit rate even when the accuracy of line-of-sight detection is reduced by transmitting a low bit rate video. The amount of communication between them can be reduced.

  The above effect is achieved by pressing or selecting a switch provided on the main body of the image processing apparatus 302, a switch provided on a remote controller (not shown), a button displayed on a display and selectable by an input means such as a mouse. Even when the line-of-sight detection processing unit 117 performs high bit rate processing, the same can be obtained.

  Another image processing apparatus of this embodiment will be described. In addition, about the function, process, etc. equivalent to Example 1, the same number is attached | subjected in drawing, and description is abbreviate | omitted.

  FIG. 10 is an example of a functional block diagram of the image processing apparatus according to the fourth embodiment. The image processing apparatus 303 according to the present exemplary embodiment includes an audio signal processing unit 203, a video signal processing unit 204, an audio encoding processing unit 205, a video encoding processing unit 206, and an AV packet synthesis processing unit in addition to the image processing apparatus 300 according to the first exemplary embodiment. 207, a network processing unit 208, an AV packet separation processing unit 210, a video decoding processing unit 211, an audio decoding processing unit 212, a display driver 213, and an audio driver 214, and a microphone 201 and a camera 202 via an external device control unit 1802. , A display 215 and a speaker 216.

  Microphone 201, camera 202, audio signal processing unit 203, video signal processing unit 204, audio encoding processing unit 205, video encoding processing unit 206, AV packet synthesis processing unit 207, network processing unit 208, AV packet separation processing unit 210, video The description of the decoding processing unit 211, the audio decoding processing unit 212, the display driver 213, the audio driver 214, the display 215, and the speaker 216 is the same as the processing unit or device having the same name described in the first embodiment, and thus the description is omitted. To do.

  Next, the operation of the line-of-sight detection processing unit 117 according to the fourth embodiment will be described using a flowchart. In the flowchart, “S” means “step”.

FIG. 11 is an example of a flowchart of line-of-sight detection processing in the fourth embodiment.
In FIG. 11, the line-of-sight detection processing unit 117 in FIG. 10 performs the line-of-sight detection processing using the video output data received from the video decoding processing unit 111 as an input, and the bit rate setting processing unit 118 It is a flowchart which shows the method which controls the process which sets the bit rate in the video encoding process of the encoding process part 111, The said process performs analysis and determination by using the output from the video decoding process part 111/211 as an input, Bit rate This is realized by outputting the input of the setting processing unit 118.

  When the line-of-sight detection processing unit 117 is activated in synchronization with other functional blocks of the image processing apparatus 303, the line-of-sight detection processing unit 117 starts the line-of-sight detection processing (S501), and sets an initial value bit rate to the bit rate setting processing unit 118. Processing is performed (S502).

  Next, the line-of-sight detection processing unit 117 analyzes the video output data received from the video decoding processing unit 111/211, performs a line-of-sight measurement process (S503), and executes a normal vision determination process from the measurement result (S517). Depending on the result of the normal vision determination process, one of the rate setting 1 process (S518), the rate setting 2 process (S519), the rate setting 3 process (S520), and the rate setting 4 process (S521) in FIG. To 118.

  Further, wait processing for a certain time is executed (S507), execution program end determination processing is performed (S508), and if not ended, the process returns to the line-of-sight measurement process (S503) again. The detection process is terminated (S509).

  The table in FIG. 12 shows the bit rate setting processing unit 118 of the rate setting 1 process (S518), the rate setting 2 process (S519), the rate setting 3 process (S520), and the rate setting 4 process (S521) in FIG. It is an example of the combination of the setting parameter performed. In this example, the image sent from the site of (Screen A) is also sent from the site of (Screen B) by the eye gaze measurement processing (S503) and the eye gaze determination processing (S517) of FIG. However, if it is determined that the user is looking straight ahead, the rate setting 1 process is selected. Hereinafter, similarly, the image sent from the base of (Screen A) is viewed normally by the line-of-sight measurement process (S503) and the line-of-sight determination process (S517), but the image sent from the base of (Screen B) is viewed normally. If it is determined that it is not, the rate setting 2 process is selected, and the transmission image from the base of (Screen B) is looking straight, but the transmission image from the base of (Screen A) is not looking straight. If it is determined, the rate setting 3 process is selected, and if it is determined that neither the transmission image from the base of (Screen A) nor the transmission image from the base of (Screen B) is viewed, the rate is set. It shall operate | move so that the setting 4 process may be selected. In the table of FIG. 12, when the rate setting 1 process is selected by the line-of-sight detection processing unit 117, the bit rate setting processing unit 118 controls the transmission image quality control to “(screen A) of“ rate setting 1 ”. "High bit rate" is set, and "High bit rate" is set as "Transmission image quality control to the base of (screen B)". Similarly, when the rate setting 2 process is selected, the bit rate setting processing unit 118 sets “high bit rate” as “transmission image quality control to the base of (screen A)” of “rate setting 2”. When the rate setting 3 process is selected to set “low bit rate” as “image quality control for transmission to base of (screen B)”, “(screen A” of “rate setting 3” is selected. ) Set “Low bit rate” as “Transmission image quality control to base” and “Set high bit rate” as “Transmission image quality control to base (screen B)”. In the case of “Rate setting 4”, “Low bit rate” is set as “Transmission image quality control to the base of (screen A)” and “Transmission image quality control to the base of (screen B)” is set. , Indicates to set "low bit rate" .

  FIG. 13 is a diagram of an operation example in the fourth embodiment. It is assumed that an image processing device 303A, an image processing device 303B, and an image processing device 303C are connected via the network 109, and a display and a speaker are connected to each.

  In the diagram of this operation example, in the image processing device 303A, the image processing device 303B, and the image processing device 303C, the viewer of the image processing device 303A is watching the screen A and the image processing device 303B that is the base of the screen A. The viewer is also watching the viewer screen of the image processing apparatus 303A. In addition, the viewer of the image processing device 303A does not watch the screen B, but the viewer of the image processing device 303C that is the base of the screen B is watching the screen of the viewer of the image processing device 303A. is there.

  The operation of the line-of-sight detection processing unit 117 of the image processing apparatus 303A in FIG. 13 is as follows based on FIG. When the line-of-sight detection processing unit 117 of the image processing device 303A is activated in synchronization with other functional blocks of the image processing device 303A, the line-of-sight detection processing starts (S501), and the bit rate setting processing unit 218 Initial value bit rate setting processing is performed (S502). Next, the video signal data received from the video decoding processing unit 111/211 is analyzed, and a line-of-sight measurement process is performed (S503).

  In this example, rate setting 1 processing (S518) is performed on the bit rate setting processing unit 118 as a result of executing the normal vision determination processing (S517) from the measurement result of the line-of-sight measurement processing.

  Further, a wait process for a predetermined time is executed (S507), an execution program end determination process is executed (S508), and since the end is not ended, the process returns to the line-of-sight measurement process (S503) again. In this example, the image processing device 303B is exactly the same processing, and the image processing device 303C performs a line-of-sight measurement process (S503), and as a result of executing a normal vision determination process from the measurement result (S517), in this example The rate setting 4 process (S521) is performed on the bit rate setting processing unit 118.

  In the case of FIG. 13, the video / audio data transmitted between the image processing devices 303A and 303B both pay close attention to the display screens, and therefore the compression ratio is emphasized. It is controlled to be a low value (that is, a data rate is a high value), and between the image processing devices 303A and 303C, the image processing device 303A to the image processing device 303C emphasizes a sense of realism, and is compressed. The rate is controlled so as to be a low value (that is, a high data rate), but since there is no need for realism from the image processing device 303C to the image processing device 303A, the data transfer efficiency is emphasized. Control is performed so that the compression rate becomes a high value (that is, the data rate is a low value). Since the image processing apparatuses 303B and 303C are not gazing at the display screens of each other, and there is no need for realism, the data transfer efficiency is emphasized, and a value with a high compression rate (that is, the data rate) Is controlled to be a low value).

  According to this embodiment, since the video processing device 303 detects the line-of-sight state between the three parties and controls the video encoding parameters, it is not necessary to send unnecessary video data. Bandwidth can be used effectively.

  Another image processing apparatus of this embodiment will be described. In addition, about the function, process, etc. equivalent to Example 1, the same number is attached | subjected in drawing, and description is abbreviate | omitted.

  FIG. 14 is an example of a block diagram of an image processing apparatus according to the fifth embodiment. The image processing apparatus 304 according to the present exemplary embodiment includes an audio encoding processing unit 205, a video encoding processing unit 206, an AV packet synthesis processing unit 207, a network processing unit 208, and an AV packet separation processing unit in addition to the image processing apparatus 300 according to the first exemplary embodiment. 210, an image decoding processing unit 211, an audio decoding processing unit 212, a display driver 213, an audio driver 214, a calibration control unit 119, and a memory 120, and is connected to a display 215 and a speaker 216 via an external device control unit 1802. Yes.

  The description of the audio encoding processing unit 205, video encoding processing unit 206, AV packet synthesis processing unit 207, AV packet separation processing unit 210, video decoding processing unit 211, audio decoding processing unit 212, display driver 213, and audio driver 214 will be described. Since this is the same as the processing unit or device having the same name described in Example 1, the description is omitted.

  Next, the operation of the line-of-sight detection processing 217 in Embodiment 5 will be described using a flowchart. In the flowchart, “S” means “step”.

FIG. 15 is an example of a flowchart of line-of-sight detection processing in the fifth embodiment.
In FIG. 15, the visual line detection processing unit 117 in FIG. 14 performs visual line detection processing using the video output data received from the video decoding processing unit 111 as an input, and based on the result of the visual line detection processing, the bit rate setting processing unit 118 It is a flowchart which shows the method which controls the process which sets the bit rate in the video encoding process of the video encoding process part 111, The said process performs analysis and determination by using the output from the video decoding process part 111/211 as an input, Bit This is realized by outputting the input of the rate setting processing unit 118.

  When the line-of-sight detection processing unit 117 is activated in synchronization with other functional blocks of the image processing device 304, the line-of-sight detection processing unit 117 starts the line-of-sight detection processing (S501), and sets an initial value bit rate to the bit rate setting processing unit 118. Processing is performed (S502).

  Next, the line-of-sight detection processing unit 117 analyzes the video output data received from the video decoding processing unit 111/211, performs a line-of-sight measurement process (S503), and the line-of-sight detection stored in the memory 120 stored in advance. A normal vision determination process is executed from the comparison data of the process (S517), and the rate setting 1 process (S518), the rate setting 2 process (S519), and the rate setting 3 process (S520) of FIG. Any one of the rate setting 4 process (S521) and the rate setting 5 process (S522) is performed on the bit rate setting processing unit 118.

  Further, wait processing for a certain time is executed (S507), execution program end determination processing is performed (S508), and if not ended, the process returns to the line-of-sight measurement process (S503) again. The detection process is terminated (S509).

  The table of FIG. 16 shows the bits of rate setting 1 processing (S518), rate setting 2 processing (S519), rate setting 3 processing (S520), rate setting 4 processing (S521), and rate setting 5 processing (S522) of FIG. It is an example of the combination of the setting parameter performed with respect to the rate setting process part 118. FIG. The criteria for selecting the rate settings 1 to 4 by the line-of-sight detection processing unit 117 are the same as those in the fourth embodiment, but in this embodiment, which of the plurality of displays is being viewed, or the plurality of displays Since one camera determines whether none of them are viewed as shown in FIG. 17, the accuracy of line-of-sight detection decreases.

  Therefore, in this embodiment, when the line-of-sight detection processing unit 117 cannot detect the line of sight and when the viewer cannot determine which display among the plurality of displays, the line-of-sight is not detected. The detection processing unit 117 selects rate setting 5.

  When the rate setting 5 is selected, the bit rate setting processing unit 118 performs both “high quality image transmission control for (screen A) base” and “high quality image transmission for (screen B) base”. “Medium bit rate”, which is a bit rate value between 1 and a low bit rate.

  Further, when a plurality of people's eyes are detected, when a plurality of people are detected, or when a plurality of people's eyes are detected and a part of the eyes is determined to be normal, rate setting 5 May be selected. In the above case, a plurality of persons can communicate with each other using a single image processing apparatus 300. Further, if the rate setting 5 is selected when a plurality of persons are detected, the bit rate can be set without detecting the line of sight, so that the processing load on the line-of-sight detection processing unit 117 is reduced. In addition, when the line of sight of a plurality of persons is detected and a part of the lines of sight is determined to be normal, when the rate setting 5 is selected, the plurality of persons captured by the camera 102 are selected. By setting the rate setting to 4 when not all of them are looking straight ahead, the amount of communication between the image processing apparatuses can be further reduced.

  When rate setting 1 processing is selected by the line-of-sight detection processing unit 117, the bit rate setting processing unit 118 sets a high bit rate as the transmission image quality control to the base of (Screen A), and (Screen B) It shows that a high bit rate is set as the image quality control for sending to the base. Similarly, in the table of FIG. 16, the bit rate setting processing unit 118 sets a high bit rate as the transmission image quality control to the base in (screen A) when the rate setting 2 processing is selected, and (screen B ) To set a low bit rate as the transmission image quality control to the base of (), when the rate setting 3 process is selected, set the low bit rate as the transmission image quality control to the base of (Screen A) and (screen B) Setting a high bit rate as the transmission image quality control to the site of (B), and setting the low bit rate as the transmission image quality control to the site of (Screen A) when the rate setting 4 process is selected, ) To set the low bit rate as the transmission image quality control to the base of (), when the rate setting 5 process is selected, set the medium bit rate as the transmission image quality control to the base of (Screen A), and (screen B) Bases Indicates that, for setting the medium-bit rate as delivery quality control.

  FIG. 17 is a diagram of an operation example in the fifth embodiment. It is assumed that an image processing device 304A, an image processing device 304B, and an image processing device 304C are connected via the network 109, and a display and a speaker are connected to each.

  In this example, the operation is the same as in the case of the fourth embodiment except for the rate setting. Therefore, the description is omitted for the sake of brevity. Further, FIG. 17 shows the case where two monitors are used, but a similar system can be constructed with one monitor by using the two-screen display function of the monitor.

  According to the present embodiment, the video processing device 303 can effectively use the network bandwidth of the network between the three parties as in the fourth embodiment, and in this embodiment, the camera, the microphone, Since the video signal processing and audio signal processing portions are shared, the system cost can be reduced. On the other hand, in this embodiment, since the camera, microphone, video signal processing, and audio signal processing are shared, it is determined whether the viewer is watching the screen A or the screen B. In such a case, the medium bit rate can be set for both the base of (Screen A) and the base of (Screen B) by rate setting 5 processing. is there. In addition, when a plurality of people use the same image processing apparatus, there are cases in which a plurality of lines of sight are detected. It may be possible to set a medium bit rate for both.

  In the above embodiment, an image processing apparatus having an external display and speakers has been described. However, the above-described example can be implemented using a digital television receiver having a display and a speaker.

  The image processing apparatus includes an MCU (Multipoint Control Unit) having the same function as the image processing apparatus described in the above embodiment, and a plurality of image processing apparatuses that do not have a function of changing the bit rate. It is also possible to configure a TV conference system having a configuration in which each is connected to the MCU.

  In this case, each image processing apparatus transmits the video / audio data of each viewer to the MCU, and the video / audio data transmitted from the MCU is reproduced by a display and a speaker. The MCU performs line-of-sight detection processing, audio level determination processing, and the like included in the video / audio data transmitted from each image processing device, and changes the bit rate to each image processing device for transmission. With the above configuration, even in a TV conference system using an MCU, the communication volume of video / audio data transmitted to each image processing apparatus can be reduced in the same manner as in the above embodiment.

  In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. In addition, a part of the configuration of a certain embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of a certain embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

  Further, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

Reference Signs List 101 microphone 102 camera 103 audio signal processing unit 104 video signal processing unit 105 audio encoding processing unit 106 video encoding processing unit 107 AV packet synthesis processing unit 108 network processing unit 109 network 110 AV packet separation processing unit 111 video decoding processing unit 112 audio decoding Processing unit 113 Display driver 114 Audio driver 115 Monitor 116 Speaker 117 Gaze detection processing unit 118 Bit rate setting processing unit 119 Calibration control unit 120 Memory 121 Audio detection processing unit 201 Microphone 202 Camera 203 Audio signal processing unit 204 Video signal processing unit 205 Audio encoding processing unit 206 Video encoding processing unit 207 AV packet synthesis processing unit 208 Network processing unit 209 Network 210 AV packet separation processing unit 211 video decoding unit 212 sound decoding unit 213 the display driver 214 audio driver 215 monitors 216 the speaker 300 the image processing apparatus 301 image processing apparatus 302 image processing apparatus 303 image processing apparatus 304 image processing apparatus

Claims (6)

A first video processing unit that detects a line of sight of the first input video information and outputs a detection result;
A second video processing unit that compresses and encodes the second input video information based on the control input information and outputs the video information;
A conversion processing unit that converts the output of the first video processing unit into the control input information of the second video processing unit;
Input video information received from another device via a network is output as the first input video information of the first video processing unit, and the video information compressed and encoded by the second video processing unit is output. An image processing apparatus comprising: a network connection unit that transmits to the other device via the network. A first video processing unit that compares the video information previously held with the first input video information to detect a line of sight and outputs a detection result;
A second video processing unit for compressing and encoding the second input video information based on the control input information;
A conversion processing unit that converts the output of the first video processing unit into the control input information of the second video processing unit;
Input video information received from another device via a network is output as the first input video information of the first video processing unit, and the video information compressed and encoded by the second video processing unit is output. An image processing apparatus comprising: a network connection unit that transmits to the other device via the network. The image processing apparatus according to claim 1 or 2,
It has a voice processing unit that detects the characteristics of the input voice information and outputs the detection results.
The image processing apparatus, wherein the conversion processing unit converts an output of the audio processing unit into the control input information of the second video processing unit. A line-of-sight detection step of analyzing the first input video information and detecting a line-of-sight included in the video information;
Creating control information according to the analysis result of the first input video information;
Controlling the compression encoding of the second input video information according to the control information;
And outputting the input video information received via the network as the first input video information and transmitting the compressed and encoded second input video information to the network. Processing method. Comparing the first input video information with pre-stored video information to detect a line of sight included in the video information;
Creating control information according to the analysis result of the first input video information;
Controlling the compression encoding of the second input video information according to the control information;
And outputting the input video information received via the network as the first input video information and transmitting the compressed and encoded second input video information to the network. Processing method. Analyzing input speech information and detecting features included in the speech information;
A step of creating control information according to a detection result of the input audio information; a step of controlling compression encoding of second input video information according to the control information;
And outputting the audio information received via the network as the input audio information and transmitting the compressed and encoded second video information to the network.

Images