JP2022071984A - Imaging device, control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it difficult to miss a subject to be imaged by a user.
SOLUTION: The detection of an image pickup means for photographing a subject, a drive means for driving the image pickup direction of the image pickup means, a first voice command for executing a specific process, and the first voice command is started. The driving means has a second voice command for detecting and a detecting means for detecting the second voice command, and the driving means limits the driving range according to the detection of the second voice command. An image pickup device characterized by.
[Selection diagram] FIG. 7

Description

The present invention relates to an image pickup device capable of recognizing voice commands.

In recent years, an imaging device capable of imaging the surroundings has been proposed. Such an image pickup device is intended to capture an image of a scene desired by a user by automatically capturing a subject in response to recognizing a voice command issued by the user.

Patent Document 1 discloses an image pickup device having an automatic shooting mode in which a pan / tilt or zoom is driven to automatically search for and shoot a subject, and manual shooting can be performed by a voice command in the automatic shooting mode. ..

Japanese Unexamined Patent Publication No. 2019-106694

Generally, in the process by voice command, for the purpose of preventing erroneous operation, the device that detects the voice command starts the detection of the command word before the voice command (command word) for executing the predetermined process. Detects voice commands (trigger words). However, Patent Document 1 does not consider the use of the trigger word in the image pickup apparatus. When such processing by voice commands is applied to the image pickup apparatus disclosed in Patent Document 1, the image pickup apparatus of Patent Document 1 detects a trigger word and a command word while automatically searching for a subject in the automatic shooting mode. Is assumed. Then, since the processing by the subsequent voice command is not specified at the time when the trigger word is detected, it is conceivable to continue the automatic shooting. However, when the command word issued by the user is a voice command instructing the subject to be imaged, the subject to be imaged by the user is the subject imaged by the image pickup device at the time when the trigger word is uttered. Is assumed. Nevertheless, if the automatic search continues even if the trigger word is detected, the subject that the user wants to capture will be out of the angle of view when the command word is detected, and the subject that the user wants to capture will be shot. There is a risk of missing it. Therefore, an object of the present invention is to make it difficult for a user to miss a subject to be imaged when inputting an image pickup instruction using a voice command.

In order to achieve the above object, the image pickup apparatus of the present invention includes an image pickup means for photographing a subject, a drive means for driving the image pickup direction of the image pickup means, and a first voice command for executing a specific process. , A second voice command for initiating detection of the first voice command, and a detection means for detecting, depending on the detection means detecting the second voice command. The driving means is characterized in that the driving range is limited.

According to the present invention, it is possible to make it difficult for a user to miss a subject to be imaged when inputting an image pickup instruction using a voice command.

(A) It is a figure for demonstrating an example of the appearance of the image pickup apparatus in 1st Embodiment. (B) It is a figure for demonstrating the operation of the image pickup apparatus in 1st Embodiment. It is a figure which shows the structure of the image pickup apparatus in the 1st Embodiment. It is a figure which shows the structure of the image pickup apparatus and the external apparatus in the 1st Embodiment. It is a figure which shows the structure of the external device in 1st Embodiment. It is a flowchart which shows the automatic photography processing in 1st Embodiment. It is a figure for demonstrating the area division in a photographed image in 1st Embodiment. It is a flowchart which shows the voice recognition processing in 1st Embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The embodiment described below is an example as a means for realizing the present invention, and may be appropriately modified or modified depending on the configuration of the apparatus to which the present invention is applied and various conditions. Moreover, each embodiment can be combined as appropriate.

[First Embodiment]
<Configuration of image pickup device>
FIG. 1 is a diagram showing a configuration of an image pickup apparatus according to the first embodiment.

The image pickup apparatus 101 shown in FIG. 1A is provided with an operation member including a power switch capable of performing an operation of switching the power on and off. The operation member also includes a touch panel.

The lens barrel 102 is a housing including an optical lens group and an image pickup element. The lens barrel 102 is attached to the image pickup apparatus 101. The tilt rotation unit 104 and the pan rotation unit 105 are rotation mechanisms capable of rotationally driving the lens barrel 102 with respect to the fixed portion 103. The tilt rotation unit 104 is, for example, a motor capable of rotating the lens barrel 102 in the pitch direction shown in FIG. 1 (b). The pan rotation unit 105 is, for example, a motor capable of rotating the lens barrel 102 in the yaw direction shown in FIG. 1 (b). The tilt rotation unit 104 and the pan rotation unit 105 can drive the lens barrel 102 to rotate in one or more directions. In this embodiment, the Y-axis shown in FIG. 1B is the rotation axis of the pan rotation unit 105. Further, in the present embodiment, the positive direction of the Z axis shown in FIG. 1B is the front direction of the image pickup apparatus 101.

The angular velocity meter 106 and the accelerometer 107 are, for example, a gyro sensor and an acceleration sensor, respectively, and are arranged in a fixed portion 103 of the image pickup apparatus 101. The image pickup device 101 detects the vibration of the image pickup device 101 based on the respective velocities measured by the angular velocity meter 106 and the accelerometer 107. The image pickup apparatus 101 can generate an image in which the shake and the tilt of the lens barrel 102 are corrected by rotationally driving the tilt rotation unit 104 and the pan rotation unit 105 based on the detected vibration of the image pickup apparatus 101.

FIG. 2 is a block diagram showing the configuration of the image pickup apparatus 101 of the present embodiment.

The first control unit 223 includes a processor (for example, CPU, GPU, microprocessor, MPU, etc.), a memory (for example, DRAM, SRAM, etc.) and the like. The first control unit 223 executes various processes to control each block of the image pickup apparatus 101, and controls data transfer between the blocks. The first control unit 223 is an example of a control means and a determination means.

The non-volatile memory 216 is a memory capable of recording and erasing data, and records constants, programs, and the like for the operation of the first control unit 223.

The zoom unit 201 is a group of optical lenses constituting a zoom lens that changes the zoom magnification. The zoom drive control unit 202 is a control unit that drives and controls the optical lens of the zoom unit 201. The focus unit 203 is a group of optical lenses for adjusting the focus. The focus drive control unit 204 drives and controls the optical lens of the focus unit 203. The image pickup unit 206 receives light incident from the image pickup element through each optical lens group, and outputs charge information corresponding to the amount of light to the image processing unit 207 as analog image data. The zoom unit 201, the zoom drive control unit 202, the focus unit 203, the focus drive control unit 204, and the image pickup unit 206 are included in the lens barrel 102.

The image processing unit 207 performs image processing such as distortion correction, white balance adjustment, and color interpolation processing on the image data input from the image pickup unit 206, and outputs digital image data. The digital image data output from the image processing unit 207 is converted by the image recording unit 208 by an image file format such as JPEG format or a moving image file format such as MPEG format. The converted digital image data is transmitted to the memory 215 and the video output unit 217 described later. When recording the digital image data recorded in the memory 215, the first control unit 223 outputs the digital image data to the recording / reproducing unit 220.

The lens barrel rotation drive unit 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to drive the lens barrel 102 in the tilt direction and the pan direction. The lens barrel rotation drive unit 205 is an example of the drive means.

The device shake detection unit 209 is equipped with, for example, an angular velocity meter 106 that detects the angular velocity in the triaxial direction of the image pickup device 101, and an accelerometer 107 that detects the acceleration in the triaxial direction of the device. The device shake detection unit 209 calculates the rotation angle of the device, the shift amount of the device, and the like based on the signals detected by the angular velocity meter 106 and the accelerometer 107.

The voice input unit 213 has a plurality of microphones. Further, the voice input unit 213 A / D-converts the voice signal input from the microphone and outputs it to the voice processing unit 214.

The voice processing unit 214 can detect the direction of sound on a plane on which the plurality of microphones are installed. The direction of the detected sound can be used for the search and automatic imaging described later. Further, the voice processing unit 214 can recognize a specific voice command. In this embodiment, there are two types of specific voice commands, a trigger word and a command word. The trigger word is a command that triggers the recognition of the command word. For example, a trigger word is a command consisting of a specific keyword such as "OK, camera" uttered by the user. Further, the command word is a command for instructing the image pickup apparatus 101 to perform a predetermined process. For example, this predetermined process is a still image imaging process, a moving image imaging start process, a moving image imaging end process, a sleep process, a subject change process, and an automatic image pickup process. Note that the command word differs for each predetermined process, for example, "take a still image" in the case of still image imaging processing, "take a moving image" in the case of moving image imaging start processing, and the like. It is a command consisting of keywords. These voice commands are pre-recorded in the memory 215 of the image pickup apparatus 101. In addition to the voice commands recorded in advance, the image pickup apparatus 101 may be configured so that voice commands for executing arbitrary processing by the user can be registered.

In addition, the voice processing unit 214 performs voice-related processing such as optimization processing and coding for the input voice signal. Then, the voice signal processed by the voice processing unit 214 is transmitted to the memory 215 by the first control unit 223. The memory 215 temporarily records the data input from the image recording unit 208 and the audio signal input from the audio processing unit 214. When recording this audio signal, the first control unit 223 outputs the audio signal from the memory 215 to the recording / reproducing unit 220.

The recording / reproducing unit 220 records image data, audio signals, other control data related to imaging, and the like on the recording medium 221. The recording medium 221 may be a recording medium built in the image pickup apparatus 101 or a removable recording medium. The recording medium 221 can record various data such as image data and audio signals. In the present embodiment, the recording medium 221 is a medium having a larger capacity than the non-volatile memory 216. For example, the recording medium 221 is a recording medium such as a hard disk, an optical disk, a magneto-optical disk, a CD-R, a DVD-R, a magnetic tape, a non-volatile semiconductor memory, and a flash memory.

Further, the recording / reproducing unit 220 can read (reproduce) the image data, the audio signal, various data, and the program recorded on the recording medium 221. When reproducing the image data and the audio signal recorded on the recording medium 221, the first control unit 223 operates as follows. The first control unit 223 outputs the image data and the audio signal read by the recording / reproducing unit 220 to the image processing unit 207 and the audio processing unit 214, respectively. The image processing unit 207 and the audio processing unit 214 decode the image data and the audio signal, respectively. The image processing unit 207 and the audio processing unit 214 output the decoded signal to the video output unit 217 and the audio output unit 218, respectively.

The second control unit 211 controls the power supply of the first control unit 223. For example, the second control unit 211 includes a processor (for example, a CPU, a microprocessor, an MPU, etc.), a memory (for example, a DRAM, a SRAM, etc.), and the like. In this embodiment, the second control unit 211 is provided separately from the first control unit 223 that controls the entire main system of the image pickup apparatus 101.

The first power supply unit 210 and the second power supply unit 212 supply electric power for operating the first control unit 223 and the second control unit 211, respectively. In the present embodiment, the electric power supplied by the first power supply unit 210 is larger than the electric power supplied by the second power supply unit 212. In the present embodiment, the first power supply unit 210 and the second power supply unit 212 are selected according to the amount of electric power to be supplied. For example, the first power supply unit 210 is a switch for supplying electric power to the first control unit 223, and the second power supply unit 212 is a lithium battery or an alkaline dry battery. By pressing the power switch provided on the image pickup apparatus 101, electric power is first supplied to the second control unit 211, and then power is supplied to the first control unit 223.

Further, the image pickup apparatus 101 has a sleep state. In the sleep state, the first control unit 223 controls the first power supply unit 210 so as to turn off the power supply to the first control unit 223. Even in the sleep state in which power is not supplied to the first control unit 223, the second control unit 211 is operating and acquires information from the device shake detection unit 209 and the voice processing unit 214. Based on such input information, the second control unit performs a determination process of whether or not to activate the first control unit 223. When the second control unit 211 determines that the first control unit 223 is activated (releases the sleep state), the second control unit 211 controls the first power supply unit 210 to supply electric power to the first control unit 223.

The audio output unit 218 outputs an audio signal such as an electronic shutter sound from a speaker built in the image pickup apparatus 101, for example, at the time of imaging. The LED control unit 224 controls the LED provided in the image pickup apparatus 101 to light or blink in a preset pattern, for example, at the time of image pickup.

The video output unit 217 is composed of, for example, a video output terminal, and outputs an image signal for displaying a video on a connected external display or the like. Further, the audio output unit 218 and the video output unit 217 may be an integrated terminal such as an HDMI (registered trademark) (High-Definition Multimedia Interface (registered trademark)) terminal.

The communication unit 222 is an interface for communicating between the image pickup device 101 and the external device. The communication unit 222 transmits and receives data such as audio signals and image data, for example. Further, when the communication unit 222 receives control signals related to image pickup such as image pickup start, image pickup end, pan drive, tilt drive, and zoom drive, the first control unit 223 receives the image pickup device 101 according to the control signals. To drive. The communication unit 222 has, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a WirelessUSB, and a GPS receiver.

The subject detection unit 225 reads the image data output from the image processing unit 207 from the memory 215 and recognizes a subject such as a person or an object. For example, when the subject detection unit 225 recognizes a person, the subject's face is detected. A pattern for determining the face of the subject is registered in advance in the image pickup apparatus 101. An identifier for distinguishing each subject is given to this pattern. In the subject face detection process, the subject detection unit 225 detects the face of the subject by detecting a portion matching the pattern for determining the face of the subject included in the captured image. In addition, the subject detection unit 225 can distinguish between a plurality of registered persons.

In addition, the subject detection unit 225 also calculates the reliability indicating the certainty of the detected subject on the face at the same time. The reliability is calculated from, for example, the size of the face region in the image, the degree of matching with the face pattern, and the like. Further, the subject detection unit 225 performs pattern matching on the face of the subject in the image to determine whether the detected face is a smile, whether the eyes are open, the orientation of the face, and the like. Face information can be detected. The method for detecting face information is not limited to pattern matching, and known techniques such as a method using deep learning can be used. The subject detection unit 225 is an example of the detection means.

Further, in the object recognition process, the subject detection unit 225 can recognize the object by determining whether or not it matches the pattern registered in advance. In addition, the subject detection unit 225 can recognize an object by extracting a feature amount of the subject by using a histogram of hue, saturation, etc. in the captured image.

By the above method, the first control unit 223 can detect a subject from the captured image data by the subject detection unit 225.

<System configuration with external device>
FIG. 3 is a diagram showing a configuration example of a wireless communication system between the image pickup device 101 and the smart device 301. The image pickup device 101 is, for example, a digital camera. The smart device 301 is, for example, a smartphone including a Bluetooth communication module and a wireless LAN communication module.

In the present embodiment, the image pickup apparatus 101 and the smart device 301 can communicate with each other via two communication paths. One is, for example, a communication path 302 by a wireless LAN conforming to the IEEE802.11 standard series. The other is a communication path 303 having a master-slave relationship between a control station and a subordinate station, such as Bluetooth Low Energy (hereinafter referred to as "BLE"). The wireless LAN and BLE are examples of communication methods. It should be noted that each communication device has two or more communication functions, and it is possible to control the other communication function by one communication function that communicates in a relationship between a control station and a subordinate station, for example. If so, other communication methods may be used. However, without losing generality, the first communication such as wireless LAN can perform higher-speed communication than the second communication such as BLE, and the second communication consumes more than the first communication. It shall be at least one of low power consumption and short communication range.

<Configuration of external device>
The configuration of the smart device 301, which is an example of the external device, will be described with reference to FIG. The smart device 301 is a so-called smartphone or mobile phone. The smart device 301 is a portable terminal.

The smart device 301 has, for example, a wireless LAN control unit 401 for wireless LAN communication, a BLE control unit 402 for BLE communication, and a public line control unit 406 for public wireless communication. Further, the smart device 301 has a packet transmission / reception unit 403.

The wireless LAN control unit 401 performs RF control of the wireless LAN, communication processing, and various controls of communication by the wireless LAN conforming to the IEEE802.11 standard series. Further, the wireless LAN control unit 401 performs protocol processing related to communication by wireless LAN. The BLE control unit 402 performs RF control of BLE, communication processing, and various controls of communication by BLE. Further, the BLE control unit 402 performs protocol processing related to communication by BLE. The public line control unit 406 performs RF control of public wireless communication, communication processing, and various controls of public wireless communication. In addition, the public line control unit 406 performs protocol processing related to public wireless communication. Public wireless communication is compliant with, for example, the IMT (International Multimedia Telecommunications) standard and the LTE (Long Term Evolution) standard. The packet transmission / reception unit 403 performs processing for executing at least one of transmission and reception of packets related to wireless LAN communication, BLE communication, and public wireless communication. In the present embodiment, the smart device 301 is described as performing at least one of transmission and reception of packets in communication, but other communication formats such as circuit switching are used in addition to packet switching. It is also good.

Further, the smart device 301 of the present embodiment further includes a control unit 411, a recording unit 404, a GPS receiving unit 405, a display unit 407, an operation unit 408, a voice input voice processing unit 409, and a power supply unit 410. The control unit 411 controls the entire smart device 301, for example, by executing the program recorded in the recording unit 404. The recording unit 404 records, for example, a program executed by the control unit 411 and various information such as parameters required for communication. Various operations of the smart device 301, which will be described later, are realized by the control unit 411 executing the program recorded in the recording unit 404.

The power supply unit 410 supplies electric power to each unit of the smart device 301. The display unit 407 has a function capable of outputting visually recognizable information such as an LCD or LED and outputting sound from a speaker or the like, and displays and outputs various information. The operation unit 408 is, for example, a button or the like that accepts an operation on the smart device 301 by a user. The display unit 407 and the operation unit 408 may be configured by a common member such as a touch panel.

Voice input The voice processing unit 409 has, for example, a microphone, and performs voice recognition processing on a voice signal input from the microphone. Voice input The voice processing unit 409 can recognize the user's operation on the image pickup device 101 by voice. In this voice recognition process, the voice input voice processing unit 409 recognizes a voice command issued by the user from the voice signal input from the microphone by a dedicated application. The smart device 301 can register a voice command for causing the voice processing unit 214 of the image pickup device 101 to execute a specific process, for example, via a communication path 302, in the image pickup device 101.

The GPS (Global Positioning System) 405 estimates the current position (longitude and latitude information) of the smart device 301 by analyzing a GPS signal received from a satellite. When the estimated current position is located within a preset range (within a predetermined radius range) such as at home, the smart device 301 notifies the image pickup device 101 of the current position information via the BLE control unit 402. do. The image pickup apparatus 101 can be used as a parameter for automatic imaging and automatic editing, which will be described later, based on this current position. In addition, when there is a position change such as when the current position deviates from the preset range, the movement information is notified to the image pickup device 101 via the BLE control unit 402 and used as a parameter for automatic image pickup and automatic editing. do. The smart device 301 may estimate the current position of the smart device 301 based on the information of the wireless network existing in the surroundings by using WPS (Wi-Fi Positioning System) or the like.

As described above, the image pickup device 101 and the smart device 301 exchange data with the image pickup device 101 by communication using the wireless LAN control unit 401 and the BLE control unit 402. For example, the image pickup device 101 and the smart device 301 transmit and receive data such as audio signals and image data. Further, the smart device 301 can transmit the setting information related to the image pickup of the image pickup device 101 to the image pickup device 101. Further, the smart device 301 can transmit a control signal related to the image pickup process of the image pickup device 101 and position information to the image pickup device 101.

<Automatic imaging process>
The automatic image pickup process is a process of determining the timing at which the first control unit 223 takes an image and automatically taking an image of the subject. In this automatic image pickup process, the first control unit 223 automatically determines and captures a subject to be imaged when it is determined that a good image or a good moving image can be captured or when a certain amount of time has passed. repeat. As a result, the user can leave a good scene suddenly appearing in daily life or a casual change in daily life by the image pickup device 101 without manually taking an image.

FIG. 5 is a flowchart of the automatic image pickup process of the image pickup apparatus 101 according to the present embodiment. The process of this flowchart is started in response to the user turning on the power switch of the image pickup apparatus 101. In this embodiment, a wireless connection is established between the image pickup device 101 and the smart device 301. In addition, the user can perform various operations on the image pickup apparatus 101 from a dedicated application on the smart device 301. The processing of each step in the following flowchart is realized by the first control unit 223 controlling each unit of the image pickup apparatus 101.

In step S501, the first control unit 223 determines whether or not the automatic imaging process is stopped. The stop of the automatic imaging process will be described later in the description of the voice recognition process. When the automatic image pickup process is stopped, the first control unit 223 waits until the stop of the automatic image pickup process is released. That is, when the automatic image pickup process is stopped, the process of step S501 is repeated until the stop of the automatic image pickup process is canceled. If the automatic imaging process is not stopped, the process of step S502 is executed.

In step S502, the first control unit 223 causes the image processing unit 207 to process the signal captured by the image pickup unit 206 to generate an image for subject recognition. Further, the first control unit 223 controls the subject detection unit 225 to perform subject recognition such as person recognition and animal recognition from the generated image for subject recognition. For example, when performing subject recognition, the first control unit 223 holds a pattern for determining the subject in advance, and determines the degree of matching between the held pattern and the pattern included in the image for subject recognition. Based on this, the subject detection unit 225 determines the subject. Here, the subject detection unit 225 personally identifies the subject. In addition, the first control unit 223 determines the subject and detects the position within the angle of view of the subject.

In step S503, the first control unit 223 calculates the image shake correction amount. Specifically, first, the first control unit 223 calculates the absolute angle of the image pickup device based on the angular velocity and acceleration information acquired by the device shake detection unit 209. Then, the first control unit 223 obtains the vibration isolation angle for moving the tilt rotation unit 104 and the pan rotation unit 105 in the angle direction for canceling the absolute angle, and uses this as the image shake correction amount.

In step S504, the first control unit 223 performs subject search processing. The subject search process is composed of the following processes.

(1) Area division The area division will be described with reference to FIGS. 6 (a) to 6 (c). In FIGS. 6A to 6C, the area on the spherical surface is divided with the position of the image pickup apparatus as the origin O. In the example of FIG. 6A, the areas are divided by 22.5 degrees in the tilt direction and the pan direction, respectively. When divided as shown in FIG. 6A, the circumference in the horizontal direction becomes smaller and the area of one area becomes smaller as the angle in the tilt direction deviates from 0 degrees. Therefore, as shown in FIG. 6B, the image pickup apparatus of the present embodiment sets the horizontal area range to be larger than 22.5 degrees when the tilt angle is 45 degrees or more.

Next, the area within the angle of view of the image captured by the image pickup apparatus 101 will be described with reference to FIGS. 6 (c) and 6 (d). The axis 1301 is a reference direction of the image pickup direction of the image pickup apparatus 101, and the area division is performed with reference to this direction. The axis 1301 is, for example, a direction defined in advance as an image pickup direction when the image pickup apparatus 101 is activated, or a direction as a reference for the image pickup direction. The area 1302 is an angle of view area of the image captured by the image pickup unit 206. FIG. 6D is an example of a through image captured by the imaging unit 206 in the area 1302. Within the angle of view of the through image of FIG. 6 (d), the area of the image is divided into areas 1303 to 1318 based on the area division shown in FIG. 6 (c).

(2) Calculation of importance level for each area For each area divided as described above, the importance level indicating the priority when searching for a subject is set according to the subject existing in the area and the scene situation of the area. calculate. The importance level based on the situation of the subject is, for example, the number of subjects existing in the area, the size of the subject's face, the orientation of the subject's face, the certainty of the subject's face detection, the facial expression of the subject, and the individual of the subject. Calculated based on the identification result. In addition, the importance level according to the situation of the scene is, for example, general object recognition result, scene judgment result (blue sky, backlight, evening scene, etc.), sound level or voice recognition result from the direction of the area, motion detection in the area. Information etc. Here, the first control unit 223 is driven so as to search the entire circumference of the image pickup apparatus 101.

Further, for example, when the face of the subject is registered, the first control unit 223 raises the importance level of the area where the face of the registered subject is detected. For example, the face of the subject is recorded in the non-volatile memory 216 as a pattern for determining the subject. When increasing the importance level of the area where the face of the subject is detected, the first control unit 223 sets the importance level of the area according to the elapse of a predetermined time or the imaging of a predetermined number of times. To return to the original importance level.

(3) Determination of search area The first control unit 223 determines the importance level of each area as described above, and then determines to focus on the area having a high importance level. Then, the first control unit 223 calculates the pan angle and the tilt angle required to image one of the areas having a high importance level.

In step S505, the first control unit 223 performs pan drive and tilt drive. Specifically, the pan drive amount and the tilt drive amount are calculated based on the image shake correction amount and the pan angle and tilt angle calculated in step S504. Then, the first control unit 223 drives and controls the tilt rotation unit 104 and the pan rotation unit 105 by the lens barrel rotation drive unit 205 based on the calculated pan drive amount and tilt drive amount, respectively. In the present embodiment, the first control unit 223 will be described as detecting a subject in an area having a high importance level by driving in step S505 and starting imaging of the subject. Then, the first control unit 223 controls the lens barrel rotation drive unit 205 so as to track the subject (continue to keep it within the angle of view).

In step S506, the first control unit 223 controls the zoom unit 201 to drive the zoom. For example, the zoom is driven according to the state of the subject that started imaging in step S505. For example, when the face of the subject is captured very small within the angle of view, the first control unit 223 zooms to the telephoto side so that the size of the face of the subject is properly (larger) within the angle of view. Control to be imaged. On the other hand, when the face of the subject is imaged very large within the angle of view, the first control unit 223 zooms to the wide-angle side so that the size of the face of the subject is appropriately (smaller) within the angle of view. Control to be imaged. By performing zoom control in this way, it is possible to maintain a state suitable for tracking the subject.

In steps S504 to S506, a method of performing subject search by pan drive, tilt drive, and zoom drive has been described, but the subject search is performed by an imaging system that uses a plurality of wide-angle lenses to capture images in all directions at once. You may be broken. In this case, the overall processing load is large in order to perform image processing such as subject detection using all the signals obtained by omnidirectional imaging as input images. Therefore, in this case, a part of the image obtained by omnidirectional imaging is cut out, and the subject search process is performed within the cut out image range. In this configuration, the first control unit 223 calculates the importance level for each area in the same manner as the above-mentioned method, changes the cutting position based on the importance level, and makes a determination of automatic imaging described later. This enables high-speed subject search while suppressing power consumption due to image processing.

In step S507, the first control unit 223 determines whether or not to perform automatic imaging.

Here, the determination of whether or not to perform automatic imaging will be described. The determination of whether or not to perform automatic imaging is performed based on whether or not the imaging score exceeds a predetermined value. The image pickup score is a parameter used for determining whether or not to perform automatic image pickup. Scores are added to the imaging score according to the detection status of the subject and the passage of time. For example, consider a case of designing so that automatic imaging is performed when the imaging score exceeds 2000 points. In this case, first, the initial value of the imaging score is 0, and points are added according to the passage of time from the time when the automatic imaging mode is entered. For example, the imaging score increases at an increasing rate so as to reach 2000 points after 120 seconds. At this time, if 120 seconds elapse without detecting the subject, the number of points added by the passage of time reaches 2000 points, and imaging is performed. Further, if a subject having a high priority is detected during the lapse of time, 1000 points will be added. For example, the subject having a high priority is a subject set by the user as a target to be imaged with priority among the subjects whose faces are registered in the image pickup apparatus 101. In a state where a subject having a high priority is detected, it is easy to reach 2000 points, and as a result, the imaging frequency tends to increase.

Further, for example, when the smiling face of the subject is recognized, 800 points are added. The points added based on this smile are added even if the subject is not a high-priority subject. Further, in the present embodiment, the points to be added based on the smile will be described by taking as an example the case where the points are the same regardless of whether or not the subject has a high priority, but the present invention is not limited to this. For example, the score added according to the detection of the smile of the subject having a high priority may be higher than the score added according to the detection of the smile of the subject having a low priority. By doing so, it becomes possible to perform imaging more in line with the user's intention. If the number of points exceeds 2000 due to the addition of points due to changes in facial expressions such as emotions of these subjects, automatic imaging is performed. In addition, even if the points added due to the change in facial expression do not exceed 2000 points, the points added after the passage of time reach 2000 points in a shorter time.

The points added over time will be described by taking as an example a case where points are added so as to reach 2000 points in 120 seconds, a case where only 2000/120 points are added every second, that is, a case where points are added linearly with respect to time. Is not limited to this. For example, points may not be added up to 110 seconds out of 120 seconds, but may be increased so that points are added by 200 points per second to reach 2000 points in 10 seconds from 110 seconds to 120 seconds. By doing so, it is possible to prevent the points added due to the change in the facial expression of the subject from reaching the points to be imaged regardless of the high or low priority. In the case of the point addition method that linearly increases with the passage of time, the points are already added over time for a long time, so even if the points are added due to the change to the smile of the subject with low priority, the points to be imaged will be reached. In many cases, it is difficult to reflect the high and low priorities. However, if the score of points added due to changes in facial expressions is lowered, the timing of changes in facial expressions will be missed, so it is desirable to avoid measures by lowering the points of points added. Therefore, do not add points until 110 seconds. By doing so, 110 seconds elapse without adding points to the subject having a low priority. On the other hand, for a subject having a high priority, 1000 points are added at the time of detection, so that 1000 points are added even if there is no point added due to the passage of time up to 110 seconds. As a result, when points are added due to changes in facial expressions, the possibility that a low-priority subject will reach the number of points to be imaged can be suppressed compared to a high-priority subject, and the high and low priorities function. Cheap. In the above explanation, the change in facial expression is taken as an example, but other criteria for adding points may be when the voice becomes louder or when the gesture becomes louder. Also for these, in order to make it easier for the high and low priorities to function, the difference in the point addition method as described above may be provided.

Further, even if the action of the subject does not exceed 2000 points, the image is always taken in 120 seconds with the passage of time, so that it is possible that the image is not taken at all for a certain period of time.

Further, when the subject is detected in the middle, the time for starting the increase may be advanced in 120 seconds. That is, for example, if a subject with a high priority is detected at 60 seconds, even if 1000 points are added, it does not exceed 2000 points, but it does not increase to 110 seconds as it is, but after detecting the subject. The linear increase may be started after 30 seconds. Alternatively, the linear increase may be started 20 seconds before 10 seconds before 120 seconds. By doing so, the possibility that a high-priority subject is imaged increases, and it becomes easier to realize the image capture according to the user's intention.

When automatic imaging is performed, the imaging score is reset to 0 points. Automatic imaging is not performed until 2000 points are exceeded again.

The determination of whether or not to perform automatic imaging has been described above. If it is determined by the above determination that the first control unit 223 automatically takes an image, the process of step S508 is executed. If the first control unit 223 determines that the image is not captured, the process of step S501 is executed.

In step S508, the first control unit 223 executes an imaging process. The imaging process is, for example, still image imaging or moving image imaging.

The automatic image pickup process of the image pickup apparatus 101 in the present embodiment has been described above. By such a process of automatically capturing a subject, the imaging device 101 can capture an image or a moving image of a scene desired by the user without an imaging instruction from the user.

The first control unit 223 may perform the automatic image pickup process by repeating the process of searching for a subject and the image pickup process for a predetermined time and the sleep process for entering a sleep state for a predetermined time. good. In the sleep process, the first control unit 223 controls the first power supply unit 210 so as to turn off the power supply to itself, and the second control unit 211 is in operation. As a result, the first control unit 223 can automatically capture an image or a moving image of a scene desired by the user while suppressing power consumption. In addition, in order to distinguish such an automatic imaging process from the above-mentioned continuous imaging process, it is referred to as an intermittent imaging process.

<Voice recognition processing>
FIG. 7 is a flowchart of the voice recognition process of the image pickup apparatus 101 in the present embodiment. The processing of this flowchart is started in response to the detection by the voice input voice processing unit 409 that the voice signal is input from the microphone. Further, the processing of this flowchart is executed in parallel with the automatic imaging process of FIG. In the present embodiment, the case where the flowchart is started in response to the detection by the voice input voice processing unit 409 that the voice signal is input from the microphone during the execution of the automatic imaging process of FIG. 5 is taken as an example. I will explain. Further, this processing is realized by the first control unit 223 executing the program recorded in the non-volatile memory 216.

In step S701, the first control unit 223 determines whether or not the trigger word has been detected. The trigger word is an activation command for starting voice command recognition that gives a specific instruction to the image pickup apparatus 101 by voice. When the user gives an instruction to the image pickup apparatus 101 by voice, it is necessary to utter a command word after the trigger word so that the image pickup apparatus 101 recognizes the command word. If the trigger word is detected, the process of step S702 is executed. If the trigger word is not detected, the process of step S701 is repeated until the trigger word is detected.

In step S702, the first control unit 223 stops the automatic imaging process. In the present embodiment, when the first control unit 223 detects the trigger word, the first control unit 223 transitions to the standby state of the command word. Further, the first control unit 223 performs the subject search process (step S504), the drive process (step S505-step S506), and the image pickup process (step S508) among the automatic image pickup processes described using the flowchart of FIG. Stop. The drive processing is, for example, pan drive, tilt drive, and zoom drive. The imaging process is, for example, still image imaging and moving image imaging. On the other hand, in this step, the first control unit 223 continues to execute the image recognition process (step S502) for subject detection and the like without stopping. The reason for stopping the automatic imaging process when the trigger word is detected in this way is as follows. That is, it is considered that the user recognizes the state in which the image pickup apparatus 100 is facing the direction of the subject to be imaged, and utters the trigger word at the timing when the user wants to image the image. On the other hand, since the first control unit 223 does not detect the instruction by the command word only by detecting the trigger word, it cannot determine whether or not the imaging process should be executed. The first control unit 223 determines that the image pickup process should be executed at the timing when the command word after the trigger word is recognized. That is, if the automatic image pickup process is continued even after the first control unit 223 detects the trigger word, the direction of the image pickup device 100 may change from the direction in which the user wants to image when the command word is recognized. be. Therefore, the first control unit 223 stops the automatic imaging process in response to the detection of the trigger word to maintain the state facing the imaging direction at the timing at which the user wants to image. Then, when the user utters a command word for instructing imaging after the trigger word, the first control unit 223 can capture the subject to be imaged by the user by executing the imaging process. That is, the first control unit 223 can capture the subject to be captured by the user by stopping the drive process.

In step S703, the first control unit 223 outputs a detection sound for indicating to the user that the trigger word has been detected by the speaker.

In step S704, the first control unit 223 determines whether or not a command word is detected following the trigger word. If the command word is detected, the process of step S706 is executed, and if it is not detected, the process of step S705 is executed.

In step S705, the first control unit 223 detects the trigger word and determines whether or not a predetermined time has elapsed since the command word was in the standby state. When the predetermined time has elapsed, the process of step S701 is executed, and the first control unit 223 stops the standby state of the command word and enters the standby state of the trigger word. Further, in this case, the first control unit 223 restarts the automatic imaging process. If the predetermined time has not elapsed, the first control unit 223 repeats the process of step S704 until the command word is detected.

In step S706, the first control unit 223 determines whether or not the detected command word is a still image imaging command. This still image image pickup command is a command for causing the image pickup apparatus 101 to execute an image pickup process and a recording process of one still image. If it is determined that the command is a still image capture command, the process of step S707 is executed, and if it is determined that the command is not a still image capture command, the process of step S708 is executed.

In step S707, the first control unit 223 performs a still image imaging process. Specifically, the signal captured by the image pickup unit 206 is converted by the image processing unit 207 according to the JPEG format and the like, and recorded on the recording medium 221 by the image recording unit 208.

In step S708, the first control unit 223 determines whether or not the detected command word is a subject change command. The subject change command is, for example, a keyword of "imaging another person". If it is determined that the command is a subject change command, the process of step S709 is executed, and if it is determined that the command is not a subject change command, the process of step S710 is executed.

In step S709, the first control unit 223 performs subject change processing. In the subject change process, the first control unit 223 drives and controls the lens barrel 102 so as to capture a subject different from the subject currently being captured. If the subject is not detected at the time when the command word is detected, the first control unit 223 does not execute the process of this step.

In step S710, the first control unit 223 determines whether or not the detected command word is a moving image recording start command. The moving image imaging command is a command for causing the imaging device 101 to execute a moving image imaging process and a recording process. If it is determined that it is a moving image recording start command, the process of step S712 is executed, and if it is determined that it is not a moving image recording start command, the process of step S713 is executed.

In step S711, the first control unit 223 starts imaging a moving image using the imaging unit 206, and records the captured moving image data on the recording medium 221. While recording the moving image data, the first control unit 223 does not search for the subject and continues to maintain the state in which the automatic imaging is stopped.

In step S712, the first control unit 223 determines whether or not the detected command word is a moving image recording stop command. If it is determined that the command is not a moving image recording stop command, the process of step S713 is executed, and if it is determined that the command is not a moving image recording stop command, the process of step S714 is executed.

In step S713, the first control unit 223 stops the image pickup of the subject by the image pickup unit 206 and the recording of the moving image data on the recording medium 221 to complete the recording of the moving image data.

In step S714, the first control unit 223 executes processing corresponding to other command words. For example, the first control unit 223 performs processing for a command word for performing pan drive and tilt drive in a direction specified by the user, and for a command word for changing various imaging parameters such as exposure compensation of the imaging device 101. Perform processing.

In step S715 and step S716, the first control unit 223 performs the process of resuming the automatic imaging stopped in step S702. For example, the first control unit 223 starts the processing of the flowchart shown in FIG. 5 from the processing of step S502.

The voice recognition process of the image pickup apparatus 101 in the present embodiment has been described above.

As described above, in the voice recognition process of the subject of the image pickup device 101 in the present embodiment, the image pickup device 101 stops the automatic image pickup process in response to the detection of the trigger word. That is, in the process of step S702 of the flowchart shown in FIG. 7, the first control unit 223 stops the automatic image pickup process in response to the detection of the trigger word, so that the user faces the image pickup direction at the timing desired to be imaged. Maintained the state. As a result, when the user utters a command word for instructing imaging after the trigger word, the first control unit 223 executes the imaging process at the image angle at the time when the trigger word is received. Therefore, it is possible to capture the subject that the user wants to capture.

In step S702, the first control unit 223 controls the drive speed of the lens barrel 102 in the image pickup direction to be slower than the drive speed immediately before detecting the trigger word, instead of stopping the automatic image pickup process. You may. In the present embodiment, in consideration of the time required for the process of detecting the command word, the drive speed of the lens barrel 102 in the image pickup direction of the first control unit 223 is the drive speed of the lens barrel 102 in the image pickup direction in the automatic image pickup process. Control to be slower than the speed. Then, when the command word does not issue an imaging instruction, the first control unit 223 restores the driving speed and continues the automatic imaging process. As a result, the image pickup apparatus 101 can achieve both the ease in which the user wants to capture the image in the angle of view and the continuation of the change in the angle of view in the automatic image pickup process.

If the subject is detected at the timing when the trigger word is detected, the first control unit 223 drives and controls the lens barrel 102 so as to track the subject without stopping the drive process in step S702. You may. This is because the subject detected by the image pickup apparatus 100 at the timing when the user issues a trigger word is the subject to be imaged by the user, and it is possible that the image desired by the user can be obtained by continuously driving and tracking the subject. This is because it is considered expensive. In this case, the first control unit 223 drives and controls the lens barrel 102 so as to track (continue to keep the angle of view) the detected subject at the timing when the trigger word is detected. In this way, the first control unit 223 may drive the image pickup direction so as to track the subject for which the trigger word is detected, so as to capture the subject to be captured by the user. Similarly, even when the subject is tracked at the timing when the trigger word is detected, the first control unit 223 tracks the subject without stopping the drive process in step S702. May be driven and controlled.

If the subject is not detected at the timing when the trigger word is detected, the first control unit 223 may drive and control the lens barrel 102 so as to execute the subject search process in step S702. For example, the first control unit 223 executes the subject search process in a range narrower than the search range of the automatic image pickup process, including the current image pickup direction. In this way, the first control unit 223 may capture the subject to be captured by the user by searching for the subject in the vicinity of the current imaging direction in response to the detection of the trigger word.

In this way, the image pickup apparatus 101 not only stops the automatic image pickup process in response to the detection of the trigger word, but also tracks the subject to be imaged at the timing when the trigger word is detected. Limit the drive range according to the detection of the word. As a result, the image pickup apparatus 101 can capture the subject to be captured by the user.

In step S702, the first control unit 223 may stop the execution of the still image imaging and the moving image imaging process. By reducing the load for the image pickup process, the first control unit 223 can quickly react to the instruction of the command word issued from the user next to the trigger word.

If it is determined in step S705 that the predetermined time has elapsed, the first control unit 223 may output a sound indicating that the predetermined time has elapsed from the voice output unit 218. As a result, the user can recognize that the image pickup apparatus 101 has failed to detect the command word, so that it becomes easy to understand the timing of giving the instruction by voice again.

When the first control unit 223 is executing the intermittent imaging process as the automatic imaging process, the first control unit 223 executes the sleep process in response to the detection of the trigger word until the process instructed by the command word is completed. You may procrastinate. In this case, the first control unit 223 executes the sleep process in the intermittent imaging process in response to the completion of the process instructed by the command word.

[Other embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

It should be noted that the present invention is not limited to the above embodiment as it is, and at the implementation stage, the components can be modified and embodied within a range that does not deviate from the gist thereof. In addition, various inventions can be formed by an appropriate combination of the plurality of components disclosed in the above-described embodiment. For example, some components may be removed from all the components shown in the embodiments. In addition, components across different embodiments may be combined as appropriate.

Claims (17)

An imaging means that captures the subject,
A driving means for driving the imaging direction of the imaging means and
It has a detection means for detecting a first voice command for executing a specific process, a second voice command for initiating detection of the first voice command, and a detection means for detecting the first voice command.
An imaging device, characterized in that, in response to the detection means detecting the second voice command, the driving means limits driving in the imaging direction. A claim characterized in that, when the detection means detects the first voice command for executing the image pickup process while the drive means limits the drive, the image pickup means images the subject. The image pickup apparatus according to 1. Determining means to automatically determine the imaging target,
Claim 1 is further characterized by further comprising a control means for controlling an automatic image pickup process for driving the image pickup means to drive the image pickup direction of the image pickup means so that the image pickup means captures an image pickup target determined by the determination means. Or the image pickup apparatus according to 2. The imaging device according to claim 3, wherein the determination means calculates the importance of each subject based on the detected face information of the subject, and determines the imaging target based on the importance. The fourth aspect of the present invention is characterized in that the importance of the subject is calculated based on the priority set by the user, the facial expression of the subject, the size of the eyes of the subject, and the orientation of the face of the subject. The imaging device described. One of claims 3 to 5, wherein the detection means detects the first voice command and the second voice command in parallel even during the execution of the automatic imaging process. The imaging device according to. The invention according to any one of claims 3 to 6, wherein when the detection means detects the second voice command during the execution of the automatic image pickup process, the automatic image pickup process is stopped. Imaging device. Any of claims 3 to 6, wherein when the detection means detects the second voice command during the execution of the automatic image pickup process, the drive means limits the drive in the image pickup direction. The image pickup apparatus according to item 1. The image pickup according to any one of claims 3 to 8, wherein in the automatic image pickup process, the drive means drives the image pickup direction of the image pickup means with the entire circumference of the image pickup device as a drive range. Device. The method according to any one of claims 3 to 9, wherein when the drive is limited, the drive means drives the image pickup direction of the image pickup means in a drive range narrower than the drive range in the automatic image pickup process. Imaging device. When limiting the drive, the driving means drives the imaging direction of the imaging means at a speed slower than the driving speed of the imaging means immediately before the detecting means detects the second voice command. The image pickup apparatus according to any one of claims 1 to 10. When the driving is limited, any one of claims 1 to 11 is characterized in that the driving means drives the subject captured when the second voice command is detected so as to fit in the angle of view. The imaging device according to the section. The image pickup apparatus according to any one of claims 1 to 12, wherein when the drive is restricted, the drive means is driven so as to search for a subject in the restricted drive. The image pickup apparatus according to any one of claims 1 to 10, wherein when the drive is limited, the drive means stops the drive of the image pickup means in the image pickup direction. Have more mics
The imaging device according to any one of claims 1 to 14, wherein the detection means detects the voice command by analyzing a voice signal input from the microphone. An imaging step to capture the subject and
A drive step that drives the imaging direction in the imaging step,
A detection step for detecting a first voice command for executing a specific process and a second voice command for initiating detection of the first voice command.
A step of limiting the drive in the drive step according to the detection of the second voice command in the detection step, and a step of limiting the drive in the drive step.
A control method characterized by having. A computer-readable program for operating a computer as each means of the image pickup apparatus according to any one of claims 1 to 15.

Images