JP2019120885A - Projection control system and projection control method - Google Patents

Abstract

To accurately detect a position of a detection object within an imaging field angle, and to adaptively support correct projection of a content that follows movement of the detection object.SOLUTION: A projection control system comprises: a light source device that irradiates infra-red light; an imaging apparatus that receives light for the irradiated infra-red light to penetrate a projected object, and that images an infra-red imaged image including the projected object as a field angle; and a projection control apparatus that recognizes a shape of an object existing around the projected object on the basis of the imaged infra-red imaged image, that generates a projection image according to the shape of the recognized object, and that outputs to a projection apparatus which projects the generated projection image to the projected object. The light source device is arranged at an opposite side of the imaging apparatus sandwiching the projected object.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to a projection control system and a projection control method that control projection of an image onto a projection target.

  Patent Document 1 discloses a projection apparatus that displays a projection image in a projection area that is easy for a person to view regardless of the position of the person by detecting the position of the person and determining the projection direction according to the detected position. It is done. In Patent Document 1, the position of a person is detected based on, for example, imaging results of a plurality of cameras.

JP, 2016-4215, A

  However, in the configuration of Patent Document 1, the camera for detecting the position of the person is a visible light camera, and depending on the image content (for example, the content of the face of the person) projected by the projection device, the camera Image content may be misrecognized as a person.

  For this reason, the projection device projects the same or other image content around the image content that has been misrecognized as a person by the camera, not the person to be detected, and the projection device In contrast, there is a problem that it is not possible to project accurate image content. For example, it is not possible to accurately project the image content following the movement of the actual person to be detected.

  The present disclosure is devised in view of the conventional circumstances described above, detects the position of the detection target within the imaging angle of view with high accuracy, and adaptively projects the accurate projection of the content following the movement of the detection target. An object of the present invention is to provide a projection control system and a projection control method to support.

  The present disclosure relates to a light source device for emitting infrared light, and an image for capturing an infrared captured image including the object to be projected as an angle of view, receiving the light transmitted through the object to be projected by the irradiated infrared light. A shape of an object around the projection object is recognized based on the device and the captured infrared image, and a projection image corresponding to the shape of the recognized object is generated and generated. A projection control device for outputting the projected image onto the object to be projected, the light source device being disposed on the opposite side of the imaging device across the object to be projected Provide a control system.

  Further, the present disclosure is a projection control method in a projection control system, in which the light source device irradiates infrared light, and the irradiated infrared light receives light transmitted through the projection object, the projection object An infrared captured image including the angle of view as an angle of view is captured by an imaging device, and based on the captured infrared captured image, the shape of an object present around the projected object is recognized, and the recognized object A projection image according to the shape of the object is generated, and the generated projection image is output to a projection device for projecting the object onto the object, the light source device sandwiching the object and the opposite side of the imaging device To provide a projection control method, which is arranged in

  According to the present disclosure, the position of the detection target within the imaging angle of view can be detected with high accuracy, and accurate projection of the content following the movement of the detection target can be adaptively assisted.

A diagram showing an installation example of a projection control system according to the first embodiment Block diagram showing in detail an example of the system configuration of the projection control system according to the first embodiment Description of an example of calibration for conversion between coordinates (Xc, Yc) indicating the position of an infrared captured image of an infrared camera and coordinates (Xp, Yp) indicating the position of a projected image projected by a projector Figure Explanatory drawing which shows an example of the infrared captured image imaged with an infrared camera A diagram showing an example of a user operation on an offset adjustment screen displayed on a display device The figure which shows the example which the projection position of the image content shifted to the left side by user operation shown in FIG. A flowchart for explaining an example of the operation procedure of the projection control process according to the first embodiment The figure which shows the example of installation of the projection control system which concerns on the 1st modification of Embodiment 1. Explanatory drawing which shows an example of the infrared captured image imaged with the infrared camera which concerns on the 1st modification of Embodiment 1. A diagram showing an installation example of a projection control system according to a second embodiment The figure which shows an example of the projection image which concerns on Embodiment 2. A diagram showing an installation example of a projection control system according to a third embodiment Block diagram showing in detail a system configuration example of a projection control system according to Embodiment 3. Coordinates (Xci, Yci) indicating the position of the infrared captured image of the infrared camera, coordinates (Xp, Yp) indicating the position of the projected image projected by the projector, and coordinates (the location of the visible light captured image of the visible light camera Explanatory diagram showing an example of calibration for conversion between Xcv and Ycv) The figure which shows an example of the projection image which concerns on Embodiment 3. A flowchart for explaining an example of the operation procedure of the projection control process according to the third embodiment The figure which shows the example of installation of the projection control system concerning the 2nd modification of Embodiment 1. A block diagram showing in detail an example of a system configuration of a projection control system according to a second modification of the first embodiment The figure which shows the example of installation of the projection control system concerning the 3rd modification of Embodiment 1.

  Hereinafter, embodiments of the projection control system and the projection control method according to the present disclosure will be described in detail with reference to the attached drawings as appropriate. However, the detailed description may be omitted if necessary. For example, detailed description of already well-known matters and redundant description of substantially the same configuration may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. It should be noted that the attached drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and they are not intended to limit the claimed subject matter.

  Hereinafter, the projection control system according to the present disclosure is, for example, an infrared imaging with an infrared camera (an example of an imaging device) of the shape of a person (an example of an object) on a screen (an example of an object to be projected) The image is recognized based on the image, and a projected image on which the image content is superimposed is projected from the projector toward the screen in accordance with the shape of the recognized person. The projection control system may be installed inside a store (in other words, indoor) or may be installed outside the store (in other words, outdoors).

Embodiment 1
FIG. 1 is a diagram showing an installation example of the projection control system 100 according to the first embodiment. Projection control system 100 includes at least projector 10 (see FIG. 2), infrared illumination 20, screen 30, infrared camera 40 (see FIG. 2), and projection control device 50 (see FIG. 2). . Each of the input device IP and the display device DP is connected to the projection control device 50 so that data or information can be input / output.

  The projector 10 as an example of a projection device has a cylindrical case, for example, and is connected to be able to transmit and receive data or information with the projection control device 50. The projector 10 is disposed to face the screen 30. When the projector 10 receives and acquires data of a projection image including a projection instruction sent from the projection control device 50, the projector 10 projects the projection image specified by the projection instruction on the screen 30 based on the projection instruction. Projected light (e.g. visible light) is generated and projected towards the screen 30. Thus, the projector 10 can project the projection image (see below) specified by the projection control device 50 on the screen 30. The details of the projector 10 will be described with reference to FIG.

  The infrared illumination 20 as an example of a light source device includes a plurality of lighting elements (not shown, for example, LED (Light Emission Diode) elements that emit infrared light) and control for controlling the presence or absence of light emission in each lighting element. It has a housing in which a substrate (not shown) is contained, and is separate from the housing of the projector 10. The infrared illumination 20 may be connected so as to be able to transmit and receive data or information with the projection control device 50, or even if transmission and reception of data or information with the projection control device 50 is not communicably connected. Good. The infrared illumination 20 is disposed to face the screen 30 in the same manner as the projector 10. The infrared illumination 20 is turned on based on, for example, a manual operation of a manager of the projection control system 100 or a control instruction from the projection control apparatus 50, and the infrared light having a wavelength band different from that of the projection light described above Irradiate towards.

  The infrared illumination 20 may have, for example, a cylindrical housing integrally attached along the radial periphery of the cylindrical housing of the projector 10. In this case, a plurality of lighting elements (for example, LED elements that emit infrared light) and a control substrate (see the above) are incorporated in the cylindrical housing. That is, the plurality of lighting elements (for example, LED elements that emit infrared light) are arranged in an annular shape along the radial circumference of the cylindrical housing of the projector 10.

  In the projection control system 100 according to the first embodiment, the infrared light emitted from the infrared illumination 20 is different in wavelength band from the projection light (visible light) projected from the projector 10, and the infrared camera 40 is a visible light. Instead, it receives infrared light and picks up an image. This is because a person who is present (for example, in front of) the screen 30 arranged so that the projection control device 50 is included within the angle of view of the infrared camera 40 based on the infrared captured image captured by the infrared camera 40 This is to detect the position of the HM with high accuracy without erroneous recognition.

  The screen 30 as an example of an object to be projected is formed using a material capable of transmitting infrared light emitted from the infrared illumination 20 and is fixedly installed. It is assumed that the front and back sides of the screen 30 conform to FIG. Infrared light transmitted through the screen 30 is received by the infrared camera 40. The screen 30 may be, for example, a transparent screen capable of alternately switching between a transparent mode capable of transmitting infrared light and a screen mode alternately (see, for example, the following reference non-patent document 1), or capable of transmitting infrared light. It may be a rear projection screen (see, for example, the following reference non-patent document 2) in which the film is attached to the rear surface of the glass.

Reference Non-Patent Document 1

  Panasonic Corporation, Transparent Screen, [online], [Search on December 27, 2017], Internet <URL: https://panasonic.biz/cns/invc/screen/technology.html>

Reference Non-Patent Document 2

  Theater house, rear transmission type film / screen type, [online], [search on December 27, 2017], Internet <http://theaterhouse.co.jp/p_rear/item_top.html>

  The infrared camera 40 as an example of an imaging device is connected so as to be able to transmit and receive data or information with the projection control device 50. The infrared camera 40 is disposed so as to face the entire screen 30 so as to entirely include it within the angle of view. The infrared camera 40 receives light including infrared light emitted from the infrared illumination 20 and transmitted through the screen 30, and generates an infrared captured image by imaging based on the received light. The infrared camera 40 transmits the data of the generated infrared captured image to the projection control device 50. The details of the infrared camera 40 will be described with reference to FIG.

  As shown in FIG. 1, the projector 10 and the infrared camera 40, or the infrared illumination 20 and the infrared camera 40 may be disposed opposite to each other with the screen 30 interposed therebetween, and exist coaxially with each other. It does not have to be arranged. For example, since the infrared illumination 20 and the infrared camera 40 are disposed opposite to each other with the screen 30 interposed therebetween, the infrared camera 40 can display the screen 30 based on the reception of the infrared transmission light transmitted through the screen 30. Thus, it is possible to capture an infrared captured image that can specify the shape of the person HM1 on the surface side of.

  The person HM1 as an example of the object is not limited to the position itself shown in FIG. 1 as long as the person HM1 is located in the object region ARF1 provided on the infrared camera 40 side of the screen 30. That is, the person HM1 may be located anywhere in the object area ARF1. The object area ARF1 is an area where a person HM1 located between the infrared illumination 20 and the infrared camera 40 exists, as in the screen 30. Also, the object area may be located either in front of the screen 30 (see FIGS. 1, 10, 12, 17) or in the rear (see FIG. 8), and may be located on both sides. Further, it is preferable that the object region ARF1 be specifically visualized by, for example, drawing a frame line or the like on the floor, as viewed from the person HM1. This makes it possible to suppress the movement of the person HM1 out of the object area ARF1. However, visualization of the object region ARF1 may not be essential. Also, the target object area ARF1 may be instructed to the person HM1 by, for example, the projector 10, another projector (not shown), a signboard, or a system administrator or the like, with the suggestion that "stand up in front of the screen". .

  The projection control device 50 is an information processing device capable of wired or wireless communication, such as a PC (Personal Computer) or a tablet terminal, and at least among the projector 10, the infrared camera 40, the input device IP, and the display device DP. It is connected so that transmission or reception of data or information is possible. The projection control device 50, based on the infrared captured image captured by the infrared camera 40 (in other words, the captured image generated based on the imaging of the infrared light transmitted through the screen 30), the periphery of the screen 30 (for example, Recognize the presence or absence of a person HM1 (an example of a target person) and the shape indicating the position of the person HM1 with respect to the screen 30). The projection control device 50 generates a projection image according to the shape of the recognized person HM1, and outputs the generated projection image to the projector 10 that projects the projection image on the screen 30. The details of the projection control device 50 will be described with reference to FIG.

  The input device IP is a device capable of receiving an operation of a user (for example, an administrator) of the projection control system 100, such as a keyboard or a mouse, for example, and a user interface for transmitting a signal corresponding to the operation of the user to the projection control device 50. It is.

  The display device DP is configured using, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence), acquires a display instruction of data or information sent from the projection control device 50, and is included in the acquired display instruction. Display the data or information that The display device DP may be configured as a touch panel that can receive and detect a user's operation. In the projection control system 100 according to the first embodiment, the input device IP and the display device DP may be omitted.

  FIG. 2 is a block diagram showing in detail an example of the system configuration of the projection control system 100 according to the first embodiment.

  The projector 10 is configured to include a communication interface 11, a control unit 12, a projection unit 13, and a memory 14. In the configuration of the projector 10 of FIG. 2, the communication interface is conveniently described as “communication I / F”.

  The communication interface 11 has a role as a communication interface regarding transmission / reception (communication) of data or information with the projection control device 50, and receives a projection instruction including data of a projection image sent from the projection control device 50. When it is acquired, the acquired projection instruction is passed to the control unit 12.

  The control unit 12 is a processor PRC1 configured using, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA). The control unit 12 functions as a controller that controls the operation of the projector 10, and performs control processing to generally control the operation of each unit of the projector 10, input / output processing of data with each unit of the projector 10, and data calculation (Calculation) processing and storage processing of data. The control unit 12 operates in accordance with programs and data stored in the memory 14. The control unit 12 may use the memory 14 at the time of operation, and temporarily store data or information generated or acquired by the control unit 12 in the memory 14.

  The projection unit 13 receives control of projection by the control unit 12, emits predetermined projection light, and projects it on the screen 30. Specifically, based on the data of the projection image passed from the control unit 12, the projection unit 13 emits projection light for projecting the projection image on the screen 30. The projection unit is configured of, for example, a light source, a lens, and the like.

  The memory 14 is configured using, for example, a RAM (Random Access Memory) and a ROM (Read Only Memory), and programs and data necessary for executing the operation of the projector 10, and further, data or information generated during the operation Hold temporarily. The RAM is, for example, a work memory used when the projector 10 operates. The ROM stores and holds, for example, a program and data for controlling the projector 10 in advance.

  The infrared camera 40 is configured to include an infrared imaging unit 41, a control unit 42, a communication interface 43, and a memory 44. Also in the configuration of the infrared camera 40 of FIG. 2, the communication interface is similarly described as “communication I / F” for convenience.

  The infrared imaging unit 41 includes an infrared light lens (not shown) capable of receiving light (infrared transmission light) transmitted by the screen 30 from infrared light emitted from the infrared illumination 20, and the light (infrared transmission) And an image sensor (not shown) capable of capturing an infrared captured image based on light). The output of the infrared imaging unit 41 is sent to the control unit 42.

  The control unit 42 is a processor PRC2 configured using, for example, a CPU, an MPU, a DSP, or an FPGA. The control unit 42 functions as a controller that controls the operation of the infrared camera 40, and performs control processing to generally control the operation of each unit of the infrared camera 40, input / output processing of data with each unit of the infrared camera 40, Data calculation (calculation) processing and data storage processing are performed. The control unit 42 operates in accordance with the program and data stored in the memory 44. The control unit 42 may use the memory 44 at the time of operation, and temporarily store data or information generated or acquired by the control unit 42 in the memory 44. The control unit 42 performs predetermined image processing based on, for example, an output of the image sensor of the infrared imaging unit 41 to generate data of an infrared pickup image.

  The communication interface 43 has a role as a communication interface related to transmission / reception (communication) of data or information with the projection control apparatus 50, and when the data of the infrared captured image generated by the control unit 42 is acquired, the acquisition is performed. The data of the captured infrared image is transmitted to the projection control device 50.

  The memory 44 is configured using, for example, a RAM and a ROM, and temporarily holds programs and data required to execute the operation of the infrared camera 40, and further, data or information generated during the operation. The RAM is, for example, a work memory used when the infrared camera 40 operates. The ROM stores and holds, for example, a program and data for controlling the infrared camera 40 in advance.

  The projection control device 50 is configured to include an input / output interface 51, a memory 52, a content storage database (DB: Database) 53, a recording device 54, and a processor 55. The input / output interface 51, the memory 52, the content storage database 53, the recording device 54, and the processor 55 are mutually connected via the internal bus BS1 so that data or information can be input / output.

  The input / output interface 51 is connected so as to be able to transmit and receive wired or wireless data or information with each of the projector 10, the infrared camera 40, the input device IP and the display device DP. Although not shown in FIG. 2, the input / output interface 51 may be connected so as to be able to transmit and receive wired or wireless data or information with the infrared light 20.

  The memory 52 is configured using, for example, a RAM and a ROM, and temporarily holds programs and data required to execute the operation of the projection control device 50, and further data or information generated during the operation. The RAM is, for example, a work memory used when the projection control device 50 operates. The ROM stores and holds, for example, a program and data for controlling the projection control device 50 in advance. The memory 52 includes, for example, information of the resolution (specifically, vertical pixels, horizontal pixels) of the infrared captured image generated by the infrared camera 40 and the size of the screen 30 projected by the projector 10 Holds information on area).

  The content storage database 53 as an example of the content storage unit is configured using, for example, a hard disk drive (HDD) or a solid state drive (SSD), and is an image content to be superimposed on a projection image projected by the projector 10. Data is stored and held.

  The recording device 54 is configured using, for example, an HDD or an SSD, and the projection control device 50 records data or information sent from the input device IP or the infrared camera 40 and acquired, or a projected image for projection by the projector 10 Data (specifically, data of a projection image generated by the processor 55).

  The processor 55 is configured using, for example, a CPU, an MPU, a DSP or an FPGA. The processor 55 functions as a controller that controls the operation of the projection control device 50, and performs control processing to totally control the operation of each portion of the projection control device 50, and inputs / outputs data from / to each portion of the projection control device 50. Processing, data calculation (calculation) processing and data storage processing are performed. The processor 55 operates in accordance with the programs and data stored in the memory 52. The processor 55 may use the memory 52 at the time of operation and temporarily store in the memory 52 data or information generated or obtained by the processor 55.

  The processor 55 is configured to include at least a calibration unit 551, an object detection unit 552, and a projection image generation unit 553, for example. Each of the calibration unit 551, the object detection unit 552, and the projection image generation unit 553 can be functionally realized by, for example, the processor 55 reading and executing programs and data stored in advance in the memory 52.

  The calibration unit 551 sets a position (specifically, coordinates) in the infrared captured image generated by the infrared camera 40 and a position (specifically, coordinates) in the projection image projected by the projector 10. A process (that is, calibration) is performed to obtain a relational expression (for example, a projective transformation matrix) of the transformation process. Specifically, the calibration unit 551 specifies where the four corners of the projection image are located in the infrared captured image by designation by using the input device IP of the user or predetermined image processing (for example, edge detection processing). For example, the projection transformation matrix described above is determined (see FIG. 3).

  FIG. 3 is for conversion between coordinates (Xc, Yc) indicating the position of the infrared captured image IRG of the infrared camera 40 and coordinates (Xp, Yp) indicating the position of the projection image PJR projected by the projector 10 It is an explanatory view showing an example of calibration of. Each of the end points EG1, EG2, EG3, EG4 in the infrared captured image IRG is, for example, in a state in which the infrared captured image IRG and the projected image PJR are displayed on the display device DP in comparison (see FIG. 3) These correspond to the end points CR1, CR2, CR3, and CR4 of the four corners of the projection image PJR selected by designation using the input device IP.

  Note that, as described above, the calibration unit 551 performs image processing (for example, edge detection processing or straight line detection processing) on the infrared captured image IRG so that the user does not specify using the input device IP. The end points EG1, EG2, EG3, and EG4 in the captured image IRG may be automatically detected. In this case, the calibration unit 551 selects the end points EG1, EG2, EG3, EG4 detected by the image processing as corresponding to the end points CR1, CR2, CR3, CR4 of the four corners of the projected image PJR.

  The calibration unit 551 determines an infrared image based on the coordinates of end points EG1, EG2, EG3, EG4 in the infrared captured image IRG and the coordinates of end points CR1, CR2, CR3, CR4 of four corners of the projected image PJR. A projective transformation matrix for transforming coordinates (Xc, Yc) in the IRG into coordinates (Xp, Yp) in the projected image PJR is calculated and determined. The calibration unit 551 stores the data or information of the obtained projective transformation matrix in the memory 52 as a calibration result. Thereby, the projection control apparatus 50 uses the calibration result to accurately set arbitrary coordinates (Xc, Yc) in the infrared captured image IRG to corresponding coordinates (Xp, Yp) in the projected image PJR. It can be converted.

  In the projection control system 100 shown in FIG. 1, the calibration in the calibration unit 551 is performed unless the resolution (in other words, the size or area) of the projected image projected by the projector 10 or the position of the infrared camera 40 is changed. Processing of the session may be performed at least once. In other words, each time the resolution of the projection image projected by the projector 10 or the position of the infrared camera 40 is changed, the calibration process in the calibration unit 551 is performed at least once.

  Further, the method of calibration in the calibration unit 551 is not limited to the method described above. For example, the calibration unit 551 can also convert coordinates in the infrared captured image IRG to corresponding coordinates in the projection image PJR in pixel units by executing calibration using a known structured light method. It is.

  The object detection unit 552 acquires data of an infrared captured image generated by the infrared camera 40 via the input / output interface 51. The object detection unit 552 recognizes (detects) the shape of the person HM1 around (for example, in front of) the screen 30 based on the acquired data of the infrared captured image (see FIG. 4). The object detection unit 552 passes the recognition result of the shape (in other words, the silhouette of the person HM1) of the recognized (detected) person HM1 to the projection image generation unit 553.

  FIG. 4 is an explanatory view showing an example of infrared captured images IRG 0 and IRG 1 captured by the infrared camera 40. In the first embodiment, the infrared camera 40 is disposed on the front side of the screen 30, and the infrared illumination 20 is disposed on the back side of the screen 30. That is, the infrared illumination 20 and the infrared camera 40 are disposed opposite to each other with the screen 30 interposed therebetween. Further, infrared light emitted from the infrared illumination 20 passes through the screen 30.

  Therefore, as shown on the left side of the drawing of FIG. 4, when there is no object such as the person HM1 on the front side of the screen 30 (in other words, there is no object that reflects infrared light) The irradiated infrared light is transmitted without reflection on the screen 30 and is received by the infrared camera 40. Therefore, the infrared captured image IRG0 captured by the infrared camera 40 is an image composed of only the white area WHR1.

  On the other hand, as shown on the right side of the drawing of FIG. 4, when an object such as a person HM1 exists on the front side of the screen 30 (in other words, an object that reflects infrared light exists) The infrared light thus reflected is reflected by the person HM1 at the position where the person HM1 is present, and is transmitted through the screen 30 and received by the infrared camera 40 at the position where the person HM1 is not present. Therefore, since infrared light is not received by the infrared camera 40 from the position where the person HM1 is in the infrared captured image IRG1 captured by the infrared camera 40, the region HM1SL where the silhouette of the person HM1 is formed and the person HM1 It becomes an image in which both the non-white area WHR1 exists. Although the inside of the outline of the person HM1 in the area HM1SL is white in FIG. 4 in order to make the FIG. 4 easy to understand, the inside of the area HM1SL in the data of the actual infrared captured image is black.

  Therefore, the object detection unit 552 recognizes, based on the data of the infrared captured image captured by the infrared camera 40, where on the screen 30 the person HM1 (an example of the target) is present with high accuracy and high accuracy. Can be identified.

  The projection image generation unit 553 acquires the recognition result of the shape of the person HM1 recognized by the object detection unit 552. The projection image generation unit 553 uses the acquired recognition result and the calibration result stored in the memory 52 to project corresponding to each position (coordinates) indicating the range of the person HM1 in the infrared captured image. A projected image is generated in which the image content read from the content storage database 53 is superimposed at a position (coordinates) in the image or a position (coordinates) separated by a predetermined distance from the position (coordinates). The projection image generation unit 553 generates a projection instruction including data of the generated projection image, and sends the generated projection instruction to the projector 10 via the input / output interface 51.

  FIG. 5 is a diagram showing an example of a user operation on the offset adjustment screen WD1 displayed on the display device DP. FIG. 6 is a diagram showing an example in which the projection position of the image content RB1 is shifted to the left by the user operation shown in FIG. The offset adjustment screen WD1 shown in FIG. 5 is generated by the projection image generation unit 553 and displayed on the display device DP, for example, by the user's operation using the input device IP. For the background projection image PJGM1, which is a background portion of the projection image, the projection image generation unit 553 generates, for example, the image content BB1 of a blue ball on the upper left of the person HM1 and the image content RB1 of a red ball on the upper right of the person HM1. Each holds the setting to be superimposed. In FIG. 6, in order to make FIG. 6 easy to understand, the inside of the rectangular portion around the person HM1 in the background projection image PJGM1 is shown white so that the background projection image PJGM1 is not present. However, the projection image generation unit 553 may generate the background projection image PJGM1 that covers the entire area of the screen 30 so that the inside of the above-described rectangular portion also becomes the background projection image, as shown in FIG. A background projection image PJGM1 excluding the inside of the rectangular portion of the image may be generated. The projection image generation unit 553 may display the projection image PJR1 using the held setting as a preview on the display device DP. Thereby, the user can visually and intuitively confirm the superimposed position of the image content before adjustment using the offset adjustment screen WD1.

  Here, in the offset adjustment screen WD1, the image content RB1 is selected to adjust the superimposed position of the image content RB1 of the red ball, for example, by an operation using the input device IP of the user. The adjustment bar CSRx is shifted to the left. In response to this operation, as shown in FIG. 6, the projection image generation unit 553 moves the amount of movement operation of the adjustment bar CSRx (for example, 452-234 = 218) from the upper right portion of the person HM1 on the red ball image content RB2. Shift to the position according to the pixel) and update it to the setting to be superimposed. Thereby, the projection image generation unit 553 can generate the projection image PJR2 using the updated setting. The projection image generation unit 553 may display the projection image PJR2 on the display device DP as a preview. Thereby, the user can visually and intuitively confirm the superimposed position of the image content (for example, the image content RB2) after adjustment using the offset adjustment screen WD1.

  Although FIG. 5 illustrates an example of movement of the adjustment bar CSRx for adjusting and designating the superimposed position of the image content in the horizontal direction, the adjustment for designating and adjusting the superimposed position of the image content in the vertical direction is described. There may be movement of the bar.

  Next, an operation procedure of the projection control system 100 according to the first embodiment will be described with reference to FIG. FIG. 7 is a flowchart for explaining an example of the operation procedure of the projection control process according to the first embodiment. Each process shown in FIG. 7 is executed by, for example, the projection control device 50.

  In FIG. 7, in the processor 55 of the projection control apparatus 50, it is determined whether the calibration process has already been executed (S1). For example, the processor 55 can execute the process of step S1 depending on whether the calibration result in the calibration unit 551 is held in the memory 52. If it is determined that the calibration process has already been performed (S1, YES), the process of the projection control device 50 proceeds to step S3. On the other hand, when it is determined that the process of calibration has not been performed even once (S1, NO), the process of calibration is performed in the calibration unit 551 (S2).

  If it is determined that the calibration process has already been performed (S1, NO), or after the process of step S2, the object detection unit 552 of the processor 55 detects the infrared captured image captured by the infrared camera 40. Data is acquired, and the presence or absence of an object (for example, an object such as a person HM1) is detected based on the acquired infrared captured image (S3). In the process of step S3, when the object detection unit 552 can recognize (detect) the shape of the person HM1 around (for example, in front of) the screen 30 based on, for example, data of the infrared captured image, the person The presence of HM1 can be detected.

  If the object detection unit 552 can not detect the shape of the person HM1 (S4, NO), the processing of the projection control device 50 returns to step S3 and continues until the object detection unit 552 detects the shape of the person HM1, The process of S4 is repeated.

  On the other hand, when the object detection unit 552 can detect the shape of the person HM1 (S4, YES), the projection image generation unit 553 determines the position of the person HM1 in the infrared captured image detected by the object detection unit 552 That is, the projection position and the size (that is, the size) of the image content to be superimposed on the projection image projected by the projector 10 are determined according to the detection position (S5). The projection position indicates, for example, the positions of the image contents RB1, RB2, and BB1 shown in FIG. The size indicates, for example, the size of the image content RB1, RB2, BB1 shown in FIG.

  The projection image generation unit 553 generates a projection image in which the image content generated using the information on the projection position and the size determined in step S5 is superimposed on the background projection image that is the background portion of the projection image (S6). The projection image generation unit 553 generates a projection instruction including data of the generated projection image, and sends the generated projection instruction to the projector 10 via the input / output interface 51 (S7).

  After step S7, when the condition for ending the projection is satisfied (S8, YES), the processing of the projection control device 50 shown in FIG. 7 ends. The conditions for ending the projection correspond to, for example, the case where the user performs an operation of closing the dedicated application installed in advance in the projection control device 50 or the case where the power of the projection control device 50 is turned off.

  On the other hand, when the condition for ending the projection is not satisfied (S8, NO), the processing of the projection control device 50 returns to step S3. That is, when the person HM1 around (for example, in front of) the screen 30 moves, the projection control device 50 again performs infrared imaging to indicate the position of the screen 30 of the person HM1 after the movement in step S3. An image is acquired from the infrared camera 40, and the processes after step S4 are repeated and continued until the condition for ending the projection is satisfied.

  As described above, in the projection control system 100 according to the first embodiment, the infrared illumination 20 emits infrared light. The infrared camera 40 receives light (infrared transmission light) in which the emitted infrared light has transmitted through the screen 30, and captures an infrared pickup image including the entire screen 30 within the angle of view. The projection control device 50 recognizes the shape of the person HM1 around the screen 30 (for example, in front or behind) based on the infrared captured image captured by the infrared camera 40, and the shape of the recognized person HM1 A projected image is generated according to and output to the projector 10 that projects the generated projected image on the screen 30. The infrared illumination 20 is disposed on the opposite side of the infrared camera 40 with the screen 30 interposed therebetween.

  Thus, in the projection control system 100, infrared light from the infrared illumination 20 disposed on the opposite side of the screen 30 from the infrared camera 40 passes through the screen 30. The transmitted infrared transmitted light is received by the infrared camera 40. Therefore, the projection control device 50 can detect the position of the detection target (for example, the person HM1) present within the viewable angle of view of the infrared camera 40 with high accuracy. Furthermore, since the projection control device 50 can recognize the shape of the person HM1 on the front side of the screen 30 (for example, the side of the infrared camera 40) specifically and with high accuracy, for example, movement occurs in the detection target (for example the person HM1). Even in this case, it is possible to adaptively support the projection of the projection image in which the image content is superimposed on the correct position following the movement.

  In addition, the projection control device 50 includes a content storage database 53 that holds image content, and generates, as a projection image, an image in which the image content is superimposed in the vicinity of the shape of the recognized person HM1. Thereby, the projection control device 50 can superimpose the image content held in advance in accordance with the position of the person HM1 near the screen 30, and project the image content from the projector 10. For example, the projection of the image with improved entertainment properties Can be precisely controlled.

  In addition, the projection control device 50 includes a content storage database 53 that holds image content, and generates an image in which the image content is superimposed at a predetermined distance from the recognized person HM1 as a projection image. Thereby, the projection control device 50 can superimpose the image content held in advance at a position separated by a predetermined distance from the position of the person HM1 near the screen 30, and project the image content from the projector 10, for example, to improve the entertainment property. Can accurately control the projection of an image.

  In addition, the predetermined distance is changed according to an input (for example, an input operation) of a user (for example, a system administrator). As a result, the projection control apparatus 50 can superimpose the image content at an arbitrary position in accordance with the position of the person HM1, and the convenience at the time of projection of the image content can be improved.

  Further, in the projection control system 100, the object region ARF1 in which the person HM1 is located is located on the side of the infrared camera 40 around the screen. Accordingly, the infrared camera 40 can capture an infrared captured image that can specify the shape of the front view when the person HM1 faces the infrared camera 40. Also, the projection control device 50 can easily generate a projection image on which the image content matched with the movement of the person HM1, for example, is superimposed using the specified shape of the person HM1 as it is.

  Although not shown, in each of the embodiments disclosed in the present specification, the infrared camera 40 is an infrared imaging unit 41 as an example of an imaging unit that receives infrared light emitted from the infrared illumination 20. A filter (for example, a visible light cut filter) that cuts visible light may be further disposed on the front surface of the infrared imaging unit 41. The arrangement of the front side indicates that, for example, a visible light cut filter is arranged on the front side of the image sensor constituting the infrared imaging unit 41. Further, instead of the filter, a visible light cut lens that cuts visible light may be used as a lens that constitutes the infrared imaging unit 41. The infrared camera 40 may have spectral sensitivity in the wavelength region of visible light. In this case, not only infrared light but also visible light is received by the infrared camera 40, and the sensitivity of the infrared captured image is degraded. It is assumed that

  However, since the visible light cut lens or the visible light cut filter is disposed, the infrared camera 40 can generate an infrared pickup image based on the infrared light which can properly receive the visible light and receive the light, which is highly accurate. The infrared captured image can be output to the projection control device 50. In other words, the projection control device 50 can recognize the shape of the person HM1 around the screen 30 with high accuracy based on the infrared captured image from the infrared camera 40.

First Modified Example of First Embodiment
In the first modification of the first embodiment, for example, in the projection control system 100 according to the first embodiment, an example will be described in which a person HM1 as an example of an object is positioned on the projector 10 side around the screen 30 ( See Figure 8). FIG. 8 is a diagram showing an installation example of a projection control system according to a first modification of the first embodiment. FIG. 9 is an explanatory view showing an example of an infrared captured image IRG2 captured by the infrared camera 40 according to the first modified example of the first embodiment. The first modification of the first embodiment is different from the first embodiment in that a person HM1 as an example of an object is located on the opposite side of the infrared camera 40 around the screen 30. Except this point, the second embodiment is the same as the first embodiment, so different contents will be described.

  As shown in FIG. 8, the person HM <b> 1 is located on the side of the projector 10 around the screen 30. Usually, the person HM1 often stands with the back toward the screen 30, so in the first modified example of the first embodiment, in the infrared captured image IRG2 captured by the infrared camera 40, as shown in FIG. , The back of person HM1 is reflected in the front direction.

  Specifically, the infrared pickup image IRG2 is a region HM1SLa in which a silhouette of the person HM1 is formed because infrared light is not received by the infrared camera 40 from the position where the person HM1 is present, as in the infrared pickup image IRG1. And the white area WHR1 without the person HM1. The area HM1SLa is opposite to the area HM1SL in which the silhouette of the person HM1 in the infrared captured image IRG1 is formed in the front-rear direction (the direction of the front and back) of the person HM1. In other words, in the infrared captured image IRG2, the front direction of the person HM1 is reversed as compared with the infrared captured image IRG1 shown in FIG. Although both of the infrared captured images IRG1 and IRG2 can discriminate only the shape of the person HM1, both of the infrared captured images IRG1 and IRG2 are common in that both of the person HM1 are captured with the back facing the screen 30 There is. In FIG. 9, the inside of the outline of the person HM1 in the area HM1SLa is shown white in order to make it easy to understand FIG. 9, but the inside of the area HM1SLa in the data of the actual infrared captured image is black.

  Therefore, also in the first modification of the first embodiment, the projection control device 50 can recognize the shape of the person HM1 in the infrared captured image IRG2 as in the first embodiment, and the shape of the recognized person HM1 A projected image is generated according to and output to the projector 10 that projects the generated projected image on the screen 30. For example, the projection control device 50 generates a projection image by superimposing image content corresponding to the shape of the person HM1 at a position (coordinates) inverted in the left-right direction with the superposition position (coordinates) in the first embodiment. it can.

  Therefore, in the first modified example of the first embodiment as well as the first embodiment, the projection control device 50 is the front side of the screen 30 (for example, the projector 10 on the opposite side to the infrared camera 40 side across the screen 30). Since the shape of the person HM1 in the target object area ARB1 located on the side can be recognized with high accuracy and accuracy, for example, even if a movement occurs in the detection target (for example, the person HM1), the movement is followed It is possible to adaptively support the projection of the projection image in which the image content is superimposed on the correct position.

  Although not illustrated, in each embodiment disclosed in the present specification, the infrared illumination 20 is a diffusion for expanding the irradiation range of infrared light on the front surface of the own device (that is, the infrared illumination 20). Further sheets may be arranged. For example, in the case where the infrared illumination 20 is configured by a light emitting element having a strong directivity, such as a light emitting diode (LED), the above-described arrangement of the diffusion sheet is more effective. This makes it possible to diffuse the infrared light emitted from the infrared illumination 20 more than when the diffusion sheet is not disposed.

  Although not shown, in each embodiment disclosed in the present specification, the position at which the infrared illumination 20 is disposed is disposed on the opposite side of the infrared camera 40 with respect to the screen 30, and the infrared rays thereof There is no particular limitation as long as the infrared light emitted from the illumination 20 passes through the screen 30. Also, the number of the infrared illuminations 20 may be more than one. For example, a plurality of infrared lights 20 may be disposed on the left and right sides of at least one projector 10. In particular, by arranging a plurality of infrared illuminations 20, the projection control device 50 can detect the shape of the person HM1 (an example of an object) around the screen 30 with higher accuracy.

  Although not shown, in each of the embodiments disclosed in the present specification, the position where the projector 10 is disposed is determined as long as the projected image from the projector 10 is projected over the entire area of the screen 30. It is not particularly limited. Further, the number of the projectors 10 may be plural instead of one (refer to a second embodiment described later). For example, a plurality of projectors 10 may be disposed on the left and right sides of at least one infrared illumination 20. In addition, the plurality of projectors 10 may be arranged in the vertical direction at the upper and lower sides of at least one infrared illumination 20 or at both the upper and lower sides in the vertical direction and in the front-rear direction. In particular, by arranging a plurality of projectors 10, for example, when the screen 30 is large, the projection control device 50 generates and projects the same or different projection images from different projectors 10, thereby forming a large screen. It is possible to manage the execution of the projection of the image with improved entertainment to 30. However, in the case where a plurality of projectors 10 are arranged, it is necessary that the process of calibration in the above-described first embodiment be performed between each projector 10 and the infrared camera 40.

  Although not shown, in each embodiment disclosed in the present specification, the position at which the infrared camera 40 is disposed is disposed on the opposite side to the infrared illumination 20 with the screen 30 interposed therebetween, and the infrared ray thereof The camera 40 is not particularly limited as long as it can receive the infrared light transmitted through the screen 30. Also, the number of infrared cameras 40 may be more than one. For example, a plurality of infrared cameras 40 may be respectively disposed on the left and right sides of the screen 30 opposite to the infrared illumination 20. In addition, a plurality of infrared cameras 40 may be disposed on both sides of the screen 30 at the upper limit both sides in the vertical direction opposite to the infrared illumination 20, respectively. Also, a plurality of infrared cameras 40 may be respectively disposed on both sides of the upper limit of the vertical direction opposite to the plurality of projectors 10 and infrared illuminations 20 with the screen 30 interposed therebetween. In particular, by arranging a plurality of infrared cameras 40, the projection control device 50, for example, when the screen 30 is large, around the screen 30 based on different infrared captured images from different infrared cameras 40. The shape of at least one person can be recognized with high accuracy, and execution of projection of an image with improved entertainment can be managed on the large screen 30. However, when a plurality of infrared cameras 40 are arranged, it is necessary that the process of calibration in the above-described first embodiment be performed between each projector 10 and each infrared camera 40.

Second Embodiment
In the second embodiment, for example, an example in which a plurality of projectors are arranged in the projection control system 100 according to the first embodiment will be described (see FIG. 10). FIG. 10 is a diagram showing an installation example of the projection control system 100B according to the second embodiment. The projection control system 100B includes at least the projectors 10 and 10B, the infrared illumination 20, the screen 30, the infrared camera 40 (see FIG. 2), and the projection control device 50. In the configuration of FIG. 10, the same components as those of FIG. 1 are denoted by the same reference numerals, and description thereof will be simplified or omitted, and different contents will be described.

  The projector 10B as an example of the second projection device has a cylindrical case, for example, and is connected to be able to transmit and receive data or information with the projection control device 50. The projector 10B is disposed opposite to the screen 30 and on the person HM1 side. When the projector 10B receives and acquires data of a projection image including a projection instruction sent from the projection control device 50, the projector 10B positions the projection image designated by the projection instruction in front of the screen 30 based on the projection instruction. Projection light (for example, visible light) for projection onto the person HM1 is generated and projected toward the person HM1. Thus, the projector 10B can project the projection image (see FIG. 11) specified by the projection control device 50 exclusively to the person HM1 located in front of the projector 10B, not over the entire area of the screen 30. . Since the internal configuration of the projector 10B is the same as the internal configuration of the projector 10, the description will be omitted. The projector 10B may project the projection light toward an object other than the person HM1. Further, as described above, since the projector 10B is disposed in addition to the projector 10, it is necessary to execute the calibration process in the above-described first embodiment between the projector 10B and the infrared camera 40.

  FIG. 11 is a view showing an example of a projected image PJR2 according to the second embodiment. In FIG. 11, the same elements as the projected images PJR1 and PJR2 of FIG. 6 are assigned the same reference numerals to simplify or omit the description, and different elements will be described. As shown in FIG. 11, the projection control device 50 controls the background projected image PJGM1 to be an image content BB1 of a blue ball on the upper left of the person HM1, an image of a red ball on the upper right of the person HM1, for example. A projected image PJR2 in which the content RB1 is superimposed is generated. In FIG. 11, in order to make FIG. 11 easy to understand, the inside of the rectangular portion around the person HM1 in the background projection image PJGM1 is shown white so that the background projection image PJGM1 is not present. However, the projection image generation unit 553 may generate the background projection image PJGM1 that covers the entire area of the screen 30 so that the inside of the above-described rectangular portion also becomes the background projection image, as shown in FIG. A background projection image PJGM1 excluding the inside of the rectangular portion of the image may be generated. The projector 10 projects the projection image PJR2 generated by the projection control device 50 over the entire area of the screen 30. In addition, the projection control device 50 generates a projection image (for example, image content HT1) for projecting only on the person HM1 instead of the entire area of the screen 30, and issues a projection instruction including the generated projection image to the projector 10B. Output. For example, as in the first embodiment, the projection control device 50 can recognize the shape of the person HM1 based on the infrared captured image from the infrared camera 40, and can grasp the approximate position of the person HM1. Therefore, the projection control device 50 can generate a projection image (for example, the image content HT1) for narrowing and projecting to the approximate position (for example, the center position of the silhouette of the person HM1) of the recognized person HM1. . The projector 10B projects the projection image (for example, the image content HT1) specified by the projection instruction from the projection control device 50 toward the person HM1.

  In the second embodiment, the projection control device 50 executes the same projection control process (see FIG. 7) as that of the first embodiment, and different projection images to be projected on the respective projectors 10 and 10B. The generated and generated different projection images are projected on the respective projectors 10 and 10B. Therefore, the operation of the projection control device 50 of the second embodiment is the same as that of the first embodiment although the arrangement number of the projectors is different, and therefore the detailed description is omitted.

  As described above, in the projection control system 100B according to the second embodiment, in addition to the configuration of the projection control system 100 according to the first embodiment, the projection device is the projector 10 as an example of the first projection device, and the second And a projector 10B as an example of a projection device. The projection control device 50 generates a projection image (for example, a projection image PJR2 shown in FIG. 11) for causing the projector 10 to project and causes the projector 10 to project. In addition, the projection control device 50 generates another image content (for example, the image content HT1) within the shape of the person HM1 recognized based on the infrared captured image from the infrared camera 40 and causes the projector 10B to project it. .

  Thus, the projection control system 100B according to the second embodiment has the effects of the projection control system 100 according to the first embodiment and can execute projection of a projection image that can be projected onto the entire area of the screen 30 from the projector 10. The projector 10B can execute the projection of the projection image focused on the person HM1 around the screen 30. Therefore, since the projection control device 50 can grasp the general position of the person HM1, the projection control device 50 squeezes the recognized position of the person HM1 (for example, the center position of the silhouette of the person HM1) and the projection is performed by the projector 10 Projection image smaller than the projection image can be projected additionally to improve the diversity of the projection image.

Third Embodiment
In the third embodiment, for example, in the projection control system 100B according to the second embodiment, an example in which a visible light camera 40C for capturing visible light is further arranged will be described (see FIG. 12). FIG. 12 is a diagram showing an installation example of a projection control system 100C according to the third embodiment. The projection control system 100C is configured to include at least the projectors 10 and 10B, the infrared illumination 20, the screen 30, the infrared camera 40, the visible light camera 40C, and the projection control device 50C. In the configuration of FIG. 12, the same components as those of FIG. 10 are denoted by the same reference numerals, and description thereof will be simplified or omitted, and different contents will be described.

  The visible light camera 40C as an example of the second imaging device is connected so as to be able to transmit and receive data or information with the projection control device 50C. The visible light camera 40C is disposed so as to face the entire screen 30 so as to entirely include the entire area within the angle of view. The visible light camera 40C receives light including visible light transmitted through the screen 30, and picks up an image based on the received light to generate a visible light captured image. The visible light camera 40C sends the data of the generated visible light captured image to the projection control device 50C. Details of the visible light camera 40C will be described with reference to FIG.

  Note that, as shown in FIG. 12, the visible light camera 40 </ b> C is not disposed on the opposite side to the infrared camera 40 across the screen 30, and is disposed on the same side as the infrared camera 40. The visible light camera 40C is disposed, for example, around the projector 10B. Accordingly, the visible light camera 40C can capture a visible light captured image that can visually and specifically identify the shape of the person HM1 around the screen 30.

  The projection control device 50C is an information processing device capable of wired or wireless communication, such as a PC or a tablet terminal, and at least the projectors 10 and 10B, the infrared camera 40, the visible light camera 40C, the input device IP, and the display device DP Transmission / reception of data or information is possible. The projection control device 50C, based on the infrared captured image captured by the infrared camera 40 (in other words, the captured image generated based on the imaging of the infrared light transmitted through the screen 30), the periphery (eg, the screen 30) Recognize the presence or absence of a person HM1 (an example of a target person) and the shape indicating the position of the person HM1 with respect to the screen 30). The projection control device 50C generates a projection image according to the shape of the recognized person HM1, and outputs the generated projection image to the projector 10 that projects the projection image on the screen 30. Further, the projection control device 50C visually and specifically recognizes the shape of the person HM1 around (for example, in front of) the screen 30 based on the visible light captured image captured by the visible light camera 40C. The projection control device 50C generates a projection image according to the specific shape of the recognized person HM1, and outputs the generated projection image to the projector 10B that projects the projection image on the screen 30. The details of the projection control device 50C will be described with reference to FIG.

  FIG. 13 is a block diagram showing in detail an example of the system configuration of a projection control system 100C according to the third embodiment. In the configuration of FIG. 13, the same components as those of FIG. 2 are denoted by the same reference numerals, and description thereof will be simplified or omitted, and different contents will be described.

  The visible light camera 40C is configured to include a visible imaging unit 41C, a control unit 42C, a communication interface 43C, and a memory 44C. Also in the configuration of the visible light camera 40C of FIG. 13, the communication interface is similarly described as “communication I / F” for convenience.

  The visible image pickup section 41C is a visible light lens (not shown) capable of receiving visible light transmitted through the screen 30 and light including visible light from the surroundings, and an image sensor capable of picking up a visible light pickup image based on the light (Not shown). The output of the visible imaging unit 41C is sent to the control unit 42C.

  The control unit 42C is a processor PRC3 configured using, for example, a CPU, an MPU, a DSP, or an FPGA. The control unit 42C functions as a controller that controls the operation of the visible light camera 40C, performs control processing to generally control the operation of each unit of the visible light camera 40C, and receives data from each unit of the visible light camera 40C. It performs output processing, data calculation (calculation) processing, and data storage processing. The control unit 42C operates in accordance with the program and data stored in the memory 44C. Control part 42C may use memory 44C at the time of operation, and may save temporarily data or information which control part 42C generated or acquired in memory 44C. The control unit 42C generates data of a visible light captured image by performing predetermined image processing based on, for example, the output of the image sensor of the visible imaging unit 41C.

  The communication interface 43C has a role as a communication interface regarding transmission / reception (communication) of data or information with the projection control device 50C, and when the data of the visible light imaging image generated by the control unit 42C is acquired, it is acquired The data of the captured visible light captured image is transmitted to the projection control device 50C.

  The memory 44C is configured using, for example, a RAM and a ROM, and temporarily holds programs and data necessary for executing the operation of the visible light camera 40C, and further, data or information generated during the operation. The RAM is, for example, a work memory used when the visible light camera 40C operates. The ROM stores and holds, for example, a program and data for controlling the visible light camera 40C in advance.

  The projection control device 50C is configured to include an input / output interface 51, a memory 52C, a content storage database 53, a recording device 54, and a processor 55C. The input / output interface 51, the memory 52C, the content storage database 53, the recording device 54, and the processor 55C are mutually connected so as to be able to input / output data or information via the internal bus BS1.

  The memory 52C is configured using, for example, a RAM and a ROM, and temporarily holds programs and data necessary for executing the operation of the projection control device 50C, and further, data or information generated during the operation. The RAM is, for example, a work memory used when the projection control device 50C operates. The ROM stores and holds, for example, a program and data for controlling the projection control device 50C in advance. The memory 52C includes, for example, information of the resolution (specifically, vertical pixels, horizontal pixels) of the infrared captured image generated by the infrared camera 40, and the size of the screen 30 projected by the projector 10 (the Information on the area (a) and information on the resolution (specifically, pixels in the vertical direction and pixels in the horizontal direction) of the visible light imaging image generated by the visible light camera 40C are held.

  The processor 55C includes, for example, at least a calibration unit 551C, an object detection unit 552, a projection image generation unit 553 and a part detection unit 554. The calibration unit 551C, the object detection unit 552, the projection image generation unit 553, and the part detection unit 554 are functionally realized, for example, by the processor 55C reading and executing programs and data stored in advance in the memory 52C. It becomes possible.

  The calibration unit 551C is a position (specifically, coordinates) in the infrared captured image generated by the infrared camera 40 and a position (specifically, the positions in the projected images projected by the projectors 10 and 10B). Processing (that is, calibration) for obtaining a relational expression (for example, a projective transformation matrix) of conversion processing between coordinates) and a position (specifically, coordinates) in a visible light imaging image generated by the visible light camera 40C Do. Specifically, the calibration unit 551C specifies where the four corners of the projected image are located in the infrared captured image by designation by using the input device IP of the user or predetermined image processing (for example, edge detection processing). To detect. Further, the calibration unit 551C detects where the four corners of the projection image are located in the visible light captured image by predetermined image processing (for example, edge detection processing). The calibration unit 551C uses, for example, these detection results to obtain the above-described projective transformation matrix (see FIG. 14).

  FIG. 14 illustrates the coordinates (Xci, Yci) indicating the position of the infrared captured image of the infrared camera 40 and the coordinates (Xp, Yp) indicating the position of the projected image projected by the projector 10 and visible light imaging of the visible light camera 40C. It is explanatory drawing which shows an example of the calibration for the conversion between the coordinates (Xcv, Ycv) which show the position of an image. Although the projector 10 is illustrated in FIG. 14 to simplify the description, the same calibration is performed for the projector 10B. In the description of FIG. 14, the same elements as the description of FIG. 3 are assigned the same reference numerals to simplify or omit the description, and different contents will be described.

  Each of the end points EG1, EG2, EG3, and EG4 in the infrared captured image IRG is, for example, in a state where the infrared captured image IRG and the projected image PJR are displayed on the display device DP in comparison (see FIG. 14). These correspond to the end points CR1, CR2, CR3, and CR4 of the four corners of the projection image PJR selected by designation using the input device IP.

  Further, each of the end points EG1C, EG2C, EG3C, EG4C in the visible light captured image VSG is detected by, for example, image processing (for example, edge detection processing) performed by the calibration unit 551C, and the end points CR1, 4 at four corners of the projected image PJR. It corresponds to CR2, CR3 and CR4, respectively. Note that each of the end points EG1C, EG2C, EG3C, EG4C in the visible light captured image VSG has the visible light captured image VSG and the projected image PJR displayed in contrast on the display device DP (see FIG. 14), It may be selected by the calibration unit 551C by designation using the input device IP.

  The calibration unit 551C determines the infrared pickup image based on the coordinates of the end points EG1, EG2, EG3 and EG4 in the infrared pickup image IRG and the coordinates of the end points CR1 to CR4 of the four corners of the projection image PJR. A projective transformation matrix (first projective transformation matrix) for transforming coordinates (Xci, Yci) in IRG into coordinates (Xp, Yp) in projected image PJR is calculated and determined. Further, the calibration unit 551C determines visible light based on the coordinates of the end points CR1, CR2, CR3, and CR4 of the four corners of the projected image PJR and the coordinates of the end points EG1C, EG2C, EG3C, and EG4C in the visible light captured image VSG. A projective transformation matrix (second projective transformation matrix) for transforming coordinates (Xcv, Ycv) in the captured image VSG into coordinates (Xp, Yp) in the projection image PJR is calculated and determined. The calibration unit 551C uses the first projective transformation matrix and the second projective transformation matrix described above to coordinate the coordinates (Xci, Yci) in the infrared captured image IRG with the coordinates in the visible light captured image VSG A projective transformation matrix (third projective transformation matrix) for transforming to Xcv, Ycv) is calculated and obtained. The calibration unit 551C stores data or information of the determined projective transformation matrix (that is, each of the first to third projective transformation matrices) as a calibration result in the memory 52C.

  In the projection control system 100C shown in FIG. 12, the resolution (in other words, the size or area) of each projection image projected by the projectors 10 and 10B, and the positions of each of the infrared camera 40 and the visible light camera 40C Unless changed, the process of calibration in the calibration unit 551C may be performed at least once. In other words, each time the resolution of each projection image projected by the projectors 10 and 10B or the position of each of the infrared camera 40 or the visible light camera 40C is changed, the calibration processing in the calibration unit 551C is performed at least once. To be executed.

  Further, the method of calibration in the calibration unit 551C is not limited to the method described above. For example, the calibration unit 551C can also convert coordinates in the infrared captured image IRG to corresponding coordinates in the projection image PJR in pixel units by executing calibration using a known structured light method. It is. Similarly, the calibration unit 551 C can also convert coordinates in the visible light captured image VSG to corresponding coordinates in the projected image PJR in pixel units, and coordinates in the infrared captured image IRG in pixel units It is also possible to convert to corresponding coordinates in the visible light captured image VSG.

  The parts detection unit 554 uses the shape of the person HM1 (an example of the target object) detected by the object detection unit 552 and the data of the visible light captured image generated by the visible light camera 40C. The parts of the person HM1 (for example, a part of the body) are detected within the shape of the detected person HM1. The parts detection unit 554 detects, as parts (parts) of the person HM1, for example, small parts such as face, eyes, mouth, nose, eyebrows, forehead, hands, feet, elbows, knees, eyebrows, cheeks, etc. of the person HM1. It is possible. In addition, each of these site | parts is an illustration to the last, and the parts detected are not limited to these. The parts detection unit 554 passes the detection result of the detected parts of the person HM1 to the projection image generation unit 553.

  The projection image generation unit 553 uses the detection result passed from the part detection unit 554 and the calibration result held in the memory 52C to indicate the position (coordinates) of the part in the shape of the person HM1 in the infrared captured image The projected image is generated by superimposing the image content read out from the content storage database 53 at a position (coordinates) in the projected image corresponding to each of (a) or a position (coordinates) separated from the position (coordinates) by a predetermined distance. The projection image generation unit 553 generates a projection instruction including data of the generated projection image, and sends the generated projection instruction to the projector 10B through the input / output interface 51.

  FIG. 15 is a view showing an example of a projected image PJR2 according to the third embodiment. In FIG. 15, the same elements as the projected images PJR1 and PJR2 of FIG. 6 are assigned the same reference numerals to simplify or omit the description, and different elements will be described. As shown in FIG. 15, the projection control device 50C, for example, the image content BB1 of the blue ball on the upper left of the person HM1 and the image of the red ball on the upper right of the person HM1 with respect to the background projection image PJGM1 serving as the background portion. A projected image PJR2 in which the content RB1 is superimposed is generated. In FIG. 15, in order to make FIG. 15 easy to understand, the rectangular portion around the person HM1 in the background projection image PJGM1 is shown white so that the background projection image PJGM1 is not present. However, the projection image generation unit 553 may generate the background projection image PJGM1 that covers the entire area of the screen 30 so that the inside of the above-described rectangular portion also becomes the background projection image, as shown in FIG. A background projection image PJGM1 excluding the inside of the rectangular portion of the image may be generated. The projector 10 projects the projection image PJR2 generated by the projection control device 50 over the entire area of the screen 30. Also, the projection control device 50C generates a projection image (for example, the image content CK1) for squeezing and projecting to the part (for example, the cheek of the person HM1) detected by the parts detection unit 554, and generates the generated projection A projection instruction including an image is output to the projector 10B. For example, as in the first embodiment, the projection control device 50C can recognize the shape of the person HM1 based on the infrared captured image from the infrared camera 40, and further, based on the visible light captured image from the visible light camera 40C. Thus, it is possible to know the exact position of the part of the person HM1. Therefore, the projection control device 50C can generate a projection image (for example, the image content CK1) for narrowing and projecting the recognized person HM1 to the correct position (for example, the position of the cheek of the person HM1). The projector 10B projects the projection image (for example, the image content CK1) designated by the projection instruction from the projection control device 50C toward the cheek of the person HM1.

  Next, an operation procedure of the projection control system 100C according to the third embodiment will be described with reference to FIG. FIG. 16 is a flowchart for explaining an example of the operation procedure of the projection control process according to the third embodiment. Each process shown in FIG. 16 is executed by, for example, the projection control device 50C. Further, in the description of FIG. 16, the same processing as that of FIG. 7 is assigned the same step number to simplify or omit the description, and different contents will be described.

  Further, in the projection control process of FIG. 16, parts in the shape of the person HM1 are detected by the projection control device 50C using the visible light captured image captured by the visible light camera 40C, and image contents (for example, an image content) An example in which the content CK1) is projected by the projector 10B is illustrated. Further, the projection control device 50C according to the third embodiment executes the same process as the projection control process (see the first embodiment) shown in FIG. 7 in parallel, separately from the projection control process shown in FIG. Good.

  The calibration process in step S2 of FIG. 16 is the calibration process according to the present embodiment (see FIG. 14). Specifically, the position (coordinates) in the infrared captured image is visible light This is processing for obtaining a relational expression for converting to a position (coordinates) in a captured image.

  In FIG. 16, when the object detection unit 552 can detect the shape of the person HM1 (S4, YES), the parts detection unit 554 detects the shape of the person HM1 detected by the object detection unit 552, and the visible light camera 40C. The part of the person HM1 is detected within the shape of the person HM1 detected by the object detection unit 552 using the data of the visible light captured image generated by the above (S9). The parts detection unit 554 passes the detection result of the detected parts of the person HM1 to the projection image generation unit 553. The projection image generation unit 553 determines the projection position of the image content to be projected by the projector 10B according to the position (that is, the detection position) in the shape of the person HM1 of the part detected by the parts detection unit 554 (S5C) ). The projection position indicates, for example, the position (coordinates) of the image content CK1 shown in FIG. The processes after step S5C in FIG. 16 are the same as the processes after step S5 in FIG.

  As described above, in addition to the configuration of the projection control system 100B according to Embodiment 2, the projection control system 100C according to Embodiment 3 further includes a visible light camera 40C as an example of a second imaging device. The visible light camera 40C captures a visible light captured image including the screen 30 as an angle of view. The projection control device 50C recognizes the internal part (part) of the shape of the person HM1 recognized based on the infrared pickup image based on the visible light pickup image, and responds to the recognized part (part). Image content (for example, image content CK1) is generated. The projection control device 50C causes the projector 10B as an example of the second projection device to project the generated image content (for example, the image content CK1).

  Thus, the projection control system 100C according to the third embodiment has the effect of the projection control system 100B according to the second embodiment, and can execute projection of a projection image that can be projected onto the entire area of the screen 30 from the projector 10. From the projector 10B, it is possible to execute projection of a projection image which is accurately focused on the part of the person HM1 around the screen 30. Therefore, the projection control device 50C can grasp the correct part of the person HM1, narrows down to the correct part of the recognized person HM1 (for example, the cheeks of the face of the person HM1), and the projection image projected by the projector 10 Even small local projection images can be added and projected, and it is possible to improve the variety of entertainment according to the projection images, such as enabling the projection of fine projection images.

Second Modification of First Embodiment
In the second modification of the first embodiment, for example, in the projection control system 100 according to the first embodiment, an example of further performing control of power on / off of the infrared illumination 20 and switching of the mode of the transparent screen 30D. Will be described (see FIG. 17). FIG. 17 is a diagram showing an installation example of a projection control system 100D according to a second modification of the first embodiment. FIG. 18 is a block diagram showing in detail an example of a system configuration of a projection control system 100D according to a second modification of the first embodiment. The projection control system 100D is configured to include at least the projector 10, the infrared illumination 20, the transparent screen 30D, the infrared camera 40, the projection control device 50, the illumination control device 60, and the screen control device 70. In the configuration of FIG. 17, the same components as those of FIG. 1 are denoted by the same reference numerals, and description thereof will be simplified or omitted, and different contents will be described.

  A transparent screen 30D as an example of an object to be projected is a screen that can switch the mode relating to the screen to the transparent mode or the screen mode under control of the screen control device 70, and is fixedly installed. The front and back sides of the transparent screen 30D are in accordance with FIG. The specific configuration of the transparent screen 30D is disclosed, for example, in the above-mentioned Reference Non-Patent Document 1, so the detailed description is omitted here. The transparent screen 30D can transmit infrared light emitted from the infrared illumination 20 in both the screen mode and the transparent mode. Infrared light transmitted through the transparent screen 30D is received by the infrared camera 40.

  The lighting control device 60 as an example of the first control unit controls the power supply of the infrared light 20 by switching on or off the power of the infrared light 20 according to the lighting control signal from the projection control device 50.

  The screen control device 70 as an example of the second control unit switches the mode of the transparent screen 30D to either the transparent mode or the screen mode according to the screen control signal from the projection control device 50, thereby the mode of the transparent screen 30D. Control.

  As described above, in the second modification of the first embodiment, when switching the transparent screen 30D to the screen mode, the projection control device 50 controls the power supply of the infrared light 20 to turn on to turn on the red light from the infrared light 20. I let outside light. That is, the projection control device 50 generates a screen control signal for switching the transparent screen 30D to the screen mode, sends it to the screen control device 70, and generates a lighting control signal for turning on the infrared lighting 20. And sends it to the lighting control device 60. Thereby, the illumination control device 60 turns on the power of the infrared illumination 20 to execute the irradiation of the infrared light, and the screen control device 70 can switch the transparent screen 30D to the screen mode, and the infrared light in the transparent screen 30D Can be transparent.

  Further, in the second modification of the first embodiment, when switching the transparent screen 30D to the transparent mode, the projection control device 50 controls the power supply of the infrared light 20 to be turned off so that the infrared light 20 transmits infrared light. Do not irradiate light. That is, the projection control device 50 generates a screen control signal for switching the transparent screen 30D to the transparent mode and sends it to the screen control device 70, and generates a lighting control signal for turning off the infrared lighting 20. And sends it to the lighting control device 60. Thereby, the screen control device 70 switches the transparent screen 30D to the transparent mode, and the illumination control device 60, for example, does not need various processing (for example, recognition of the person HM1 and projection control processing) in the projection control device 50. The power of the infrared illumination 20 can be turned off to stop the irradiation of the infrared light, and the unnecessary irradiation of the infrared light can be suppressed.

Third Modified Example of First Embodiment
In the third modification of the first embodiment, for example, in the projection control system 100 according to the first embodiment, an example will be described in which the projection control device 50 is connected to an external server connected via the network NW (see FIG. 19). FIG. 19 is a diagram showing an installation example of a projection control system 100 according to a third modification of the first embodiment. The projection control system 100 includes at least a projector 10, an infrared illumination 20, a screen 30, an infrared camera 40, a projection control device 50, and a plurality of content storage devices CLD1, ..., CLDk. k is an integer of 2 or more. In the configuration of FIG. 19, the same components as those of FIG. 1 are denoted by the same reference numerals, and description thereof will be simplified or omitted, and different contents will be described.

  Each of the content storage devices CLD1 to CLDk as an example of the external server is configured using, for example, an HDD or an SSD, and stores and holds data of image content to be superimposed on a projection image projected by the projector 10 . Each of the content storage devices CLD1 to CLDk is connected so as to be able to transmit and receive data or information with the projection control device 50 via the network NW. The network NW may be a wired network (for example, a LAN (Local Area Network)) or a wireless network (for example, a wireless LAN).

  The projection control device 50, for example, periodically accesses each of the content storage devices CLD1 to CLDk to obtain data of the image content stored and held in each of the content storage devices CLD1 to CLDk, and the device itself (that is, , The data of the image content in the content storage database 53 in the projection control device 50).

  As described above, in the third modification of the first embodiment, the projection control device 50 is connected to each of the content storage devices CLD1 to CLDk that store and hold the image content to be superimposed when generating the projection image, and the content storage device The image content is acquired from each of CLD1 to CLDk, and the image content stored and held in each of the content storage devices CLD1 to CLDk is updated. Thus, the projection control device 50 stores and holds the image content data in the content storage database 53 in the own device (that is, the projection control device 50), for example, periodically. It is possible to acquire non-image content and accumulate various image content.

  As mentioned above, although various embodiments were described, referring to an accompanying drawing, this indication is not limited to this example. It is obvious that those skilled in the art can conceive of various modifications, alterations, replacements, additions, deletions and equivalents within the scope of the claims. It is understood that within the technical scope of the present disclosure. Moreover, in the range which does not deviate from the meaning of invention, you may combine each component in various embodiment mentioned above arbitrarily.

  The present disclosure is useful as a projection control system and a projection control method that detects the position of a detection target within the imaging angle of view with high accuracy, and adaptively supports accurate projection of content following the movement of the detection target. is there.

10, 10B Projector 11, 43, 43C Communication Interface 12, 42, 42C Control Unit 13 Projection Unit 14, 44, 44C, 52, 52C Memory 20 Infrared Illumination 30 Screen 40 Infrared Camera 40C Visible Light Camera 41 Infrared Imaging Unit 41C Visible Imaging unit 50, 50C Projection control unit 51 Input / output interface 53 Content storage database 54 Recording unit 55, 55C, PRC1, PRC2 Processor 551, 551C Calibration unit 552 Object detection unit 553 Projected image generation unit 554 Part detection unit DP Display device IP Input device

Claims (14)

A light source device for emitting infrared light;
An imaging device configured to receive an infrared imaging image including the object to be projected as an angle of view, the irradiated infrared light receiving light transmitted through the object to be projected;
The shape of an object present around the object to be projected is recognized based on the captured infrared image, the projection image corresponding to the shape of the recognized object is generated, and the generated image is generated. And a projection control device for outputting the projection image to a projection device that projects the projection image onto the object.
The light source device
It is disposed on the opposite side of the imaging device with the object to be projected interposed therebetween.
Projection control system. The projection control device
A content holding unit for holding image content, and generating an image in which the image content is superimposed near the shape of the recognized object as the projection image;
The projection control system according to claim 1. The projection control device
A content holding unit for holding image content, and generating, as the projection image, an image in which the image content is superimposed at a predetermined distance from the recognized object;
The projection control system according to claim 1. The predetermined distance is changed according to the user's input.
The projection control system according to claim 3. An object region where the object is located is located on the imaging device side of the projection object,
The projection control system according to claim 1. The imaging device is
An imaging unit for receiving the infrared light, and a filter or a lens for cutting visible light is disposed in front of the imaging unit;
The projection control system according to claim 1. An object region where the object is located is located on the opposite side of the object to be projected to the imaging device side.
The projection control system according to claim 1. The light source device
A diffusion sheet is disposed on the front of the device to expand the irradiation range of the infrared light,
The projection control system according to claim 1. The projection device
At least a first projection device and a second projection device,
The projection control device
Causing the first projection device to project the generated projection image;
Causing the second projection device to project other image content within the shape of the recognized object
A projection control system according to claim 1 or 2. And a second imaging device for capturing a visible light imaging image including the object to be projected as an angle of view.
The projection device
At least a first projection device and a second projection device,
The projection control device
A portion within the shape of the object recognized based on the infrared captured image is recognized based on the visible light captured image, and image content corresponding to the recognized portion is generated and generated. Projecting the image content onto the second projection device,
A projection control system according to claim 1 or 2. The projection control device
A first control unit that controls on / off of the irradiation of the infrared light in the light source device;
And a second control unit for switching the transparent mode or the screen mode of the object to be projected.
When switching the object to be projected to the screen mode, the light source device is caused to emit the infrared light.
The projection control system according to claim 1. The projection control device
A first control unit that controls on / off of the irradiation of the infrared light in the light source device;
And a second control unit for switching the transparent mode or the screen mode of the object to be projected.
When the object to be projected is switched to the transparent mode, the light source device is not irradiated with the infrared light.
The projection control system according to claim 1. The projection control device
It is connected to an external server that stores image content to be superimposed when the projection image is generated, acquires the image content from the external server, and updates the image content held in the content holding unit.
The projection control system according to claim 2. A projection control method in a projection control system,
The infrared light is emitted by the light source device,
The irradiated infrared light receives the light transmitted through the object to be projected, and the imaging device captures an infrared captured image including the object to be projected as an angle of view.
Recognizing the shape of an object around the projection object based on the captured infrared image.
Generating a projection image according to the shape of the recognized object, and outputting the generated projection image to a projection device that projects the projection image onto the projection object;
The light source device
It is disposed on the opposite side of the imaging device with the object to be projected interposed therebetween.
Projection control method.