JP2022069007A - Information processing system and information processing method and information processing program - Google Patents

Abstract

To provide an information processing system, an information processing method, and an information processing program that can reduce a production cost of a video including a three-dimensional performance.SOLUTION: An information processor comprises an UI provision part providing a configuration screen for configuring a three dimensional effect, an acquisition part acquiring scan data from a sensor that acquires geography by scanning a space, a rendering part that renders a three dimensional effect set through the configuration screen provided by the UI provision part on the scan data, and a display part making the three dimensional effect rendered by the rendering part displayed.SELECTED DRAWING: Figure 1

Description

There is an application for exception of loss of novelty

The present invention relates to an information processing system, an information processing method and an information processing program.

Conventionally, there has been a technique of superimposing an image on data scanned by an optical sensor called LIDAR (Light Detection and Ringing). For example, Patent Document 1 discloses a technique of rendering a graphical element and displaying it on an AR display by rendering point cloud data from a lidar sensor.

Japanese Unexamined Patent Publication No. 2018-206366

However, in the conventional technique, only a graphical element is added to the scan data from the LIDAR sensor, and the scan data itself is not used for the production of the image. Therefore, in order to produce a video including a three-dimensional effect, it is necessary to separately prepare three-dimensional effect data, which may increase the production cost.

The present invention has been made in view of the above circumstances, and one object of the present invention is to provide an information processing system, an information processing method, and an information processing program capable of reducing the production cost of an image including a three-dimensional effect. And.

(1) In order to solve the above problems, the information processing device receives scan data from a UI provider that provides a setting screen for setting a three-dimensional effect and a sensor that scans the space and acquires the terrain. Displays the drawing unit that draws the 3D effect set via the acquisition unit and the setting screen provided by the UI providing unit on the scan data, and the drawing unit that draws the 3D effect drawn by the drawing unit. It is provided with a display unit to be displayed.

(2) Further, in the information processing apparatus of the embodiment, the acquisition unit acquires scan data as mesh data, and the drawing unit draws a three-dimensional effect on the mesh data acquired by the acquisition unit. May be.

(3) Further, in the information processing apparatus of the embodiment, the drawing unit determines whether or not the scan data acquired by the acquisition unit has been updated, and when the scan data is updated, the three-dimensional effect is updated. And draw it.

(4) Further, in the information processing apparatus of the embodiment, even if the UI providing unit provides a setting screen for setting the drawing order of the three-dimensional effect in the display range displayed on the display unit. good.

(5) Further, in the information processing apparatus of the embodiment, the drawing unit may draw a three-dimensional effect by performing rendering using a Bezier curve on the scan data.

(6) In order to solve the above problem, the information processing method is an information processing method using an information processing device, and a UI providing step for providing a setting screen for setting a three-dimensional effect and a space are provided. The acquisition step of acquiring the scan data from the sensor that scans and acquires the terrain, and the drawing step of drawing the 3D effect set via the setting screen provided in the UI provision step on the scan data. , Includes a display step to display the three-dimensional effect drawn in the drawing step.

(7) In order to solve the above problems, the information processing program is provided with a UI providing process that provides a setting screen for setting a three-dimensional effect to the computer, and a sensor that scans the space and acquires the terrain. The acquisition process for acquiring scan data, the drawing process for drawing the 3D effect set via the setting screen provided in the UI provision process for the scan data, and the 3D drawing process drawn in the drawing process. Execute the display process to display the effect.

According to one embodiment of the present invention, the information processing apparatus provides a setting screen for setting a three-dimensional effect, and acquires and provides scan data from a sensor that scans a space and acquires a terrain. By drawing the 3D effect set via the set screen on the scan data and displaying the drawn 3D effect, the production cost of the video including the 3D effect is reduced. be able to.

It is a block diagram which shows an example of the structure of the information processing apparatus in embodiment. It is a figure which shows the 1st display example of the information processing apparatus in embodiment. It is a figure which shows the 2nd display example of the information processing apparatus in embodiment. It is a block diagram which shows an example of the hardware composition of the information processing apparatus in embodiment. It is a flowchart which shows 1st example of the operation of the information processing apparatus in embodiment.

Hereinafter, the information processing apparatus, the information processing method, and the information processing program according to the embodiment of the present invention will be described in detail with reference to the drawings.

First, the configuration of the information processing apparatus will be described with reference to FIG. FIG. 1 is a block diagram showing an example of the configuration of the information processing apparatus according to the embodiment.

In FIG. 1, the information processing apparatus 1 has a functional unit of a UI providing unit 11, an acquisition unit 12, a drawing unit 13, and a display unit 14. Each functional unit of the information processing apparatus 1 will be described as being realized by software. The information processing device 1 is, for example, a smartphone, a tablet PC, or a head-mounted, glasses-type, or wristwatch-type device. The information processing apparatus 1 is communicably connected to the server 2 via the network 9.

The UI providing unit 11 provides a setting screen (UI: User Interface) for setting a three-dimensional effect. The acquisition unit 12 acquires scan data from a sensor that scans the space and acquires the terrain. The drawing unit 13 draws a three-dimensional effect set via the setting screen provided by the UI providing unit 11 on the scan data acquired by the acquiring unit 12. Further, the display unit 14 displays the three-dimensional effect drawn by the drawing unit 13.

A sensor that scans space to acquire terrain (hereinafter, may be abbreviated as "sensor") is a sensor that scans the shape of an object existing in space, for example, a sensor using lidar technology. be. A sensor using the LIDAR technique can measure the shape of an object existing in space in three dimensions by measuring the distance to the object (distance measurement) using, for example, a laser beam. The sensor using the lidar technique may use ultraviolet rays, infrared rays, or near infrared rays. The function of the sensor may be realized in Vuforia (registered trademark) or AR Foundation (registered trademark) for realizing AR (Augmented Reality) in Unity (registered trademark). Further, the sensor may be, for example, a stereo camera.

The scan data is three-dimensional data created based on the three-dimensional distance measurement data of an object existing in the space measured by the sensor, and is data expressing the shape of the object existing in the space. The scan data is, for example, point cloud data (mesh data) in which the outer shape of an object in space is represented by a set of points having three-dimensional coordinate values. The scan data may be line data indicating a straight line connecting point cloud data, a Bezier curve, or a free curve. Further, the scan data may be surface data indicating a plane or a curved surface including point cloud data. The three-dimensional data includes, for example, data obtained by scanning a plane such as a road, a floor surface, or a wall surface.

The three-dimensional effect is a three-dimensional effect added to the scan data. The 3D effect is, for example, the addition of an image of an object represented by a point, line or surface to the 3D surface of an object existing in the space represented by the point group data, line data or surface data described above. Or, coloring the image of the added object, etc.

The image of the object to be added may be either a still image or a moving image. When the object to be added is a moving object, the three-dimensional effect can be, for example, an effect in which the object moves in a wavy manner on the surface of the three-dimensional object. Further, when the object to be added is a moving object, the three-dimensional effect may be an effect in which the object moves as if moving on the surface of the three-dimensional object. Further, when the added object is a still image, the three-dimensional effect may be an effect such that the object (for example, a texture) is attached to the surface of the three-dimensional object.

Further, the three-dimensional effect may be added sequentially to the scan data. For example, the three-dimensional effect may be sequentially added from the front side (closer side to the sensor) to the back side (far side from the sensor) of the scan data in the three-dimensional object. As a result, it is possible to create an effect in which the object spreads from the front side to the back side of the three-dimensional object.

Further, the three-dimensional effect may be sequentially added to the sequentially acquired scan data. For example, when the user moves (for example, PAN) the direction of the sensor of the information processing apparatus 1 to sequentially acquire horizontal or vertical scan data, the three-dimensional effect is sequentially obtained by moving the sensor. It is added sequentially to the acquired scan data. As a result, it is possible to produce an effect in which the object is sequentially added to the three-dimensional object according to the movement of the sensor.

Further, for example, even when a moving object is scanned by a fixed sensor, scan data is sequentially acquired as the object moves. Even in this case, the three-dimensional effect is sequentially added to the scan data sequentially acquired by the movement of the object, and the object can be sequentially added to the three-dimensional object.

The UI providing unit 11 provides a setting screen for setting a three-dimensional effect. The setting screen is a setting screen for setting the display mode of the various types of three-dimensional effects described above. The settings screen may display, for example, the type of 3D effect, the drawing speed of the object, the color of the object, or the type of object that exists in space (eg, people, animals, buildings, walls, floors, heavens, furniture, cars). , Electric poles, road signs, etc.), and includes a UI for the user to set a display mode of a three-dimensional effect. The setting screen may include a UI for selecting a preset effect.

Further, the UI providing unit 11 may set the drawing order of the objects in the three-dimensional effect. The drawing order of the objects is, for example, the order of drawing the objects from the front side to the back side of the display screen, the order of drawing the objects from the back side to the front side, the order of drawing the objects from the right side to the left side, and the like. Further, the drawing order of the objects may be a drawing order or the like according to the type of the object.

The drawing unit 13 draws a three-dimensional effect set via the setting screen provided by the UI providing unit 11 on the scan data. The drawing unit 13 draws the image of the scan data so as to superimpose the image of the three-dimensional effect. The image superposition may be such that the image layer of the scan data (data layer) and the layer of the effect image are drawn as the same layer, or may be drawn as different layers. By drawing so that the image of the three-dimensional effect is superimposed on the image of the scan data, it is possible to create an effect that the effect is added to the image of the object existing in the space.

Further, the drawing unit 13 may draw only the image of the three-dimensional effect without drawing the image of the scan data. By drawing only the image of the three-dimensional effect, it is possible to create an effect in which the effect appears in an empty space.

Further, the drawing unit 13 may superimpose an image of a three-dimensional effect on a captured image captured by a camera (not shown) included in the information processing apparatus 1. By superimposing the image of the three-dimensional effect on the captured image, it is possible to produce the three-dimensional effect in AR including the captured image.

The drawing unit 13 may draw a three-dimensional effect by performing rendering using a Bezier curve on the scan data. The Bezier curve is an N-1 degree curve obtained from N control points. The drawing unit 13 can represent a three-dimensional object by calculating a Bezier curve for a point cloud of scan data and connecting and rendering the calculated Bezier curve. For example, the drawing unit 13 can draw a three-dimensional effect by executing graphic processing on the scan data using graphic software having the above-mentioned function.

The display unit 14 displays, for example, the three-dimensional effect drawn by the drawing unit 13 on the display unit of the information processing apparatus 1. The display unit of the information processing device 1 is, for example, a liquid crystal display device, a touch panel, a head mount type, glasses type, or a wristwatch type display.

The display unit 14 may display a captured image captured by a camera (not shown) included in the information processing apparatus 1. By displaying the scan data and the captured image at the same time, it is possible to produce a three-dimensional effect in AR including the captured image.

The server 2 has an effect providing unit 21. The effect providing unit 21 stores a preset display mode of the three-dimensional effect as a setting file and provides it to the information processing apparatus 1. For example, the effect providing unit 21 stores the display mode of the three-dimensional effect set from the setting screen provided by the UI providing unit 11. By storing the display mode of the three-dimensional effect, the effect providing unit 21 can provide the set display mode of the three-dimensional effect to the information processing apparatus 1, and display the three-dimensional effect. Allows reuse of aspects. The effect providing unit 21 may share the display mode by storing the display mode set by a person other than the user and providing the display mode to the information processing apparatus 1.

The effect providing unit 21 may authenticate the user and provide the display mode to the authenticated user. Further, the effect providing unit 21 may authenticate only the charged user and provide the display mode.

In the present embodiment, the term "provided" may be either push-type transmission or pull-type transmission. Further, in the present embodiment, the term "acquisition" may be a pull-type reception or a push-type reception.

Further, each of the above-mentioned functional units possessed by the information processing apparatus 1 shows an example of the functions, and does not limit the functions possessed by the information processing apparatus 1. For example, the information processing apparatus 1 does not have to have all the above-mentioned functional parts, and may have some of the functional parts. Further, the information processing apparatus 1 may have a function other than the above. For example, the functional unit of the information processing apparatus 1 may be realized in the server 2. Further, the functional unit of the server 2 may be realized in the information processing apparatus 1.

Further, as described above, each of the above functional units has been described as being realized by software. However, at least one or more functional units in the above functional units may be realized by hardware.

Further, any of the above functional units may be implemented by dividing one functional unit into a plurality of functional units. Further, any two or more of the above functional units may be integrated into one functional unit. FIG. 1 shows the functions of the information processing apparatus 1 represented by functional blocks, and does not show, for example, that each functional unit is composed of a separate program file or the like.

Further, the information processing device 1 may be a device realized by one housing or a system realized by a plurality of devices connected via a network or the like. For example, the information processing device 1 may realize a part or all of its functions by another virtual device such as a cloud service provided by a cloud computing system. That is, the information processing apparatus 1 may realize at least one or more functional units among the above-mentioned functional units in another device.

Next, a display example of the display device of the information processing device 1 will be described with reference to FIGS. 2 to 3.

FIG. 2 is a diagram showing a first display example of the information processing apparatus according to the embodiment.

In FIG. 2, the display screen 1000 is displayed on the display unit 14. The display screen 1000 displays a photographed image taken by a camera of a person walking beside a building with a three-dimensional effect added. In FIG. 2, the object by the three-dimensional effect is not drawn on the human part. In order not to draw the object by the three-dimensional effect on the human part, for example, the transparent object 1001 may be drawn on the human part. The drawing unit 13 may recognize a person by extracting a feature amount indicating a person from the scan data. The feature amount is, for example, a distribution (histogram) of color data existing in space, a shape of an object, or vector data of color data. Objects with similar features can be recognized as the same object.

Further, the display screen 1000 displays an object 1002 that fills the floor surface on which a person is walking. The wavy moving object 1003 is displayed on the object 1002. Further, the display screen 1000 displays a blind-shaped object 1004 on a wall surface on the side where a person is walking. The object 1004 is a still image object, and is drawn sequentially from the front side to the back side of the drawing.

FIG. 3 is a diagram showing a second display example of the information processing apparatus according to the embodiment.

In FIG. 3, the display screen 2000 is displayed on the display unit 14. The display screen 2000 displays a photographed image taken by a camera of a person walking in the city with a three-dimensional effect added. In FIG. 3, the object 2001, which is three-dimensionally colored with respect to the shape of the human, is drawn and displayed on the human portion. As a person walks, the shape of the person changes, so the coloring of the object changes and is displayed. Further, on the display screen 2000, the mosaic object 2002 is drawn and displayed on the vending machine installed facing the road. In addition, objects by the three-dimensional effect are not drawn on other parts such as roads and buildings. Here, although the vending machine is fixed, the distance from the sensor changes and the mosaic display changes by PANing the sensor.

Next, the hardware configuration of the information processing apparatus 1 will be described with reference to FIG. FIG. 4 is a block diagram showing an example of the hardware configuration of the information processing apparatus 1 according to the embodiment. The hardware configuration of the server 2 is similar to that of the information processing device 1, and the description thereof will be omitted.

The information processing device 1 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, an I / O device 104, and a communication I / F (Interface) 105. The information processing device 1 is a device that executes the information processing program described with reference to FIG.

The CPU 101 controls the user terminal by executing the information processing program stored in the RAM 102 or the ROM 103. The information processing program is acquired from, for example, a recording medium on which the program is recorded, a program distribution server via a network, or the like, installed in the ROM 103, read from the CPU 101, and executed.

The I / O device 104 has an operation input function and a display function (operation display function). The I / O device 104 is, for example, a touch panel. The touch panel enables the user of the information processing terminal 10 to input operations using a fingertip, a stylus, or the like. The case where the I / O device 104 in the present embodiment uses a touch panel having an operation display function will be described, but the I / O device 104 separates the display device having the display function and the operation input device having the operation input function. It may have. In that case, the display screen of the touch panel can be performed as the display screen of the display device, and the operation of the touch panel can be performed as the operation of the operation input device. The I / O device 104 may be realized by various forms such as a head mount type, a glasses type, and a wristwatch type display.

The communication I / F 105 is an I / F for communication. The communication I / F 105 executes short-range wireless communication such as a wireless LAN, a wired LAN, and infrared rays. Although the figure shows only the communication I / F 105 as the communication I / F, the information processing terminal 10 may have an I / F for each communication in a plurality of communication methods.

Next, the operation of the information processing apparatus 1 will be described with reference to FIG.

FIG. 5 is a flowchart showing the operation of the information processing apparatus 1 in the embodiment. The flowchart shown in the first example describes the operation related to the arrangement of the anchor and the sound source.

In FIG. 5, the information processing apparatus 1 provides a setting screen for setting a three-dimensional effect (step S11). The provision of the setting screen is executed, for example, by the user activating the information processing program installed in the information processing apparatus 1. The user sets the three-dimensional effect via the provided setting screen and saves the effect setting information in the information processing apparatus 1. The effect setting information can be saved, for example, by storing the effect setting information in the RAM 102.

After executing the process of step S11, the information processing apparatus 1 acquires the scan data from the sensor that scans the space and acquires the terrain (step S12).

After executing the process of step S12, the information processing apparatus 1 determines whether or not the effect setting has been changed (step S13). Whether or not the effect setting has been changed can be determined by, for example, whether or not the user has changed the three-dimensional effect setting using the setting screen provided in step S11.

When it is determined that the effect setting has been changed (step S13: YES), the information processing apparatus 1 changes the setting of the effect to be drawn for the scan data and saves the setting information (step S14).

After executing the process of step S14, or when it is determined that there is no change in the effect (step S13: NO), the information processing apparatus 1 applies the effect according to the setting information of the effect saved for the scan data. Draw (step S15).

After executing the process of step S15, the information processing apparatus 1 causes the display device to display the three-dimensional effect drawn in step S15 (step S16). After executing the process of step S16, the information processing apparatus 1 ends the operation shown in the flowchart.

It should be noted that the illustrated flowchart shows an example of the operation, and does not limit the operation.

Further, a program for realizing the function constituting the apparatus described in the present embodiment is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. Therefore, the above-mentioned various processes of the present embodiment may be performed. The "computer system" here may include hardware such as an OS and peripheral devices. Further, the "computer system" includes the homepage providing environment (or display environment) if the WWW system is used. The "computer-readable recording medium" includes a flexible disk, a magneto-optical disk, a ROM, a writable non-volatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, and the like. Refers to the storage device of.

Further, the "computer-readable recording medium" is a volatile memory inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line (for example, DRAM (Dynamic)). It also includes those that hold the program for a certain period of time, such as Random Access Memory)). Further, the program may be transmitted from a computer system in which this program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the above program may be for realizing a part of the above-mentioned functions. Further, a so-called difference file (difference program) may be used, which realizes the above-mentioned function in combination with a program already recorded in the computer system.

Although the embodiment of the present invention has been described above with reference to the drawings, the specific configuration is not limited to this embodiment and includes various changes within the range not deviating from the gist of the present invention. Will be.

1 Information processing device 11 UI providing unit 12 Acquisition unit 13 Drawing unit 14 Display unit 2 Server 21 Production providing unit 9 Network 101 CPU
102 RAM
103 ROM
104 I / O equipment 105 Communication I / F
1000 display screen 1001 object 1002 object 1003 object 2000 display screen 2001 object 2002 object

Claims (7)

A UI provider that provides a setting screen for setting 3D effects,
An acquisition unit that acquires scan data from a sensor that scans space and acquires terrain,
A drawing unit that draws a three-dimensional effect set via the setting screen provided by the UI providing unit on the scan data, and a drawing unit.
An information processing device including a display unit for displaying a three-dimensional effect drawn in the drawing unit. The acquisition unit acquires the scan data as mesh data and obtains the scan data.
The information processing apparatus according to claim 1, wherein the drawing unit draws the three-dimensional effect on the mesh data acquired by the acquisition unit. The drawing unit determines whether or not the scan data acquired by the acquisition unit has been updated, and when the scan data is updated, the drawing unit updates and draws the three-dimensional effect. The information processing device described in. The information according to any one of claims 1 to 3, wherein the UI providing unit provides a setting screen for setting a drawing order of the three-dimensional effect in the display range displayed on the display unit. Processing equipment. The information processing apparatus according to any one of claims 1 to 4, wherein the drawing unit draws the three-dimensional effect by executing rendering of the scan data using a Bezier curve. It is an information processing method using an information processing device.
A UI providing step that provides a setting screen for setting 3D effects,
The acquisition step to acquire the scan data from the sensor that scans the space and acquires the terrain,
A drawing step for drawing a three-dimensional effect set via the setting screen provided in the UI providing step on the scan data, and a drawing step.
An information processing method including a display step for displaying a three-dimensional effect drawn in the drawing step. On the computer
UI provision processing that provides a setting screen for setting 3D effects, and
The acquisition process to acquire the scan data from the sensor that scans the space and acquires the terrain,
A drawing process for drawing a three-dimensional effect set via the setting screen provided in the UI providing process on the scan data, and a drawing process.
An information processing program for executing a display process for displaying a three-dimensional effect drawn in the drawing process.

Images