JP2016071682A - Object arrangement device, object arrangement method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a high quality composite image that suppresses costs by use of a two-dimensional image while allowing a user to confirm the compatibility of a design or a color of an object with a background including furniture and interior accessories which the user owns at the time of renewing furniture or interior accessories, to help the user make a decision.SOLUTION: An object arrangement device 1 stores a plurality of objects and extracts the image data of a target object from an original image(S101), receives a selection of a specific object (102), determines an arrangement position of the object (S105), reverses the binary image of the object, acquires the logical product of the reversed image and the binary image of the target object, calculates a restoration part (S106), complements the pixel data belonging to the restoration part of the original image with pixel data in the vicinity of the restoration part in the original image, generates a restoration image (S107), and combines the image data of the object with the restoration image (S108).SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technique for synthesizing an image such as furniture or interior with a background image such as a room, and more particularly to a technique for synthesizing an image such as furniture or interior included in a background image.

  When a consumer purchases or replaces furniture or interior (hereinafter also referred to as an object), whether it fits in the expected location (confirmation of the space size of the room and the size of the object), and how comfortable it is (chair In addition to the comfort and comfort of the sofa, brand and price, the design and color of the object and the affinity of the furniture and interior that you already have are important points to judge.

  About the affinity of design and color with what you already have, users can see and replace their homes and rooms by looking at the actual showroom, catalogs, brochures, and website images. Think about what happens when you place it in your head. However, when you actually purchase and place it, you may find it dissatisfied with the image you had before purchase, or the affinity with what you already have.

  In order to solve such a problem, there is an object arrangement simulation system. Object placement simulation systems include 3D (three-dimensional) simulation and 2D (two-dimensional) simulation.

  Since 3D simulation is based on CG (Computer Graphics), information such as 3D information and lighting can be used, so that a natural composite image can be created.

  For example, Patent Document 1 discloses an image obtained by combining an image of an interior owned by a user and an image of an interior (object) in a store using an existing CAD (Computer Aided Design) model and 3D CAD software. The invention to be created is disclosed.

  Further, in Patent Document 2, a virtual space image (object) and a real space image are superimposed using depth information recognized from an image captured by a camera in consideration of hiding by an object in the real space. An invention of an image generation apparatus that generates an image by superimposing images to be displayed without considering hiding in a spatial image is disclosed.

  On the other hand, in 2D simulation, it is not necessary to produce content such as CG, and only shooting is performed, so there is no cost for content creation / update. Also, since it is a live-action image, the quality of the content is high.

  For example, in Patent Document 3, a vanishing point where parallel lines converge from a photographed image is obtained, an imaging position is estimated, and three-dimensional spatial information is calculated in a pseudo manner so that natural synthesis is performed as in 3D simulation. An invention of an image composition device that enables creation of an image is disclosed.

  Further, in Patent Document 4, a horizontal image and a vertical region of an image of a placement object are deformed based on a shooting angle and a line-of-sight angle, and synthesized with a venue object image, a synthesized image that looks natural without a sense of incongruity. The invention to be created is disclosed.

JP2005-202797 Japanese Patent No. 4401728 Japanese Patent No. 3693735 Patent No. 4984257

  However, in the inventions of Patent Document 1 and Patent Document 2, 3D CG content must be created with high definition in order to achieve image quality. Therefore, when there is no existing content that can be diverted or a large amount of content is newly created. When trying to add, there is a problem that the production cost of the content becomes enormous.

  The inventions of Patent Literature 3 and Patent Literature 4 originally assume an empty scene in which no object exists in the background, but there is a problem that this is not considered when there is an object in the background. And when a user takes a picture of a place that is supposed to be placed, or moves to a new house and takes a picture, an empty scene can be taken, but when replacing furniture or interior, the one before replacement is placed. In most cases, it is very difficult to shoot an empty scene at the time of replacement.

  In the inventions of Patent Document 3 and Patent Document 4, it is assumed that the background and the shooting environment of the object are the same or are in accordance with a certain standard. For this reason, when a background and an object shot with different shooting environments or imaging devices are combined, an unnatural image is created in which the object appears to float well without matching the background. is there.

  The present invention has been made to solve such a problem. The design and color of an object and the furniture already possessed even at the time of replacement of furniture and interior while suppressing cost by using 2D images. It is an object of the present invention to provide a technology capable of confirming the affinity with the background including the interior and the interior, and obtaining a high-quality composite image that can assist the user's decision making.

  In order to solve the above-described problem, the first invention is stored by a storage unit that stores a plurality of objects, an extraction unit that extracts an image of the object from an original image including the object, and the storage unit. A receiving unit that receives selection of a specific object from a plurality of objects, and an object that is received by the receiving unit is arranged at the position of the target in the original image, and the image of the object is within the range of the target image. A calculation unit that calculates a non-overlapping portion as a restoration unit, a generation unit that complements pixel data located in the restoration unit of the original image with pixel data in the vicinity of the original image, and generates a restoration image; and the restoration An object placement apparatus comprising: a combining unit that combines the image of the object with the image. According to the first invention, the object image can be replaced with the image of the object included in the original image without a sense of incongruity, and the object image can be synthesized with the original image. Accordingly, a general consumer can take a picture of his / her room in which furniture or interior to be replaced is installed, and obtain a high-quality composite image in which new furniture or interior is arranged.

  The first invention further comprises resizing means for resizing the object received by the receiving means according to the size of the object, wherein the calculating means displays the object resized by the resizing means as the original image. It is desirable to calculate the restoration part by arranging the object at the position of the object. As a result, the general consumer can accurately confirm the size of the space when new furniture or interior is arranged.

  The resizing means in the first invention preferably resizes the object based on a circumscribed rectangle of the object and a circumscribed rectangle of the object. Thus, it is not always necessary to shoot from the front direction of the object, and it is convenient because it can shoot from a favorite direction.

  In the first aspect of the invention, it is preferable that the calculation unit calculates the restoration unit by obtaining a logical product of an inverted image of the mask image of the object and a mask image of the object. As a result, it is possible to easily calculate a region to be repaired along with image synthesis.

  In the first invention, the generation means obtains the closest patch by template matching using a peripheral pixel group of the target pixel as a template for the specific target pixel belonging to the restoration unit of the original image. It is desirable to complement the pixel data of the pixel of interest. As a result, high-speed processing can be realized.

  The synthesizing means in the first invention preferably transforms the image of the object by Poisson blending based on the image of the original image and synthesizes it with the repaired image. A natural composite image can be obtained by matching an object (furniture or interior) to the color of the original image.

  According to a second aspect of the present invention, there is provided an extraction step in which a control unit of a computer including a storage unit that stores a plurality of objects extracts an image of the target object from an original image including the target object, and a plurality of storage units stored by the storage unit An accepting step for accepting selection of a specific object from the objects, and an object accepted by the accepting step is arranged at the position of the object in the original image, and the image of the object does not overlap within the range of the image of the object A calculation step of calculating a portion as a restoration unit, a generation step of generating a restoration image by complementing pixel data located in the restoration unit of the original image with pixel data in the vicinity of the original image, and And a synthesis step of synthesizing the image of the object. It is a 置方 method. According to the second invention, the same effect as that of the first invention can be obtained.

  According to a third aspect of the invention, a computer is specified from storage means for storing a plurality of objects, extraction means for extracting an image of the object from an original image including the object, and a plurality of objects stored by the storage means A receiving unit that accepts selection of the object, and an object that is received by the receiving unit is arranged at the position of the object in the original image, and a portion in which the image of the object does not overlap within the range of the image of the object is repaired A calculation means for calculating as a part, pixel data located in the restoration part of the original image is supplemented with pixel data in the vicinity of the original image, and a generation means for generating a repair image; It is a program for functioning as a combining means for combining the images of the object. By installing the program of the third invention on a general-purpose computer, the object placement device of the first invention can be obtained, and the object placement method of the second invention can be executed.

  According to the present invention, it is possible to check the affinity between the design and color of an object and the background including the furniture and interior already possessed even when replacing furniture or interior while reducing the cost by using 2D images. Therefore, it is possible to obtain a high-quality composite image that can assist the user in making decisions.

Block diagram showing the hardware configuration of the object placement device The figure which shows the example of the content memorize | stored in the memory | storage part of an object arrangement | positioning apparatus The flowchart which shows the flow of the object arrangement | positioning process by an object arrangement | positioning apparatus. The figure which shows the example of the information input into an object arrangement | positioning apparatus The figure which shows the example of the binarization process of an object arrangement | positioning apparatus The figure which shows the example of the rectangle calculation process of an object arrangement | positioning apparatus The figure which shows the example of the resizing process of an object arrangement | positioning apparatus The figure which shows the example of the layout process of an object arrangement | positioning apparatus The figure which shows the example of the restoration | repair part calculation process of an object arrangement | positioning apparatus The figure which shows the example of the image restoration process of an object arrangement | positioning apparatus The figure which shows the example of the image compositing process of an object arrangement | positioning apparatus The figure explaining the image compositing process of an object arrangement | positioning apparatus

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a hardware configuration of the object placement apparatus 1. Note that the hardware configuration in FIG. 1 is an example, and various configurations can be adopted depending on the application and purpose.

  As shown in FIG. 1, the object placement apparatus 1 includes a control unit 11, a storage unit 12, a media input / output unit 13, a communication control unit 14, an input unit 15, a display unit 16, a peripheral device I / F unit 17, and the like. Connection is made via a bus 18.

The control unit 11 includes a CPU (Central Processing Unit) and a ROM (Read Only).
Memory), RAM (Random Access Memory), etc. The CPU calls a program stored in the storage unit 12, ROM, recording medium, etc. to the work memory area on the RAM and executes it, drives and controls each device connected via the bus 18, and the object placement device 1 The process to be described later is realized. The ROM is a non-volatile memory and permanently holds a computer boot program, a program such as BIOS, data, and the like. The RAM is a volatile memory, and temporarily stores programs, data, and the like loaded from the storage unit 12, ROM, recording medium, and the like, and includes a work area used by the control unit 11 for performing various processes.

  The storage unit 12 is an HDD (Hard Disk Drive) or the like, and stores a program executed by the control unit 11, data necessary for program execution, an OS (Operating System), and the like. As for the program, a control program corresponding to the OS and an application program for causing a computer to execute processing to be described later are stored. Each of these program codes is read by the control unit 11 as necessary, transferred to the RAM, read by the CPU, and executed as various means.

  The media input / output unit 13 (drive device) inputs / outputs data, for example, media such as a CD drive (-ROM, -R, -RW, etc.), DVD drive (-ROM, -R, -RW, etc.) Has input / output devices. The communication control unit 14 includes a communication control device, a communication port, and the like, and is a communication interface that mediates communication between a computer and a network, and performs communication control between other computers via the network. The network may be wired or wireless.

  The input unit 15 inputs data and includes, for example, a keyboard, a pointing device such as a mouse, and an input device such as a numeric keypad. An operation instruction, an operation instruction, data input, and the like can be performed on the computer via the input unit 15. The display unit 16 includes a display device such as a liquid crystal panel, and a logic circuit or the like (video adapter or the like) for realizing a video function of the computer in cooperation with the display device. The input unit 15 and the display unit 16 may be integrated like a touch panel display.

  The peripheral device I / F (Interface) unit 17 is a port for connecting a peripheral device to the computer, and the computer transmits and receives data to and from the peripheral device via the peripheral device I / F unit 17. The peripheral device I / F unit 17 is configured by USB (Universal Serial Bus), IEEE 1394, RS-232C, or the like, and usually includes a plurality of peripheral devices I / F. The connection form with the peripheral device may be wired or wireless. The bus 18 is a path that mediates transmission / reception of control signals, data signals, and the like between the devices.

  The object placement apparatus 1 may be configured by a single computer, or a plurality of computers may be configured via a network. For example, when the object placement device 1 is configured by a server and a client terminal, the client terminal may receive data input, the server may perform various calculation processes, and the client terminal may display the calculation result. . In the following, in order to avoid confusion, the object placement apparatus 1 will be described as being composed of one computer.

  FIG. 2 is a diagram illustrating an example of contents stored in the storage unit 12 of the object placement device 1. Information relating to a plurality of objects is stored in the storage unit 12 of the object placement device 1. The object is, for example, furniture such as a chair or interior. The information on the object is, for example, an object image, a mask image, and size information. Information about the object may be acquired from furniture or interior catalog data, or may be registered by the user via the input unit 15 or the like.

  For example, FIG. 2 illustrates an object image 21 (an image 23 including an object), a mask image 25, and “width: 550 mm, height: 1000 mm” as information related to “object 1”.

  The object image 21 is, for example, a two-dimensional image obtained by photographing the object from the front. The mask image 25 is, for example, a binary image in which the pixel belonging to the object image 21 is “white” and the other pixels are “black” in the image 23 including the object. In the mask image 25 shown in FIG. 2, the pixel region that does not belong to the object image 21 is illustrated as “shaded” instead of “black”. Hereinafter, in other drawings, “black” is used instead of “black”. In the mask image 25, the pixels belonging to the image 21 of the object may be “black” and the other pixels may be “white”.

  The size information indicates the actual size of the object related to the object image 21. That is, in the example of “object 1” shown in FIG. 2, it means that the actual width of the chair (object) is “550 mm” and the actual height is “1000 mm”.

  FIG. 3 is a flowchart showing the flow of object placement processing by the object placement device 1. As illustrated in FIG. 3, the control unit 11 of the object placement device 1 receives an input of an original image from a user and extracts image data of an object from the original image (step S <b> 101).

  The original image is, for example, a two-dimensional image of a room in which the user assumes an object arrangement, and is taken in advance by the user. In the embodiment of the present invention, an object arrangement simulation when a user replaces an object (furniture or interior) is assumed, and an object to be replaced is included in the original image. Therefore, in the embodiment of the present invention, the target means an object to be replaced that is included in the original image.

  FIG. 4 is a diagram illustrating an example of information input to the object placement apparatus 1. FIG. 4A shows an example of the original image 31 input by the user. The original image 31 includes a chair for replacement by purchase as the object 33.

  For example, the input unit 15 and the display unit 16 of the object placement device 1 are touch panel displays. As shown in FIG. 4B, the user roughly traces the outline of the object 33 with his / her finger or touch pen, and draws an outline 35 surrounding the entire object 33. When receiving the input of the contour line 35, the control unit 11 of the object placement device 1 performs binary processing to set the area of the object 33 to “white” and the other areas to “black” by the process shown in FIG. Create an image. This binary image is a mask image of the object 33 included in the original image 31. When the control unit 11 generates a mask image of the target object 33, the occlusion within the outline 35 (closed region) (region where the object in the foreground hides the object behind), shadow, reflection, etc. Fill all white areas with “white”. Since the area filled with “white” is an area where a restoration process is performed in a process described later or an object image is synthesized, there is no problem even if an ambiguous area is filled with “white”.

  When the control unit 11 receives an input of the contour line 35, the control unit 11 also receives an input of size information regarding the object 33. The size information regarding the object 33 is, for example, the actual size width and height of the object 33 viewed from the photographing direction of the original image 31. In FIG. 4C, “width: 800 mm, height: 700 mm” is illustrated as size information regarding the object 33.

  FIG. 5 is a diagram illustrating an example of the binarization process of the object placement apparatus 1. The control unit 11 generates a binary image 41 in which the outline 35 input by the user is “white” and the others are “black”.

  Next, the control unit 11 performs an edge extraction process and a labeling process on the binary image 41 illustrated in FIG. In the labeling process, the control unit 11 generates a three-color label image in which the background area is “black”, the foreground area is “white”, and the indeterminate area (outline 35) is “gray”. This label image becomes a trimap image 43 in the processing described later. FIG. 5B shows a trimap image 43 including a contour line (gray) 37.

  Next, the control unit 11 determines the indeterminate region to be “black” or “white”. For example, there is a method of formulation as a minimum cost cutting problem (Graph Cut). According to this method, the control unit 11 is formulated as a least cost cutting problem using similarity between a plurality of adjacent foreground and background pixels, with a pixel in the gray region as a base point, and the cost is the highest. A cutting trajectory that becomes smaller is obtained, and the indeterminate area is determined as “black” or “white” based on the cutting trajectory. The present invention is not limited to this method, and other known techniques may be used. FIG. 5C shows a mask image 45 of the object 33 included in the original image 31.

  Returning to the description of FIG. Next, the control part 11 receives selection of a specific object from a user, and confirms whether the object was selected (step S102). For example, the control unit 11 reads a plurality of objects from the storage unit 12 and displays them on the display unit 16. On the other hand, the user selects one of the objects displayed on the display unit 16.

  If not selected (No in step S102), the control unit 11 continues to accept selection of an object from the user. When selected (Yes in Step S102), the control unit 11 calculates a circumscribed rectangle between the object and the object (Step S103).

  FIG. 6 is a diagram illustrating an example of a rectangle calculation process of the object placement device 1. FIG. 6A shows a process for calculating the circumscribed rectangle 51 from the mask image 45 of the object 33. FIG. 6B shows processing for calculating the circumscribed rectangle 53 from the mask image 25 of the object.

  The control unit 11 scans the mask image 45 and the mask image 25 from four directions, up, down, left, and right, and calculates a circumscribed rectangle 51 and a circumscribed rectangle 53 by first obtaining a position where a “white” pixel is present. . That is, the circumscribed rectangle 51 and the circumscribed rectangle 53 are (1) scanned from the left, and first, a vertical straight line passing through the position where the “white” pixel is located is scanned from the left side and (2) from the right, First, the vertical line passing through the position where the “white” pixel is located is scanned from the right side, (3), and the straight line passing through the position where the “white” pixel is located is the upper side, (4 ) A rectangle that scans from the bottom and first has a “white” pixel passing through a certain position in the horizontal direction as the lower side.

  Returning to the description of FIG. Next, the control unit 11 resizes the object according to the size of the object based on the circumscribed rectangle calculated in step S103 (step S104).

  FIG. 7 is a diagram illustrating an example of the resizing process of the object placement device 1. For the mask image 25 and circumscribed rectangle 53 of the object shown in FIG. 7A, the control unit 11 resized the circumscribed rectangle 55 shown in FIG. 7B and the resized rectangle shown in FIG. 7C. An object image 21a and a resized mask image 25a are generated.

First, the control unit 11 determines a resizing rate used for resizing an object. W_bg_ex is the actual width (mm) of the object included in the original image, W_bg_px is the pixel width (pixel) of the object included in the original image, W_ob_ex is the actual width (mm) of the object,
If the pixel width (pixel) of the image of the object is W_ob_px, the width resizing rate W_R is obtained by W_R = (W_bg_ex × W_bg_px) / (W_ob_ex × W_ob_px). The height resize rate H_R is also obtained in the same manner.

  Next, the control unit 11 resizes the circumscribed rectangle 53 based on the resizing rate, and calculates the resized circumscribed rectangle 55. Further, the control unit 11 resizes the object image 21 and the mask image 25 based on the resizing rate, and generates the resized object image 21a and the resized mask image 25a.

  By synthesizing the resized object image 21a according to the size of the object with the original image, the general consumer can accurately confirm the size of the room when new furniture or interior is arranged. .

  Further, by resizing the object based on the circumscribed rectangle, the resizing process can be performed regardless of the orientation of the object in the original image and the orientation of the object in the object image 21 and the mask image 25. Therefore, it is not always necessary to prepare an original image taken from the front direction of the object, and the object image 21 and the mask image 25 do not have to be images from the front direction of the object. Furthermore, the orientation of the object in the original image and the orientation of the object in the object image 21 and the mask image 25 do not need to match.

  The object placement apparatus 1 stores a large number of images and circumscribed rectangle data of different sizes in the storage unit 12 for each object instead of the process of step S104, and the control unit 11 searches the storage unit 12. Then, data closest to the size of the circumscribed rectangle of the target object included in the original image may be acquired, and the resized circumscribed rectangle 55, the resized object image 21a, and the resized mask image 25a may be used.

  Returning to the description of FIG. Next, the control unit 11 determines the arrangement position of the object resized in step S104 (step S105).

  FIG. 8 is a diagram illustrating an example of layout processing of the object placement device 1. The control unit 11 performs alignment in order to place the object at the position of the object in the original image. The alignment in the left-right direction is, for example, three ways: (1) align to the left, (2) align to the center, and (3) align to the right. Which position is to be aligned may be stored in advance in the storage unit 12 as setting information, or a selection may be received from the user.

  The control unit 11 aligns the circumscribed rectangle 55 of the resized mask image 25a shown in FIG. (1) When adjusting to the left, as shown in FIG. 8B, the control unit 11 moves the object to a position where the left side of the resized circumscribed rectangle 55 of the object and the left side of the circumscribed rectangle 51 of the object overlap. Deploy. (2) In the case of matching with the center, as shown in FIG. 8C, the control unit 11 causes the bisector extending in the vertical direction of the resized circumscribed rectangle 55 of the object and the upper and lower sides of the circumscribed rectangle 51 of the target object. The object is arranged at a position where the bisector extending in the direction overlaps. (3) When aligning to the right, as shown in FIG. 8D, the control unit 11 moves the object to a position where the right side of the resized circumscribed rectangle 55 of the object and the right side of the circumscribed rectangle 51 of the object overlap. Deploy.

  In the example shown in FIG. 8, the vertical alignment is performed downward, but is not limited to this. Similar to the horizontal alignment, the vertical alignment can be performed in three ways: (1) align to the top, (2) align to the center, and (3) align to the bottom. In this case, the processing content by the control unit 11 is the same as that in the horizontal alignment.

  Returning to the description of FIG. Next, the control unit 11 inverts the binary image of the object, obtains a logical product of the inverted image and the binary image of the object, and calculates a restoration unit (step S106). That is, the control unit 11 calculates, as a restoration unit, a portion where the object image does not overlap within the range of the target image. In the present embodiment, the restoration unit excludes the object (furniture to be replaced) in the original image (room where the furniture is arranged) and places an object (furniture desired to be purchased) as blank. Means an area.

  FIG. 9 is a diagram illustrating an example of the restoration unit calculation process of the object arrangement device 1. The control unit 11 inverts the mask image 25a (binary image) of the object shown in FIG. 9A to obtain an inverted image 61 of the mask image 25a shown in FIG. 9B. Next, as shown in FIG. 9C, the control unit 11 arranges the circumscribed rectangle 55 of the object on the mask image 45 of the object based on the arrangement position of the object determined in step S105. Then, as shown in FIG. 9D, the control unit 11 generates a logical product image 63 of the reverse image 61 and the mask image 45 (binary image) of the object, and restores the “white” portion to the restoration unit. 65. In this way, the restoration unit 65 can be easily calculated by performing the processing with the logical operation.

  Returning to the description of FIG. Next, the control unit 11 complements pixel data belonging to the restoration unit of the original image with pixel data in the vicinity of the original image, and generates a restoration image (step S107). For example, the control unit 11 uses the peripheral pixel group of the target pixel as a template for a specific target pixel located in the restoration unit of the original image, and obtains the closest patch by template matching, thereby obtaining pixel data of the target pixel. To complement. Note that the image restoration processing method is not limited to template matching, and other known techniques may be used.

  FIG. 10 is a diagram illustrating an example of image restoration processing of the object placement apparatus 1. As shown in FIG. 10A, the control unit 11 sequentially selects the target pixel 67 from the pixels belonging to the restoration unit 65, and determines a template 69 including a peripheral pixel group of the target pixel. For example, as illustrated in FIG. 10B, the template 69 is a 5 × 5 grid that includes two pixels in the vertical and horizontal directions around the target pixel 67. Here, the pixel value of each pixel included in the template 69 is the pixel value of the original image, excluding the pixels belonging to the restoration unit 65. Further, the pixel value of each pixel included in the patch is the pixel value of the original image. Since the pixel values of the pixels belonging to the restoration unit 65 are NULL (no value) until they are complemented, if the template includes pixels of the restoration unit that have not yet been complemented, Remove from the sum of squares. On the other hand, the pixel of the repaired portion that is complemented may be a calculation target of the sum of the squares of the differences.

  Next, the control unit 11 performs template matching on the original image using the template 69. As illustrated in FIG. 10C, the control unit 11 obtains the closest patch 70 and determines the pixel value of the pixel of interest 67 based on the patch 70. The control unit 11 executes this process for all the pixels belonging to the restoration unit 65, and generates a restoration image 71 shown in FIG. As a result, high-speed processing can be realized. Note that the target object (remaining image) 33 that does not belong to the repair unit 65 remains in the repair image 71. However, there is no problem because the image data of the object is superimposed on this area by the synthesis process described later. .

Criterion of the similarity between the template 69 and the patch 70 may, for example, a total S _SSD of square of the difference between the pixel values shown in the following equation. Here, g (x, y): original image, f (x, y): template 69, g (d _x , _dy ): patch 70, i: x direction counter, j: y direction counter.

  Returning to the description of FIG. Next, the control unit 11 combines the image data of the object with the repaired image (step S108).

  FIG. 11 is a diagram illustrating an example of image composition processing of the object placement device 1. The control unit 11 synthesizes the resized object image 21a shown in FIG. 11B in the circumscribed rectangle 55 of the repair image 71 shown in FIG. 11A, and forms a composite image 73 shown in FIG. Is generated.

  For example, the control unit 11 converts the image data of the object by Poisson blending (also referred to as “Poisson Image Editing”) based on the image data of the original image, and synthesizes it with the repaired image. Note that the method of image composition processing is not limited to Poisson blending, and other known techniques may be used.

  FIG. 12 is a diagram for explaining image composition processing of the object placement apparatus 1. Poisson blending is a technique in which the color is made dependent on the image of the pasting destination (restored image), and the gradient of the image is synthesized using the image of the pasting source (image of the object). In Poisson blending, the image data of the pasting source is treated as a collection of differential values, the differential values are stored as much as possible, the values of the pasting boundaries are matched, and natural image synthesis is performed. What realizes this is a simultaneous differential equation (Poisson equation), which can be solved by the Gauss-Seidel method.

  FIG. 12 shows an image of processing by Poisson blending. 12A shows an image before processing by Poisson blending, and FIG. 12B shows an image after processing by Poisson blending.

  In FIG. 12A, the paste-source image data 81 is arranged in the paste-destination image data 83 as it is. Accordingly, there is a large difference between the values of the image data at both ends of the pasting range 85, that is, at the pasting boundary 87. If there is such a difference, in the actual image, the image data 81 of the pasting source will feel floating, and a sense of incongruity will occur.

  On the other hand, in FIG. 12B, the paste-source image data 81 is converted by Poisson blending so that the color depends on the paste-destination image data 83. Accordingly, at the pasting boundary 87, there is no significant difference between the values of the pasted source image data 89 and the pasted destination image data 83. As a result, the actual image does not feel uncomfortable.

  In this way, by changing the color of the image of the object (furniture desired to be replaced) and bringing it closer to the color of the original image, the user can change the wallpaper of the room, floor, etc. The color of furniture desired to be replaced can be selected by comprehensively considering the color of the furniture.

  As described above, the object placement apparatus 1 executes the object placement processing according to the flowchart shown in FIG. The object placement device 1 according to the embodiment of the present invention includes the design and color of an object and the furniture and interior that the user already has, even at the time of replacement of furniture and interior, while reducing costs by using 2D images. Therefore, it is possible to confirm the affinity with the background and to obtain a high-quality composite image that can assist the user in making decisions.

  The preferred embodiments of the object placement device and the like according to the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications can be conceived within the scope of the technical idea disclosed in the present application, and these naturally belong to the technical scope of the present invention. Understood.

DESCRIPTION OF SYMBOLS 1 ......... Object arrangement | positioning apparatus 21 ......... Object image 21a ......... Resized object image 23 ......... Image including object 25 ......... Mask image 25a ......... Resized mask image 31 ... ... original image 33 ...... object 35 ......... contour line 37 ......... contour line (gray)
41... Binary image 43... Trimap image 45... Mask image 51... Circumscribed rectangle 53... Circumscribed rectangle 55... Resized circumscribed rectangle 61. Image 63... AND image of inverted image 61 and mask image 45 65... Restoration unit 67... Pixel of interest 69... Template 70... Patch 71.

Claims (8)

Storage means for storing a plurality of objects;
Extracting means for extracting an image of the object from an original image including the object;
Receiving means for receiving selection of a specific object from a plurality of objects stored by the storage means;
A calculating unit that arranges an object received by the receiving unit at the position of the target in the original image, and calculates a portion in which the image of the object does not overlap within the range of the image of the target as a restoration unit;
Generating means for complementing pixel data located in the restoration unit of the original image with pixel data in the vicinity of the original image and generating a restored image;
A combining means for combining the image of the object with the repaired image;
An object placement apparatus comprising: Resizing means for resizing the object received by the receiving means according to the size of the object;
The object placement apparatus according to claim 1, wherein the calculation unit places the object resized by the resizing unit at a position of the target in the original image, and calculates the restoration unit.   The object resizing device according to claim 2, wherein the resizing unit resizes the object based on a circumscribed rectangle of the object and a circumscribed rectangle of the object.   4. The calculation unit according to claim 1, wherein the calculation unit calculates the restoration unit by calculating a logical product of a reverse image of the mask image of the object and a mask image of the object. The object arrangement device according to item 1.   The generating means obtains the closest patch by template matching for a specific pixel of interest belonging to the restoration unit of the original image and using a peripheral pixel group of the pixel of interest as a template, thereby obtaining pixel data of the pixel of interest The object placement device according to any one of claims 1 to 4, wherein the object placement device is supplemented.   The said synthesis | combination means converts the image of the said object by Poisson blending based on the image of the said original image, and synthesize | combines it with the said restored image. Object placement device. A control unit of a computer including a storage unit that stores a plurality of objects,
An extraction step of extracting an image of the object from an original image including the object;
An accepting step of accepting selection of a specific object from a plurality of objects stored by the storage unit;
A calculation step of arranging an object received by the reception step at a position of the target in the original image, and calculating a portion in which the image of the object does not overlap within a range of the target image as a restoration unit;
A generation step of complementing pixel data located in the restoration unit of the original image with pixel data in the vicinity of the original image, and generating a restored image;
A synthesis step of synthesizing the image of the object with the repaired image;
An object placement method characterized by executing Computer
Storage means for storing a plurality of objects;
Extracting means for extracting an image of the object from an original image including the object;
Receiving means for receiving selection of a specific object from a plurality of objects stored by the storage means;
A calculating unit that arranges an object received by the receiving unit at the position of the target in the original image, and calculates a portion in which the image of the object does not overlap within the range of the image of the target as a restoration unit;
Generating means for complementing pixel data located in the restoration unit of the original image with pixel data in the vicinity of the original image and generating a restored image;
A program for causing the repaired image to function as a combining unit that combines the image of the object.

Images