[go: up one dir, main page]

WO2021261619A1 - Dispositif électronique de détection d'un plan dans une image et procédé de fonctionnement correspondant - Google Patents

Dispositif électronique de détection d'un plan dans une image et procédé de fonctionnement correspondant Download PDF

Info

Publication number
WO2021261619A1
WO2021261619A1 PCT/KR2020/008264 KR2020008264W WO2021261619A1 WO 2021261619 A1 WO2021261619 A1 WO 2021261619A1 KR 2020008264 W KR2020008264 W KR 2020008264W WO 2021261619 A1 WO2021261619 A1 WO 2021261619A1
Authority
WO
WIPO (PCT)
Prior art keywords
plane
image
electronic device
detecting
dimensions
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/KR2020/008264
Other languages
English (en)
Korean (ko)
Inventor
어스미카일로
부트안드리
호로크올레나
자리치니이고르
코디치세르기
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of WO2021261619A1 publication Critical patent/WO2021261619A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/30—Polynomial surface description
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/10—Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2215/00—Indexing scheme for image rendering
    • G06T2215/06—Curved planar reformation of 3D line structures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/008—Cut plane or projection plane definition

Definitions

  • the present disclosure relates to an electronic device for detecting a plane included in an image, and an operating method thereof.
  • an image in which a scene of the real world and an augmented reality image are mixed may be provided to a user in real time. Accordingly, in services such as advertisements, navigation, games, and the like, various contents may be provided to users through augmented reality technology.
  • AR augmented reality
  • the augmented reality image may be arranged based on a plane detected in a scene in the real world so that a user may hardly feel a sense of heterogeneity.
  • the augmented reality image may be disposed at an incorrect position, and thus a user experience according to the augmented reality technology may be deteriorated.
  • An object of the present disclosure is to solve the above-described problem, and to provide an electronic device for detecting a plane in an image and an operating method thereof.
  • Another object of the present invention is to provide a computer-readable recording medium in which a program for executing the method in a computer is recorded.
  • the technical problem to be solved is not limited to the technical problems as described above, and other technical problems may exist.
  • FIG. 1 is a block diagram illustrating an example of detecting a plane in an image according to an embodiment.
  • FIG. 2 is a diagram illustrating an example of detecting a plane according to an embodiment.
  • FIG. 3 is a block diagram illustrating an internal configuration of an electronic device according to an exemplary embodiment.
  • FIG. 4 is a block diagram illustrating an internal configuration of an electronic device according to an exemplary embodiment.
  • FIG. 5 is a flowchart illustrating a method of detecting a plane in an image according to an exemplary embodiment.
  • FIG. 6 is a block diagram illustrating an example of detecting a plane in an image according to an embodiment.
  • FIG. 7 is a diagram illustrating an example of identifying a surface in contact with a plane among surfaces of an object according to an embodiment.
  • FIG. 8 is a diagram illustrating an example of identifying a plane based on an object according to an embodiment.
  • FIG. 9 is a diagram illustrating an example of modeling an object in three dimensions according to a direction of gravity according to an embodiment.
  • FIG. 10 is a diagram illustrating an example in which a plane is detected based on an image according to an embodiment.
  • a first aspect of the present disclosure is a method for detecting a plane in an image in an electronic device, the method comprising: acquiring an image including a plane; detecting an object in the image; modeling the shape of the object in three dimensions; identifying a surface of the object in contact with the plane based on the modeled result; based on the identified surface, detecting the plane in the image.
  • a second aspect of the present disclosure is an electronic device for detecting a plane in an image, acquiring an image including a plane, detecting an object in the image, modeling the shape of the object in three dimensions, at least one processor configured to identify a surface in contact with the plane among the surfaces of the object based on the modeled result, and detect the plane from the image based on the identified surface; and a display for displaying information related to a result of detecting the plane.
  • a third aspect of the present disclosure may provide a recording medium in which a program for performing the method of the first aspect is stored.
  • the processor may consist of one or a plurality of processors.
  • one or more processors may be a general-purpose processor such as a CPU, an AP, a digital signal processor (DSP), or the like, a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU.
  • DSP digital signal processor
  • One or a plurality of processors control to process input data according to a predefined operation rule or artificial intelligence model stored in the memory.
  • the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model.
  • a predefined action rule or artificial intelligence model is characterized in that it is created through learning.
  • being made through learning means that a basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, so that a predefined action rule or artificial intelligence model set to perform a desired characteristic (or purpose) is created means burden.
  • Such learning may be performed in the device itself on which artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and/or system.
  • Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.
  • the artificial intelligence model may be composed of a plurality of neural network layers.
  • Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between the operation result of a previous layer and the plurality of weights.
  • the plurality of weights of the plurality of neural network layers may be optimized by the learning result of the artificial intelligence model. For example, a plurality of weights may be updated so that a loss value or a cost value obtained from the artificial intelligence model during the learning process is reduced or minimized.
  • the artificial neural network may include a deep neural network (DNN), for example, a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), There may be a Deep Belief Network (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), or a Deep Q-Networks, but is not limited thereto.
  • DNN Deep Neural Network
  • DNN Deep Belief Network
  • BBDNN Bidirectional Recurrent Deep Neural Network
  • Deep Q-Networks Deep Q-Networks
  • FIG. 1 is a block diagram illustrating an example of detecting a plane in an image according to an embodiment.
  • the electronic device 1000 detects 130 a plane based on objects 110 and 120 included in an image 100 , and augments it according to the detected plane.
  • the image 100 may be displayed on the real image 140 .
  • the electronic device 1000 may be a device capable of displaying the image 100 and the augmented reality image 140 , and may be implemented in various forms.
  • the electronic device 1000 described herein may include a digital camera, a smart phone, a laptop computer, a tablet PC, an electronic book terminal, a digital broadcasting terminal, and a personal digital assistant (PDA). , a Portable Multimedia Player (PMP), a navigation system, an MP3 player, a vehicle, and the like, but is not limited thereto.
  • PDA personal digital assistant
  • PMP Portable Multimedia Player
  • the electronic device 1000 described herein may be a wearable device that can be worn by a user.
  • a wearable device is an accessory-type device (e.g., watch, ring, wristband, ankle band, necklace, eyeglasses, contact lens), a head-mounted-device (HMD), a fabric or clothing-integrated device (e.g., electronic clothing), a body attachable device (eg, a skin pad), or a bioimplantable device (eg, an implantable circuit).
  • HMD head-mounted-device
  • a fabric or clothing-integrated device e.g., electronic clothing
  • a body attachable device eg, a skin pad
  • a bioimplantable device eg, an implantable circuit
  • the electronic device 1000 may acquire the image 100 in which the surrounding environment including the objects 110 and 120 is captured by using a camera provided in the electronic device 1000 .
  • the electronic device 1000 may acquire the image 100 for detecting the plane 130 by receiving the image 100 from an external device (not shown).
  • the electronic device 1000 detects at least one object 110 , 120 in the image 100 , and makes contact with the plane 130 among at least one surface of the object 110 , 120 . surface can be identified.
  • the electronic device 1000 identifies the shapes of the objects 110 and 120 and determines the shapes of the identified objects 110 and 120 in order to identify the surface in contact with the plane 130 . It can be modeled as a 3D model.
  • the electronic device 1000 may detect the plane 130 in the image 100 by identifying a surface in contact with the plane 130 based on the modeling result of the objects 110 and 120 . have.
  • the identified surface of the object 110 , 120 may be one of at least one surface of the object 110 , 120 included in the 3D model of the object 110 , 120 . .
  • the electronic device 1000 identifies at least one surface of the objects 110 and 120 in the 3D model of the objects 110 and 120 and detects a plane 130 among the at least one surface.
  • the surface to be used can be identified.
  • the shapes of the objects 110 and 120 are based on the shapes of the objects 110 and 120 displayed in the image 100 , and the shapes of the objects 110 and 120 not displayed in the image 100 are As the shape is predicted, the shape of the objects 110 and 120 may be modeled as a 3D model based on the predicted result.
  • the shape of the object may be predicted based on a result of recognizing the object in the image. For example, when the object is recognized as a book, box, table, etc., the shape of the object may be three-dimensionally modeled in the shape of a cuboid. Also, when the object is recognized as a cup, a water bottle, or the like, the shape of the object may be 3D modeled in a cylindrical shape.
  • the shapes of the objects 110 and 120 may include geometric shapes such as a cuboid, a cylinder, and a cone. According to an embodiment, even if the shapes of the objects 110 and 120 do not have an exact geometric shape, the shapes of the objects 110 and 120 may be modeled in three dimensions according to geometric shapes within a similar range. Without being limited to the above-described example, the shapes of the objects 110 and 120 that can be identified by the electronic device 1000 are based on the shapes of the objects 110 and 120 displayed on the image 100, By predicting the shapes of the objects 110 and 120 that are not indicated in 100), various types of shapes that can be modeled in three dimensions may be included.
  • the electronic device 1000 is not limited to the shape of the objects 110 and 120 displayed on the image 100 , and the objects 110 and 120 are viewed from a different viewpoint from the image 100 .
  • the shape of the objects 110 and 120 may be modeled in three dimensions by further using other images captured by .
  • the electronic device 1000 may identify the plane 130 based on the 3D model in which the shapes of the objects 110 and 120 are modeled. According to an embodiment, in the three-dimensional model of the objects 110 and 120 , the plane 130 may be identified based on a surface in contact with the plane 130 .
  • the electronic device 1000 may identify the bottom surfaces of the objects 110 and 120 , respectively, and identify the plane 130 on which the two bottom surfaces are placed from the image 100 .
  • the The plane 130 can be accurately detected.
  • the electronic device 1000 may display the augmented reality image 140 based on the identified plane 130 .
  • the augmented reality image 140 may be displayed as if it were lying on the plane 130 , so that the user may feel as if the augmented reality image 140 actually exists in the real world.
  • FIG. 2 is a diagram illustrating an example of detecting a plane according to an embodiment.
  • the electronic device 1000 may identify at least one object 211 , 212 , 213 , and 214 included in an image. Also, at 220 , the electronic device 1000 may detect the two planes 222 and 223 by modeling the shapes of the identified objects 211 , 212 , 213 , and 214 in three dimensions.
  • the objects 211 and 214 are recognized as boxes and tables, respectively, and thus may be 3D modeled in a rectangular parallelepiped shape. Also, as the objects 212 and 213 are recognized as mugs, they may be 3D modeled in a cylindrical shape.
  • Object recognition according to an embodiment may be performed by a pre-trained artificial intelligence model (ex. CNN) to recognize an object in an image and perform three-dimensional modeling, but is not limited thereto, and according to various methods, An object may be recognized in the image.
  • a pre-trained artificial intelligence model ex. CNN
  • the electronic device 1000 extracts a feature point for the object from a region including the object in the image, in addition to the object recognition result, and determines the shapes of the objects based on the extracted feature point It can be modeled in 3D.
  • the shapes of the objects may be modeled in three dimensions according to a geometric shape determined for the object based on a feature point located in at least one of a corner and an edge region of the object among the feature points of the object.
  • the feature point according to an embodiment may include various types of feature points indicating the characteristics of an image in a region where an object is displayed.
  • the feature point according to an embodiment may be extracted through a point cloud, but is not limited thereto, and may be extracted according to various methods.
  • the electronic device 1000 may identify at least one object 211 , 212 , 213 , and 214 included in an image and model the shape of each object in three dimensions.
  • the electronic device 1000 may set the shape of each object so as to know the shape of each object viewed from various viewpoints (eg, above, below, and behind) based on the result of modeling the shape of each object.
  • the shape can be modeled in 3D.
  • the shape of the object may be modeled to include a surface in contact with the plane.
  • a table-shaped object 214 may actually have four legs abutting on the plane 223 , but instead of the four legs, the plane 223 on which the table rests can be detected so that it has four legs.
  • the object 214 may be modeled in a shape including a bottom surface, each of which is a vertex.
  • FIG. 3 is a block diagram illustrating an internal configuration of the electronic device 1000 according to an embodiment.
  • FIG. 4 is a block diagram illustrating an internal configuration of the electronic device 1000 according to an embodiment.
  • the electronic device 1000 may include a processor 1300 and a display 1210 . However, not all of the components shown in FIG. 3 are essential components of the electronic device 1000 .
  • the electronic device 1000 may be implemented by more components than those illustrated in FIG. 3 , or the electronic device 1000 may be implemented by fewer components than those illustrated in FIG. 3 .
  • the electronic device 1000 includes a user input unit 1100 and an output unit 1200 in addition to the processor 1300 and the display 1210 . ), a sensing unit 1400 , a communication unit 1500 , an A/V input unit 1600 , and a memory 1700 may be further included.
  • the user input unit 1100 means a means for a user to input data for controlling the electronic device 1000 .
  • the user input unit 1100 includes a key pad, a dome switch, and a touch pad (contact capacitive method, pressure resistance film method, infrared sensing method, surface ultrasonic conduction method, integral type).
  • a tension measurement method a piezo effect method, etc.
  • a jog wheel a jog switch, and the like, but is not limited thereto.
  • the user input unit 1100 may receive a user input for detecting a plane in an image.
  • the user input unit 1100 may receive a user input for displaying an augmented reality image on an image of the real world.
  • an operation for detecting a plane in the image may be performed.
  • the output unit 1200 may output an audio signal, a video signal, or a vibration signal, and the output unit 1200 may include a display unit 1210 , a sound output unit 1220 , and a vibration motor 1230 . have.
  • the display unit 1210 displays and outputs information processed by the electronic device 1000 .
  • the display unit 1210 may display information related to a result of detecting a plane in an image. For example, based on a plane detected in the image, an augmented reality image may be displayed.
  • the display unit 1210 may be used as an input device in addition to an output device.
  • the display unit 1210 includes a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a three-dimensional display ( 3D display) and electrophoretic display (electrophoretic display) may include at least one. Also, depending on the implementation form of the electronic device 1000 , the electronic device 1000 may include two or more display units 1210 .
  • the sound output unit 1220 outputs audio data received from the communication unit 1500 or stored in the memory 1700 .
  • the vibration motor 1230 may output a vibration signal. Also, the vibration motor 1230 may output a vibration signal when a touch is input to the touch screen.
  • the sound output unit 1220 and the vibration motor 1230 may output information related to a result of detecting a plane in the image.
  • information related to the provided augmented reality service may be output by the sound output unit 1220 and the vibration motor 1230 based on the detected plane.
  • the processor 1300 generally controls the overall operation of the electronic device 1000 .
  • the processor 1300 executes programs stored in the memory 1700 , and thus the user input unit 1100 , the output unit 1200 , the sensing unit 1400 , the communication unit 1500 , and the A/V input unit 1600 . ), etc., can be controlled in general.
  • the electronic device 1000 may include at least one processor 1300 .
  • the electronic device 1000 may include various types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU).
  • CPU central processing unit
  • GPU graphics processing unit
  • NPU neural processing unit
  • the processor 1300 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations.
  • the command may be provided to the processor 1300 from the memory 1700 or may be received through the communication unit 1500 and provided to the processor 1300 .
  • the processor 1300 may be configured to execute instructions according to program codes stored in a recording device such as a memory.
  • the processor 1300 may detect an object from an image, model the shape of the detected object in three dimensions, and may identify a surface in contact with a plane among surfaces of the object according to the modeling result. Also, the processor 1300 may detect a plane in the image based on the identified surface of the object. According to an embodiment, based on the detected plane, an augmented reality image may be displayed in the image.
  • the processor 1300 may extract a plurality of feature points from a region in which an object detected from an image is displayed, and identify a feature point located at an edge or a corner of the object among the extracted feature points. Also, the processor 1300 may model the shape of the object in three dimensions based on feature points located at edges or corners of the object. Also, the processor 1300 according to an embodiment may 3D model the shape of the object as a geometric shape according to a result of recognizing the object.
  • the processor 1300 may identify the type of the object detected in the image, and determine the type of the plane on which the object is placed based on the identified type.
  • a plane on which the object is placed may be determined to be a type of a table, a floor, or the like.
  • the plane on which the object is placed may be determined to be a ceiling or wall type.
  • a plane type corresponding to each of at least one object may be determined, 3D modeling may be performed on at least one object having the same plane type, and one plane may be detected. For example, with respect to an image of a scene in which a mug and a book are placed on a table, three-dimensional modeling may be performed, considering that the mug and the book are objects on one plane, respectively, and the three-dimensional modeling result is Based on it, a plane corresponding to the table can be detected.
  • the shape of the object when it is determined that the plane on which the object is placed is a type such as a table or a floor, the shape of the object may be modeled in three dimensions based on gravity information collected by the electronic device 1000 .
  • a direction of gravity with respect to the object included in the image 100 may be predicted based on gravity information collected by the electronic device 1000 , and based on the predicted direction of gravity, the shape of the object is three-dimensional. can be modeled as
  • a plane included in an image may be detected based on a result of modeling the plane in three dimensions.
  • the sensing unit 1400 may detect a state of the electronic device 1000 or a state around the electronic device 1000 , and transmit the sensed information to the processor 1300 .
  • the sensing unit 1400 includes a geomagnetic sensor 1410 , an acceleration sensor 1420 , a temperature/humidity sensor 1430 , an infrared sensor 1440 , a gyroscope sensor 1450 , and a position sensor. (eg, GPS) 1460 , a barometric pressure sensor 1470 , a proximity sensor 1480 , and at least one of an illuminance sensor 1490 , but is not limited thereto.
  • GPS GPS
  • the sensing unit 1400 may further include a gravity sensor for sensing gravity, and based on the direction of gravity sensed by the sensing unit 1400 , an object included in the image 100 .
  • the direction of gravity can be predicted.
  • the shape of the object may be modeled in three dimensions based on the predicted direction of gravity with respect to the object.
  • the communication unit 1500 may include one or more components that allow the electronic device 1000 to communicate with the server 2000 or an external device (not shown).
  • the communication unit 1500 may include a short-distance communication unit 1510 , a mobile communication unit 1520 , and a broadcast receiving unit 1530 .
  • Short-range wireless communication unit 1510 Bluetooth communication unit, BLE (Bluetooth Low Energy) communication unit, short-range wireless communication unit (Near Field Communication unit), WLAN (Wi-Fi) communication unit, Zigbee (Zigbee) communication unit, infrared ( It may include an IrDA, infrared Data Association) communication unit, a Wi-Fi Direct (WFD) communication unit, an ultra wideband (UWB) communication unit, an Ant+ communication unit, and the like, but is not limited thereto.
  • the mobile communication unit 1520 transmits/receives a radio signal to and from at least one of a base station, an external terminal, and a server on a mobile communication network.
  • the wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call signal, or a text/multimedia message.
  • the broadcast receiver 1530 receives a broadcast signal and/or broadcast-related information from the outside through a broadcast channel.
  • the broadcast channel may include a satellite channel and a terrestrial channel.
  • the electronic device 1000 may not include the broadcast receiver 1530 .
  • the communication unit 1500 may receive the image 100 from an external device (not shown). Also, the communication unit 1500 may transmit/receive data required to detect the plane of the image 100 .
  • the A/V (Audio/Video) input unit 1600 is for inputting an audio signal or a video signal, and may include a camera 1610 , a microphone 1620 , and the like.
  • the camera 1610 may obtain an image frame such as a still image or a moving image through an image sensor in a video call mode or a shooting mode.
  • the image captured through the image sensor may be processed through the processor 1300 or a separate image processing unit (not shown).
  • the camera 1610 may acquire an image 100 for detecting a plane by photographing an image 100 including an object and a plane.
  • the microphone 1620 receives an external sound signal and processes it as electrical voice data.
  • the microphone 1620 may receive a user's voice input for detecting the plane of the image 100 .
  • the memory 1700 may store a program for processing and control of the processor 1300 , and may also store data input to or output from the electronic device 1000 .
  • the memory 1700 may store at least one image 100 from which a plane may be detected. Also, the memory 1700 may store various data for detecting a plane in the image 100 .
  • the memory 1700 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory), and a RAM.
  • RAM Random Access Memory
  • SRAM Static Random Access Memory
  • ROM Read-Only Memory
  • EEPROM Electrically Erasable Programmable Read-Only Memory
  • PROM Programmable Read-Only Memory
  • magnetic memory magnetic disk
  • magnetic disk may include at least one type of storage medium among optical disks.
  • Programs stored in the memory 1700 may be classified into a plurality of modules according to their functions, for example, may be classified into a UI module 1710 , a touch screen module 1720 , a notification module 1730 , and the like. .
  • the UI module 1710 may provide a specialized UI, GUI, etc. interworking with the electronic device 1000 for each application.
  • the touch screen module 1720 may detect a touch gesture on the user's touch screen and transmit information about the touch gesture to the processor 1300 .
  • the touch screen module 1720 according to some embodiments may recognize and analyze a touch code.
  • the touch screen module 1720 may be configured as separate hardware including a controller.
  • Various sensors may be provided inside or near the touch screen to detect a touch or a proximity touch of the touch screen.
  • a sensor for detecting a touch of a touch screen there is a tactile sensor.
  • a tactile sensor refers to a sensor that senses a touch of a specific object to the extent or higher than that of a human being.
  • the tactile sensor may sense various information such as the roughness of the contact surface, the hardness of the contact object, and the temperature of the contact point.
  • the user's touch gesture may include tap, touch & hold, double tap, drag, pan, flick, drag and drop, swipe, and the like.
  • the notification module 1730 may generate a signal for notifying the occurrence of an event in the electronic device 1000 .
  • FIG. 5 is a flowchart illustrating a method of detecting a plane in an image according to an exemplary embodiment.
  • the electronic device 1000 may acquire an image for detecting a plane.
  • the electronic device 1000 may display an augmented reality image based on a plane detected from an image in order to provide an augmented reality service to the user.
  • an image of a surrounding environment being photographed in real time may be acquired by the electronic device 1000 as an image for detecting a plane. Accordingly, the electronic device 1000 may provide the user with an image in which an augmented reality image is combined with an image of the surrounding environment.
  • the electronic device 1000 may detect an object serving as a reference for detecting a plane in the image.
  • an object for detecting a plane may be detected from among objects lying on or attached to a plane included in an image.
  • the electronic device 1000 is configured to recognize various objects and planes included in an image using a pre-trained artificial intelligence model (ex. CNN), and detect the plane according to the recognized result.
  • An object may be identified.
  • the electronic device 1000 may detect an object for detecting a plane from an image according to various methods.
  • the electronic device 1000 may model the shape of the object detected in operation 520 in three dimensions.
  • the electronic device 1000 may model the shape of the object in three dimensions based on at least one of an edge and a corner of the object detected in the image.
  • the electronic device 1000 identifies at least one feature point in a region where an object is detected in an image, and identifies feature points corresponding to edges and corners of the object according to the feature of the at least one feature point. , the edges and corners of objects can be detected from the image.
  • the electronic device 1000 may model the shape of the object in three dimensions based on the type of the object.
  • the electronic device 1000 may determine the type of the plane on which the object is placed based on the type of the object, and may model the shape of the object in three dimensions based on the type of the plane.
  • the plane on which the object is placed may be determined to be the floor type. Also, when it is determined that the object is an object attached to the wall, the plane on which the object is placed may be determined to be a wall type.
  • the electronic device 1000 may model the shape of the object in three dimensions based on the predicted direction of gravity with respect to the object.
  • the electronic device 1000 may sense a direction of gravity applied to the electronic device 1000 according to a gravity sensor provided in the electronic device 1000 .
  • the electronic device 1000 may predict the direction of gravity with respect to the object of the image based on the direction of gravity sensed by the electronic device 1000 .
  • the electronic device 1000 may predict the direction of gravity with respect to the object from the direction of gravity sensed by the electronic device 1000 based on tilt information of the electronic device 1000 when the object is captured.
  • the electronic device 1000 may predict the direction of gravity with respect to the object according to various methods.
  • the shape of the object may be modeled as a geometric shape to indicate at least one of an edge and a corner area of the object.
  • an object recognized as a box, a table, a book, etc. may be modeled in a rectangular parallelepiped shape according to a result of the object recognition.
  • an object recognized as a mug, a plastic bottle, etc. may be modeled in a cylindrical shape. Even if the shape of the object according to an embodiment is not clearly a geometric shape, it may be modeled as a geometric shape within a similar range. It is not limited to the above-described example, and the shape of the object may be modeled in various shapes.
  • the electronic device 1000 may identify a surface of the object that is in contact with a plane to be detected in the image based on the modeled result in operation 530 .
  • the shape of the object may be modeled as a geometric shape based on at least one of an edge and a corner region of the object, and according to the geometric shape, an edge or a corner area of the object that does not appear in the image is predicted.
  • a surface in contact with a plane among surfaces of the object may be identified based on the predicted edge or corner area of the object.
  • the electronic device 1000 may detect a plane included in the image based on the surface of the object in contact with the plane identified in operation 540 .
  • the electronic device 1000 may detect a plane based on a direction and a position of the surface of the object in contact with the plane identified in operation 540 .
  • a plane included in an image may be detected by detecting a plane modeled in three dimensions based on the shape of the object modeled in three dimensions.
  • a plane modeled in three dimensions based on the shape of the object modeled in three dimensions.
  • at least one feature point for a plane may be detected, and a three-dimensionally modeled plane may be detected based on the detected feature point.
  • a plane when a plurality of objects detected in an image exist, a plane may be detected based on the surfaces of the plurality of objects. For example, a plane may be detected based on values averaged with respect to the directions and positions of the surfaces of the plurality of objects. The above-described example is not limited, and the plane according to an embodiment may be detected based on the position and direction of the contact surface with respect to the plane of the object according to various methods.
  • the electronic device 1000 may display the augmented reality image by combining the plane included in the image captured in real time based on the detected plane. For example, the electronic device 1000 may recognize at least one plane included in an image captured in real time, and identify a plane corresponding to the detected plane according to an embodiment among the recognized planes. Accordingly, even if the scene of the image captured in real time is changed, the electronic device 1000 detects a plane in the scene of the currently captured image based on the previously detected plane information, and based on the detected plane , can be displayed by placing augmented reality images.
  • FIG. 6 is a block diagram illustrating an example of detecting a plane in an image according to an embodiment.
  • the electronic device 1000 may detect at least one object included in an image in 610 .
  • an object may be detected from an image by a pre-trained artificial intelligence model (eg, CNN) to recognize and detect the object from the image.
  • a pre-trained artificial intelligence model eg, CNN
  • the operations included in 620 may be continuously performed in accordance with an image being photographed in real time, but operations included in 610 and 630 are not performed in real time according to an image currently being photographed, but asynchronously. can be performed with
  • the electronic device 1000 may detect a plane based on an object by modeling the shape of the object detected in step 621 in three dimensions and identifying the surface of the object in contact with the plane.
  • the electronic device 1000 may identify a plane included in the currently captured image based on the plane detected at 622 , 621 .
  • the electronic device 1000 identifies an object corresponding to the three-dimensionally modeled object in 621 from among the objects included in the currently captured image in 623, and sets the object included in the current image to 3 It can be modeled dimensionally.
  • an operation for identifying the object 623 may be performed according to a method such as a simultaneous localization and mapping (SLAM) or a time of flight (ToF) point cloud of the 625 .
  • SLAM simultaneous localization and mapping
  • ToF time of flight
  • the electronic device 1000 may identify a contact surface with respect to the plane of the object based on the 3D model of the object included in the current image, in operation 624 .
  • a contact surface with respect to a plane may be identified based on various information such as a type of an object and a direction of gravity applied to the object. For example, when the type of object corresponds to an object that can be placed on a floor or a table, a surface of the object in the same direction as the direction of gravity applied to the object may be identified as a contact surface with respect to a plane.
  • the electronic device 1000 may filter feature points related to a plane in the image.
  • plane-related feature points may be extracted from 624.
  • the electronic device 1000 may identify the direction and position of the plane based on the direction and position of the contact surface identified in step 624 .
  • the electronic device 1000 may extract plane-related feature points from among the feature points extracted with respect to the image, based on the identified direction and location of the plane.
  • the feature point of the plane extracted may include a feature point located at an edge or a corner of the plane.
  • the electronic device 1000 may identify a position, a direction, and a size of a plane included in an image based on the feature points related to the plane extracted in 627 and 626. You can create a dimensional model. Also, in 630 , the electronic device 1000 may arrange at least one AR object on at least one plane identified as a 3D model included in the currently captured image.
  • the electronic device 1000 may determine visibility of each plane based on a result of arranging the AR object on each plane, in operation 628 .
  • the visibility of each plane included in the image may be determined based on whether the size of the plane displayed in the image is less than or equal to the reference value or whether the plane is located in an unsuitable location for an AR object to be placed.
  • visibility of each plane may be determined according to various methods for determining whether the plane is a suitable plane to be used for arranging an AR object.
  • a plane that is not suitable for arranging the AR object is excluded, and then a 3D model of the plane may be generated according to steps 626 and 627 .
  • FIG. 7 is a diagram illustrating an example of identifying a surface in contact with a plane among surfaces of an object according to an embodiment.
  • the electronic device 1000 may extract at least one feature point from a region in which an object related to a plane to be identified is displayed.
  • the electronic device 1000 identifies various types of objects in an image through image recognition, identifies a region in which a plane-related object is displayed, from among the identified objects, and identifies at least one in the region.
  • feature points can be extracted.
  • at least one feature point may be extracted with respect to a region in which at least one object related to a plane is displayed according to various methods.
  • the electronic device 1000 identifies a feature point located in at least one of the edges and corners of the objects 711 , 712 , and 713 among at least one feature point extracted from the region of each object, and identifies Based on the obtained feature points, each of the objects 711 , 712 , and 713 may be modeled in three dimensions.
  • the electronic device 1000 according to an embodiment, according to a three-dimensional modeling result for each of the objects 711 , 712 , and 713 , respectively, is a flat surface among the surfaces of the respective objects 711 , 712 , and 713 .
  • Surfaces in contact with can be identified.
  • the surface of the object in contact with the plane may be identified from among the surfaces of the object based on various information about the object, such as the type of the object and the direction of gravity of the object.
  • a plane displayed on the image may be identified based on the surface of each object in contact with the plane.
  • a three-dimensional model of the plane is generated, so that the plane can be identified, and based on the three-dimensional model of the plane, the AR object is to be placed.
  • FIG. 8 is a diagram illustrating an example of identifying a plane based on an object according to an embodiment.
  • the electronic device 1000 may identify the planes 841 , 842 , 843 , 844 , and 845 included in the image 810 by three-dimensional modeling as shown in 840 .
  • step 820 the electronic device 1000 uses the pre-trained artificial intelligence model to recognize the object in the image, and displays the regions 821, 822, 823, 824, and the object included in the image. 825) can be identified.
  • 821 of the object regions recognized in the image may be used to detect the wall-type plane 841 .
  • 822 of the object regions recognized in the image may be used to detect the table type 1 plane 842 .
  • 823 of the object regions recognized in the image may be used to detect the table type 2 plane 843 .
  • 824 including the table among the object regions recognized in the image may be used to detect the floor-type plane 844 .
  • the table 825 among the object regions recognized in the image may be used to detect the table type 3 plane 845 .
  • the electronic device 1000 may extract at least one feature point 831 , 832 from the regions 821 , 822 , 823 , 824 , and 825 in which the object identified in 820 is displayed. have.
  • a feature point 831 positioned at at least one of an edge and a corner of an object may be finally extracted as a feature point for identifying a plane.
  • the electronic device 1000 generates a 3D model for each object based on the feature point 831 positioned at at least one of an edge and a corner of the object, and based on the generated 3D model Thus, the contact surface for the plane of each object can be identified. Accordingly, the electronic device 1000 according to an embodiment identifies the planes 841 , 842 , 843 , 844 , and 845 included in the image in three dimensions as shown in 840 based on the identified contact surface. can do.
  • FIG. 9 is a diagram illustrating an example of modeling an object in three dimensions according to a direction of gravity according to an embodiment.
  • the electronic device 1000 may capture an image including a chair 911 and a table 912 .
  • the chair 911 and the table 912 may be recognized in the image, and based on the shapes of the chair 911 and the table 912 , the directions ( 911-1 and 912-1) may be determined.
  • the electronic device 1000 applies the image captured by the electronic device 1000 to the chair 911 and the table 912 based on the direction of gravity measured by the electronic device 1000 .
  • Gravity directions 911-2 and 912-2 may be determined.
  • the chair ( 911) and the table 912 may be determined to be in a state of being placed on a plane (eg, a floor plane) orthogonal to gravity, respectively.
  • a three-dimensional model of the chair 911 and the table 912 may be used to detect a plane orthogonal to gravity.
  • the electronic device 1000 may capture an image including a chair 921 and a table 922 .
  • an image may be captured in a state in which the chair 921 of 920 faces a direction 921-1 different from that of the chair 911 of 910 .
  • the directions 921 - 2 and 922 - 2 may be determined. According to an embodiment, it may be determined that the direction 921-1 of the chair 921 is different from the direction of gravity 921-2 applied to the chair 921 . Also, the direction 922-1 of the table 922 may be determined to coincide with the direction of gravity 922-2 applied to the table 922 .
  • the table 912 may be determined that the table 912 is placed on a plane orthogonal to gravity, and the chair 921 is not positioned on a plane orthogonal to gravity. Therefore, according to an embodiment, to detect a plane (eg, a floor plane) perpendicular to gravity, the 3D model of the chair 911 is not used, but the 3D model of the table 912 may be used. .
  • a plane eg, a floor plane
  • FIG. 10 is a diagram illustrating an example in which a plane is detected based on an image according to an embodiment.
  • a plurality of images including the same object 1001 , 1002 , and 1003 may be simultaneously captured by the plurality of electronic devices 1000 - 1 and 1000 - 2 .
  • a plurality of images including the same object 1001 , 1002 , and 1003 may be captured at different time points by one electronic device 1000 .
  • the 3D model for each object 1001 , 1002 , and 1003 generated based on images captured from different viewpoints is the same despite being based on different images. It can be created as a three-dimensional model (1004, 1005) with size and position. Accordingly, values measured in the three-dimensional models 1004 and 1005 for each object 1001, 1002, 1003, for example, a value indicating the size of each object 1001, 1002, 1003 (d1, d2, w3 and l3) and values L13 and L12 representing distances between objects may be obtained as the same value.
  • 3D models of the same object generated based on images captured from different viewpoints may be identically generated. Accordingly, according to an embodiment, the same plane included in each image may be identified based on a three-dimensional model of at least one object obtained based on a plurality of images captured from different viewpoints.
  • the user experience using the augmented reality service may be increased by effectively detecting the flat area using an object placed on the flat surface.
  • the device-readable storage medium may be provided in the form of a non-transitory storage medium.
  • 'non-transitory storage medium' is a tangible device and only means that it does not contain a signal (eg, electromagnetic wave). It does not distinguish the case where it is stored as
  • the 'non-transitory storage medium' may include a buffer in which data is temporarily stored.
  • the method according to various embodiments disclosed in this document may be included in a computer program product (computer program product) and provided.
  • Computer program products may be traded between sellers and buyers as commodities.
  • the computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store TM ) or on two user devices ( It can be distributed online (eg download or upload), directly between smartphones (eg smartphones).
  • a portion of the computer program product eg, a downloadable app
  • a machine-readable storage medium such as a memory of a manufacturer's server, a server of an application store, or a relay server. It may be temporarily stored or temporarily created.
  • unit may be a hardware component such as a processor or circuit, and/or a software component executed by a hardware component such as a processor.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Mathematical Optimization (AREA)
  • Multimedia (AREA)
  • Algebra (AREA)
  • Mathematical Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Mathematical Physics (AREA)
  • Pure & Applied Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un procédé de détection d'un plan dans une image, dans un dispositif électronique, comprenant : l'acquisition d'une image comprenant un plan ; la détection d'un objet dans l'image ; la modélisation de la forme de l'objet en trois dimensions ; l'identification d'une surface en contact avec un plan parmi des surfaces de l'objet sur la base du résultat modélisé ; et la détection d'un plan dans l'image sur la base de la surface identifiée.
PCT/KR2020/008264 2020-06-24 2020-06-25 Dispositif électronique de détection d'un plan dans une image et procédé de fonctionnement correspondant Ceased WO2021261619A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020200077374A KR102840430B1 (ko) 2020-06-24 2020-06-24 영상에서 평면을 검출하는 전자 장치 및 그 동작 방법
KR10-2020-0077374 2020-06-24

Publications (1)

Publication Number Publication Date
WO2021261619A1 true WO2021261619A1 (fr) 2021-12-30

Family

ID=79177869

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2020/008264 Ceased WO2021261619A1 (fr) 2020-06-24 2020-06-25 Dispositif électronique de détection d'un plan dans une image et procédé de fonctionnement correspondant

Country Status (2)

Country Link
KR (1) KR102840430B1 (fr)
WO (1) WO2021261619A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230260202A1 (en) * 2022-02-11 2023-08-17 Shopify Inc. Augmented reality enabled dynamic product presentation

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20250137311A (ko) * 2024-03-11 2025-09-18 주식회사 큐픽스 3차원 리얼리티 캡처 비용 산출 방법

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150095868A (ko) * 2012-12-18 2015-08-21 퀄컴 인코포레이티드 증강 현실 인에이블 디바이스들을 위한 사용자 인터페이스
KR20170035991A (ko) * 2014-07-25 2017-03-31 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 삼차원 혼합 현실 뷰포트
KR102061984B1 (ko) * 2018-08-07 2020-01-02 신상용 스마트글라스의 보행보조 컨트롤 방법 및 보행보조 컨트롤 시스템
JP2020509505A (ja) * 2017-03-06 2020-03-26 Ｌｉｎｅ株式会社 拡張現実を提供するための方法、装置及びコンピュータプログラム
KR20200061279A (ko) * 2018-11-23 2020-06-02 삼성전자주식회사 전자 장치 및 그 제어 방법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150095868A (ko) * 2012-12-18 2015-08-21 퀄컴 인코포레이티드 증강 현실 인에이블 디바이스들을 위한 사용자 인터페이스
KR20170035991A (ko) * 2014-07-25 2017-03-31 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 삼차원 혼합 현실 뷰포트
JP2020509505A (ja) * 2017-03-06 2020-03-26 Ｌｉｎｅ株式会社 拡張現実を提供するための方法、装置及びコンピュータプログラム
KR102061984B1 (ko) * 2018-08-07 2020-01-02 신상용 스마트글라스의 보행보조 컨트롤 방법 및 보행보조 컨트롤 시스템
KR20200061279A (ko) * 2018-11-23 2020-06-02 삼성전자주식회사 전자 장치 및 그 제어 방법

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230260202A1 (en) * 2022-02-11 2023-08-17 Shopify Inc. Augmented reality enabled dynamic product presentation
US11941750B2 (en) * 2022-02-11 2024-03-26 Shopify Inc. Augmented reality enabled dynamic product presentation

Also Published As

Publication number Publication date
KR102840430B1 (ko) 2025-07-29
KR20210158695A (ko) 2021-12-31

Similar Documents

Publication Publication Date Title
WO2020050636A1 (fr) Procédé et appareil de reconnaissance de gestes basée sur l'intention de l'utilisateur
CN112749613B (zh) 视频数据处理方法、装置、计算机设备及存储介质
CN110807361A (zh) 人体识别方法、装置、计算机设备及存储介质
CN111382624A (zh) 动作识别方法、装置、设备及可读存储介质
CN110570460A (zh) 目标跟踪方法、装置、计算机设备及计算机可读存储介质
CN110991491A (zh) 图像标注方法、装置、设备及存储介质
WO2019164092A1 (fr) Dispositif électronique de fourniture d'un second contenu pour un premier contenu affiché sur un dispositif d'affichage selon le mouvement d'un objet externe, et son procédé de fonctionnement
CN112395921B (zh) 异常行为检测方法、装置及系统
WO2017126741A1 (fr) Visiocasque et procédé de commande de celui-ci
WO2021045552A1 (fr) Dispositif électronique de synthèse d'image et son procédé de fonctionnement
CN109886208B (zh) 物体检测的方法、装置、计算机设备及存储介质
WO2018124500A1 (fr) Procédé et dispositif électronique pour fournir un résultat de reconnaissance d'objet
CN113918767A (zh) 视频片段定位方法、装置、设备及存储介质
WO2022080869A1 (fr) Procédé de mise à jour d'une carte tridimensionnelle au moyen d'une image et dispositif électronique prenant en charge ledit procédé
CN111027490A (zh) 人脸属性识别方法及装置、存储介质
CN110839128A (zh) 拍照行为检测方法、装置及存储介质
WO2021221436A1 (fr) Dispositif et procédé d'acquisition de profondeur d'espace à l'aide d'une caméra
WO2021230568A1 (fr) Dispositif électronique permettant de fournir un service de réalité augmentée et son procédé de fonctionnement
CN113378705B (zh) 车道线检测方法、装置、设备及存储介质
WO2021261619A1 (fr) Dispositif électronique de détection d'un plan dans une image et procédé de fonctionnement correspondant
WO2021149938A1 (fr) Dispositif électronique et procédé de commande de robot
CN113934878A (zh) 标注数据的审核方法、装置、终端及存储介质
CN111754272A (zh) 广告推荐方法、推荐广告显示方法、装置及设备
CN112990424A (zh) 神经网络模型训练的方法和装置
CN111192072A (zh) 用户分群方法及装置、存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20942309

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20942309

Country of ref document: EP

Kind code of ref document: A1