TWI897131B - Method of displaying three-dimentionsl virtual object, electronic device for executing the method and non-transitory computer-readable recording medium for recording computer program for executing the method - Google Patents

Abstract

The display method of a three-dimensional virtual object implemented by an electronic device in various embodiments of the present invention may include the following steps: generating a three-dimensional virtual object based on user input; and synthesizing the three-dimensional virtual object with a real image to display it in an augmented reality manner.

Description

Method for displaying three-dimensional virtual objects, electronic device for executing the method, and non-transitory computer-readable recording medium for recording a computer program for executing the method

The present invention relates to an electronic device for providing three-dimensional virtual objects related to products and its operation method.

Augmented reality (AR) is a technology that enhances the user experience by adding virtual elements (such as three-dimensional virtual objects) to the real world. By superimposing or compositing visual information into the real world, this technology provides users with rich interactions and has applications in various fields. In particular, the retail industry is exploring the use of AR to provide product information and enhance consumer interaction.

According to various embodiments of the present invention, the technical subject is to provide a three-dimensional virtual object that is synthesized with the real image and displayed in an augmented reality manner based on the real object within the real image, thereby allowing users to easily identify the size of the three-dimensional virtual object.

According to various embodiments of the present invention, the technical problem is that a processor determines the display size of a three-dimensional virtual object based on the actual size of the three-dimensional virtual object and the actual size of the real object within the real image.

According to various embodiments of the present invention, the technical problem is to generate three-dimensional virtual objects even when the electronic device does not include a lidar sensor.

According to various embodiments of the present invention, the technical problem is that a processor determines the display size of a three-dimensional virtual object based on a magnification determined by enlargement or reduction.

According to various embodiments of the present invention, the technical subject is that a processor displays various product information (e.g., delivery type, price discount, etc.) around a three-dimensional virtual object.

The technical topics of this invention are not limited to the technical topics mentioned above. Ordinary technicians in the technical field of this invention can clearly understand other technical topics not mentioned based on the description below.

In one embodiment, a method for displaying a three-dimensional virtual object using an electronic device may include the following steps: generating a three-dimensional virtual object based on user input; and combining the three-dimensional virtual object with a real image for display in an augmented reality manner. In one embodiment, the display size of the three-dimensional virtual object may be determined based on the actual size of the three-dimensional virtual object and the actual size of the real object within the real image.

In one embodiment, the step of generating a three-dimensional virtual object may include the following steps: generating at least one of an image set obtained from two or more different compositions, lidar sensor data, or the actual size of the three-dimensional virtual object based on user input; and generating the three-dimensional virtual object based on at least one of the image set, lidar sensor data, or the actual size.

In one embodiment, the step of combining a three-dimensional virtual object with a real image and displaying the object in an augmented reality manner may include the following step: displaying the texture of the three-dimensional virtual object on the surface of the three-dimensional virtual object.

In one embodiment, the step of generating a three-dimensional virtual object may include the following step: determining at least one of the actual size or shape of the three-dimensional virtual object based on user input.

In one embodiment, the step of generating a three-dimensional virtual object may include the following steps: transmitting at least one of an image set generated based on user input and obtained from two or more different compositions, lidar sensor data, or the actual size of the three-dimensional virtual object to a server; and receiving from the server the three-dimensional virtual object generated based on at least one of the image set, lidar sensor data, or the actual size.

The method of one embodiment may further include the step of determining whether the actual image includes a reference object having a predetermined actual size and shape.

In one embodiment, the step of combining a three-dimensional virtual object with a real image and displaying the object in an augmented reality manner may include the following steps: when the real image includes a reference object, determining the display size of the three-dimensional virtual object based on the actual size of the three-dimensional virtual object and the actual size of the reference object.

In one embodiment, the step of combining a three-dimensional virtual object with a real image and displaying the object in an augmented reality manner may include the following steps: determining the position of the three-dimensional virtual object based on the position of a reference object.

In one embodiment, the step of combining a three-dimensional virtual object with a real image and displaying the object in an augmented reality manner may include the following steps: moving or rotating the position of the three-dimensional virtual object in the real image based on user input.

In one embodiment, the step of combining a three-dimensional virtual object with a real image and displaying the object in an augmented reality manner may include the following steps: adjusting the magnification of a camera included in the electronic device based on user input; and changing the display size of the three-dimensional virtual object based on the adjusted magnification.

The method of one embodiment may further include the step of displaying at least one of delivery type information, price discount information, review information, or purchase items of a product corresponding to the three-dimensional virtual object within a fixed distance from the three-dimensional virtual object.

In one embodiment, the delivery type data is data related to the delivery mode, and may include at least one of regular delivery, express delivery, fresh food delivery, or overseas delivery.

The method of one embodiment may further include the following steps: displaying product sales catalog data of a seller's account that sells a product in the actual image, where the product corresponds to the three-dimensional virtual object; or displaying recommended product data of the same product category as the product in the actual image.

According to various embodiments of the present invention, augmented reality technology is used to present product information to customers. This allows consumers to verify the product and make a purchasing decision in a virtual environment, just as if they were experiencing the product in person.

According to various embodiments of the present invention, an electronic device displays a three-dimensional virtual object corresponding to a product reserved for purchase in an augmented reality format, thereby allowing the user to easily compare the size of the three-dimensional real object included in the actual image with the size of the product reserved for purchase.

According to various embodiments of the present invention, even when the electronic device does not include a lidar sensor, a three-dimensional virtual object can be generated based on images captured by a camera in multiple compositions.

According to various embodiments of the present invention, even when a real image is enlarged or reduced, the displayed size of a three-dimensional virtual object is adjusted to match the displayed size of the real object based on the magnification. This allows users to compare the size of the real three-dimensional object included in the real image with the size of the product they have reserved for purchase.

The effects of the technical concept of this invention are not limited to the effects mentioned above. A person of ordinary skill in the art can clearly understand other effects not mentioned based on the description in the specification.

110: Electronic devices

120: User terminal

200: Electronic devices

210: Communication Circuit

220: Processor

230: Memory

300: Screen

310: Three-dimensional virtual objects

330: Base Object A

370: Base Object B

380: Price Discount Information

381: Delivery type information

390: Other Products

391:Commodity

392:Commodity

393:Commodity

400: Screen

410: Area

430: Scan object

450: Image upload object

470: Dimension Input Object

500: Screen

510: Actual objects

530: Actual objects

610: Commodity

611: Image Gallery

612: Image Gallery

613: Image Gallery

614: Image Gallery

615: Image Gallery

616: Image Gallery

617: Image Gallery

618: Image Gallery

619: Image Gallery

620: Image Gallery

621: Image Gallery

622: Image Gallery

700: Screen

701: Screen

710: Shape

720: Actual size

721: Actual size

722: Actual size

730: Three-dimensional virtual objects

750: Actual objects

810:Action

820:Action

Figure 1 is a diagram showing a system of various embodiments of the present invention.

Figure 2 is a block diagram showing an electronic device according to various embodiments of the present invention.

FIG3 is a diagram illustrating various embodiments of the present invention for combining three-dimensional virtual objects with real images for display in an augmented reality manner.

FIG4 is a diagram illustrating a screen for generating a three-dimensional virtual object according to various embodiments of the present invention.

FIG5 is a diagram illustrating a method for obtaining lidar sensor data for generating a three-dimensional virtual object according to various embodiments of the present invention.

FIG6 is a diagram illustrating a method for obtaining images of multiple compositions for generating a three-dimensional virtual object according to various embodiments of the present invention.

FIG7a and FIG7b are diagrams of a three-dimensional virtual object used to illustrate the actual size or shape of the three-dimensional virtual object determined based on user input according to various embodiments of the present invention.

Figure 8 is a flowchart of the operation of the electronic device according to various embodiments of the present invention.

The embodiments of this invention are provided for the purpose of illustrating the technical concept of this invention. The scope of the invention application is not limited to the following embodiments or the detailed description of such embodiments.

Unless otherwise defined, all technical and scientific terms used in this invention have the meanings commonly understood by those with ordinary skill in the art to which this invention belongs. All terms used in this invention are selected for the purpose of more clearly describing the invention and are not selected to limit the scope of the invention application.

Unless otherwise stated in the sentence or text containing the corresponding expression, expressions such as "including," "having," and "having" used in the present invention should be understood as open-ended terms that may include other embodiments.

Unless otherwise specified, singular expressions described in the present invention may include plural expressions. This also applies to singular expressions described in the scope of the invention application.

The terms "first," "second," etc. used in this invention are used to distinguish multiple components from each other and do not limit the order or importance of these components.

The term "component" as used in the present invention refers to software, or a hardware component such as an FPGA (field-programmable gate array) or an ASIC (application specific integrated circuit). However, a "component" is not limited to hardware and software. A "component" can be configured in a manner located on an addressable storage medium, or can be configured in a manner that reproduces one or more processors. Therefore, as an example, a "component" includes components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, snippets of code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided by components and parts can be combined into a smaller number of components and parts, or further separated into other components and parts.

The expression "based on..." used in this invention is used to describe one or more factors described in the sentence or article containing the expression that influence the behavior or action of determining or judging. The expression does not exclude other factors that influence the behavior or action of determining or judging.

In the present invention, when a component is referred to as being "connected" or "linked" to another component, it should be understood that the component can be directly connected or linked to the other component, or can be connected or linked to the other component through another new component.

The following describes embodiments of the present invention with reference to the accompanying drawings. In the accompanying drawings, identical or corresponding components are denoted by the same reference numerals. In the following descriptions of the embodiments, repeated descriptions of identical or corresponding components may be omitted. However, even if the description of a component is omitted, it does not mean that such component is not included in a particular embodiment.

Figure 1 illustrates an environment in which electronic devices can be used in accordance with one embodiment of the present invention. An electronic device 110 and a user terminal (120, hereinafter referred to as a "terminal") can communicate with each other via a network connection. Electronic device 110 can be a server or another terminal distinct from terminal 120.

The terminal 120 can be implemented as a terminal capable of transmitting and receiving various types of information with the electronic device 110 via a network. For example, the terminal 120 can be a computer, a laptop, a portable communication terminal (such as a smartphone), a portable multimedia device, a wearable device, or an HMD (head-mounted display). However, the type of terminal 120 is not limited thereto. The terminal 120 can be any device that includes an input/output interface for inputting information to or outputting information to a user and can communicate with the electronic device 110 or other devices via a network. The terminal 120 can provide the user with information received from the electronic device 110 and can receive input from the user and transmit it to the electronic device 110. Input obtained from the user may include various forms of input, such as clicks using a mouse, touches using a touch panel or touch screen, voice recognition, and other electronic input.

A network can connect the electronic device 110 to the terminal 120 or other external devices. For example, the network can provide a connection path so that the terminal 120 can connect to the electronic device 110 and send and receive packet data with the electronic device 110. For example, the network can be implemented as any type of wired or wireless network, such as a local area network (LAN), a wide area network (WAN), a mobile radio communication network, or Wibro (wireless broadband internet).

FIG2 is a block diagram of an electronic device according to an embodiment of the present invention. In one embodiment, electronic device 200 may be a server or a terminal. In one embodiment, electronic device 200 may include a communication circuit 210, one or more processors 220, and/or one or more memories 230 as components. In one embodiment, at least one of the components of electronic device 200 may be omitted, or other components may be added to electronic device 200. In one embodiment, a portion of the components may be additionally or alternatively integrated to achieve the desired effect, or the device may be implemented as a single or multiple components. In the present invention, one or more processors 220 may be referred to as processor 220. Unless otherwise specified, the processor 220 may refer to a collection of one or more processors. In the present invention, one or more memories 230 may be referred to as memory 230. Unless otherwise specified, the memory 230 may refer to a collection of one or more memories. In one embodiment, at least some of the internal/external components of the electronic device 200 are connected to each other via a bus, GPIO (General Purpose Input/Output), SPI (Serial Peripheral Interface), or MIPI (Mobile Industry Processor Interface), thereby enabling the transmission and reception of information (data, signals, etc.).

The communication circuit 210 can communicate with the user's terminal and external devices. The communication circuit 210 can implement wireless or wired communication between the electronic device 200 and the terminal. For example, the communication circuit 210 can implement technologies based on eMBB (enhanced Mobile Broadband), URLLC (Ultra Reliable Low-Latency Communications), MMTC (Massive Machine Type Communications), LTE (Long-Term Evolution), LTE-A (LTE Advance), NR (New Radio), UMTS (Universal Mobile Telecommunications System), GSM (Global System for Mobile communications), CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), WiBro (Wireless Broadband), WiFi (Wireless Fidelity), Bluetooth, and NFC (Near Field Communication circuit 210 can implement wireless communication via methods such as near-field communication (NFC), GPS (Global Positioning System), or GNSS (Global Navigation Satellite System). For example, communication circuit 210 can implement wired communication via methods such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), RS-232 (Recommended Standard-232), or POTS (Plain Old Telephone Service). In one embodiment, electronic device 200 can also be integrated with other devices. In this case, communication circuit 210 can connect electronic device 200 to the corresponding connection circuit or interface of the other device.

The processor 220 can drive software (e.g., commands, programs, etc.) to control at least one component of the electronic device 200 connected to the processor 220. Furthermore, the processor 220 can perform various operations related to the present invention, such as calculations, processing, data generation, and processing. Furthermore, the processor 220 can load data from the memory 230 or store data in the memory 230. Furthermore, the processor 220 can transmit and receive various information with the user's terminal and external devices via the communication circuit 210. In one embodiment, the processor 220 can control the communication circuit 210 to transmit various information, such as information on various web pages, to the user's terminal.

Memory 230 can store various information (data). Information stored in memory 230 is information obtained, processed, and used by at least one component of electronic device 200, and may include software (e.g., commands, programs, etc.). Memory 230 may include volatile and/or non-volatile memory. In the present invention, commands or programs are software stored in memory 230, and may include an operating system, applications, and/or middleware used to control the resources of electronic device 200. The middleware provides various functions to applications so that the applications can utilize the resources of electronic device 200. In one embodiment, the memory 230 may store commands that, when executed by the processor 220, cause the processor 220 to perform operations. The memory 230 may also store at least a portion of information received from a terminal via the communication circuit 210 and/or information sent to a terminal via the communication circuit 210. The processor 220 may store at least a portion of information received from a terminal via the communication circuit 210 and/or information sent to a terminal via the communication circuit 210 in the memory 230.

Below, the operations described in Figures 3 to 8 as being performed by the electronic device can be understood as being performed by the processor 220 of the electronic device 200 described in Figure 2.

In the configuration diagrams and flow charts shown in Figures 3 and 8 , process steps, method steps, and algorithms are described sequentially. However, such processes, methods, and algorithms can be constructed so that the steps are performed in any appropriate order. In other words, the steps of the processes, methods, and algorithms described in the various embodiments of the present invention do not necessarily need to be performed in the order described. Furthermore, even if some steps are described as being performed asynchronously, in other embodiments, such steps may be performed simultaneously. Furthermore, the examples of processes described in the accompanying figures do not exclude other variations or modifications of the illustrated processes, nor do they imply that the illustrated processes or any of their steps are required for one or more of the various embodiments of the present invention, nor do they imply that the illustrated processes are preferred.

FIG3 is a diagram illustrating various embodiments of the present invention for combining three-dimensional virtual objects with real images for display in an augmented reality manner.

In one embodiment, the processor 220 may generate a three-dimensional virtual object based on user input. In one embodiment, when the electronic device 200 is a terminal, the terminal may generate a three-dimensional virtual object based on user input. The processor 220 may generate at least one of an image set obtained from two or more different compositions, lidar sensor data, or the actual size (and/or shape) of the three-dimensional virtual object based on the user input. The processor 220 may generate the three-dimensional virtual object based on at least one of the image set, lidar sensor data, or the actual size (and/or shape). In another embodiment, the processor 220 may transmit the data generated based on the user input to a server. The server can generate a three-dimensional virtual object based on the received data and transmit it to the terminal. The processor 220 included in the terminal can display the three-dimensional virtual object received from the server on the screen.

In one embodiment, user input may correspond to a request to generate a three-dimensional virtual object for a specific product. For example, referring to FIG. 4 , upon receiving user input for at least one of a scan object 430 , an image upload object 450 , or a size input object 470 , the processor 220 may generate data corresponding to the user input and, based on the generated data, generate a three-dimensional virtual object.

In one embodiment, the processor 220 may generate a three-dimensional virtual object based on user input regarding a scanned object 430. The scanned object 430 may be a target for generating the three-dimensional virtual object based on lidar sensor data. Lidar sensor data may be obtained from the lidar sensor. Lidar (Light Detection and Ranging) sensors are a technology that uses wavelengths of light to measure distance and detect the environment. Lidar sensors primarily calculate distance by emitting laser light and measuring the time it takes for the light to reflect and return to an object. This generates lidar sensor data (e.g., spatial information of the surrounding environment, object size, distance to the object, and object shape).

When the scan object 430 is selected in the screen 400 of FIG. 4 , the screen 500 of FIG. 5 may be displayed. Referring to FIG. 5 , the screen 500 is a screen that displays guide lines in the actual image based on the lidar sensor data. Guide lines can be lines used to display spatial information about the surrounding environment. When multiple guide lines intersect, a grid can be generated with a fixed spacing, and the grid length can correspond to a specific length (e.g., 1 cm, 2 cm). The user can know the length of a specific object based on the number of grids. The processor 220 can calculate the distance between the electronic device 200 and the actual object 510 based on the lidar sensor data. In this way, processor 220 can calculate the actual dimensions (e.g., width, length, height, etc.) of object 510. Similarly, by performing the above operations on real object 530, processor 220 can calculate the actual dimensions of real object 530. Processor 220 can generate a three-dimensional virtual object based on lidar sensor data captured in multiple different compositions of real object 510. For example, a user can simultaneously capture various views of real object 510, such as the front, back, left side, and right side, while using mobile electronic device 200. In this way, processor 220 can obtain lidar sensor data for various views of real object 510.

The term "actual size" used in this specification may refer to the size of an object in a real environment, and "display size" may refer to the size of an object displayed on a screen within an actual image displayed on the display unit of the electronic device 200.

In one embodiment, the processor 220 can display the texture of a three-dimensional virtual object on the surface of a virtual object generated based on the actual size and shape of the three-dimensional virtual object included in the lidar sensor data. Texture is an image or pattern used in computer graphics. It can be data applied to the surface of a three-dimensional model to determine the object's appearance. Texture can be mapped onto the surface of an object to simulate a specific material, color, uniformity, texture, etc., to impart visual realism or to create a specific design. Texture can be generated based on images captured by a camera included in the electronic device 200. Processor 220 generates the size and shape of a 3D virtual object based on lidar sensor data. However, to display the 3D virtual object's surface identical to the actual product, texture may be required. Therefore, processor 220 can utilize texture to display the 3D virtual object's surface. This allows processor 220 to provide users with a more realistic 3D virtual object. By displaying texture on the object's surface, users can compare the color of the real object included in the actual image with the color of the 3D virtual object to determine whether they match. Therefore, when users make a purchase decision, they should consider not only the product's size but also its color, and whether the color of the product coordinates with the surrounding environment.

In one embodiment, the processor 220 can generate a three-dimensional virtual object based on user input to the image upload object 450. The image upload object 450 can be an object used to generate a three-dimensional virtual object based on a set of images obtained from two or more different compositions. The electronic device 200 may or may not include a lidar sensor. Even if the electronic device 200 does not include a lidar sensor, there may still be a need to generate a three-dimensional virtual object. Therefore, in the absence of lidar sensor data, the processor 220 can generate a three-dimensional virtual object based on a set of images obtained from two or more different compositions. The processor 220 can generate the actual size of a three-dimensional virtual object based on depth data (e.g., the distance between an electronic device and an object) obtained by analyzing images included in the image set. Furthermore, the processor 220 can determine the shape of the three-dimensional virtual object based on the depth information. After forming the virtual object based on the actual size and shape of the three-dimensional virtual object, the processor 220 can display the surface of the virtual object based on a texture. The texture can be generated based on the image set. In another embodiment, the processor 220 can transmit the image set to a server. The server can generate the data required to generate the three-dimensional virtual object based on the image set. The processor 220 may generate the 3D virtual object using the data required for generating the 3D virtual object received from the server. Alternatively, the server may generate the 3D virtual object based on an image set.

In one embodiment, the processor 220 can generate a three-dimensional virtual object based on user input into a dimension input object 470. In the absence of lidar sensor data and an image set, the dimension input object 470 can be used to generate the three-dimensional virtual object. When the dimension input object 470 in FIG. 4 is selected, the screen 700 in FIG. 7 a can be displayed. The processor 220 can receive the shape 710 (e.g., a cylinder, a cuboid, a pyramid, etc.) and actual dimensions 720, 721, and 722 of the three-dimensional virtual object from the user via screen 700. The actual dimensions can include, for example, width 720, length 721, and height 722. Processor 220 may determine actual dimensions 720, 721, 722 and shape 710 based on user input. Furthermore, processor 220 may generate a three-dimensional virtual object based on the determined actual dimensions 720, 721, 722 and shape 710. In reality, the actual dimensions of a three-dimensional virtual object obtained by a lidar sensor or camera may be inaccurate. Alternatively, a lidar sensor or image may be difficult to obtain in an environment. In such cases, the user may also enter the actual dimensions of the product displayed in the product details to generate a three-dimensional virtual object with the exact dimensions and a shape similar to the product.

In one embodiment, the processor 220 can combine a three-dimensional virtual object generated based on actual dimensions 720, 721, 722 and shape 710 with a real image for augmented reality display. Referring to FIG. 7b , in a screen 701 of a real image captured by the camera of the electronic device 200, not only a real object 750 can be displayed, but a three-dimensional virtual object 730 can also be combined and displayed simultaneously. This allows the processor 220 to display a three-dimensional virtual object within the real image, allowing comparison of the real object's dimensions, without using a lidar sensor or camera. Furthermore, by comparing accurately sized 3D virtual objects with real objects, users can compare the dimensions of the corresponding product with other real objects. In one embodiment, users can also input texture information for the 3D virtual object to display the color or texture of the 3D virtual object.

The screens shown in Figures 4, 5, 7a, and 7b above may be screens of a terminal connected to a seller's account (or administrator's account). The seller's terminal may generate a three-dimensional virtual object for the e-commerce platform and upload it along with product information. The seller's terminal may combine the three-dimensional virtual object with a real image and display it in augmented reality. This allows the seller to confirm within the real image whether the product being sold is accurately represented as a three-dimensional virtual object. For example, screen 300 of Figure 3 may be displayed on the seller's terminal. In this way, sellers can confirm whether the three-dimensional virtual object 310 corresponding to the product they are selling effectively presents the product's characteristics.

In one embodiment, a consumer may need to combine a three-dimensional virtual object corresponding to a product with a real image to display it in an augmented reality format. To do this, the three-dimensional virtual object generated corresponding to the product may be transmitted to the server of the e-commerce platform. The server may receive a request for a three-dimensional virtual object for a specific product from the consumer's account terminal. The server may transmit the three-dimensional virtual object to the consumer's account terminal. The screen of the consumer's account terminal may display a real object and the three-dimensional virtual object, as shown in screen 300 of FIG. 3 . For example, the seller's account terminal may generate a three-dimensional virtual object of product A. The seller's account terminal can transmit the 3D virtual object of Product A to the e-commerce platform's server. The consumer's account terminal can transmit a request for the 3D virtual object of Product A to the server. The server can transmit the 3D virtual object of Product A to the consumer's account terminal. The consumer's account terminal can display the received 3D virtual object of Product A on a screen (e.g., a real-life image).

In one embodiment, the processor 220 can determine whether the real image includes a reference object with a predetermined real size and shape. A reference object may be an object whose real size and shape data is stored in memory (or server memory). For example, the reference object may include common real-life items. For example, soda cans, cans, snack boxes, mobile phones, books, and electrical outlets are all common in real life. After storing the actual size and shape data of such objects in memory, processor 220 compares the actual size of the reference object with the actual size of the 3D virtual object when a reference object appears in the real image. This allows the 3D virtual object to be displayed more accurately on the screen in an augmented reality manner. Processor 220 can retrieve the actual size and shape of the reference object from memory included in the electronic device or request the actual size and shape of the reference object from a server. For example, referring to FIG3 , a socket is a common object in real life and can be used as a reference object. Therefore, the processor 220 can determine the displayed size of the three-dimensional virtual object 310 based on the actual size of the reference object A 330 corresponding to the socket and the actual size of the three-dimensional virtual object. The size of the three-dimensional virtual object 310 can be determined based on the display position of the three-dimensional virtual object. For example, the displayed size of the three-dimensional virtual object 310 is larger when the three-dimensional virtual object 310 is placed closer to the user than when the three-dimensional virtual object 310 is placed farther away from the user. The display position of the three-dimensional virtual object 310 can be determined and/or changed by the user. As another example, a can can also be used as a reference object. The processor 220 can determine the display size of the three-dimensional virtual object 310 based on the actual size of the reference object B 370 corresponding to the can and the actual size of the three-dimensional virtual object.

In one embodiment, when a reference object is included in the real image, the processor 220 may compare the actual size of the reference object with the actual size of the three-dimensional virtual object to determine the displayed size of the three-dimensional virtual object. If the real image does not include a reference object, the processor 220 may determine the displayed size of the three-dimensional virtual object based on the actual size of the real object in the real image and the actual size of the three-dimensional virtual object. The actual size of the real object may be calculated based on the displayed size of the real object in the real image and the distance from the electronic device 200 to the real object.

In one embodiment, the processor 220 can determine the position of a displayed three-dimensional virtual object based on the position of a reference object. For example, the processor 220 can identify the position of a reference object within a real image. Based on the identified position of the reference object, the processor 220 can determine the position of the three-dimensional virtual object within the real image. If the reference object within the real image is located below the real image, the processor 220 can display the three-dimensional virtual object at a fixed distance upward from the real reference object. For illustration, referring to FIG. 3 , reference object B 370 can be located below the real image. Therefore, the processor 220 can display the three-dimensional virtual object 310 at a fixed distance upward from the position of the reference object B 370. In another embodiment, when the reference object in the real image is located at the top of the real image, the processor 220 can display the three-dimensional virtual object at a fixed distance downward from the real reference object. For illustration, referring to FIG3 , the reference object A 330 can be located at the top of the real image. Therefore, the processor 220 can display the three-dimensional virtual object 310 at a fixed distance downward from the position of the reference object A 330. In this way, the processor 220 can display both the reference object and the three-dimensional virtual object within the real image.

The processor 220 of one embodiment may move or rotate the position of a three-dimensional virtual object in a real image based on user input. The processor 220 of one embodiment may display the three-dimensional virtual object 310 at a predetermined position (e.g., the center of the real image). The processor 220 of one embodiment may determine the predetermined position by taking into account the position at which the real object is displayed in the real image. For example, the processor 220 may determine the predetermined position to prevent overlap between the real object and the three-dimensional virtual object. The processor 220 of one embodiment may move the position of the three-dimensional virtual object 310 based on user input. Furthermore, the processor 220 can rotate the three-dimensional virtual object 310 based on user input. This allows the user to arrange products (e.g., furniture, electronic products, etc.) corresponding to the three-dimensional virtual object within the real image, thereby simulating product placement.

In one embodiment, the processor 220 can adjust the magnification of a camera included in the electronic device based on user input. For example, the device can zoom in or out based on user input. Zooming in may refer to increasing the magnification, while zooming out may refer to decreasing the magnification. Based on the adjusted magnification, the processor 220 can change the displayed size of a three-dimensional virtual object. Adjusting the magnification can also change the displayed size of a real object within the real image. Since the displayed size of the real object changes, the displayed size of the three-dimensional virtual object also needs to change. The processor 220 can know the ratio between the actual size of the real object and its displayed size. Therefore, the processor 220 can change the displayed size of the 3D virtual object based on the ratio of the actual size of the real object to the displayed size. This allows the actual size of the 3D virtual object to change even when the magnification is changed, allowing the user to accurately compare the sizes of the real object with the 3D virtual object.

Within a fixed distance from a three-dimensional virtual object, the processor 220 of one embodiment can display at least one of the delivery type data of the product corresponding to the three-dimensional virtual object, the price discount data of the product, the review data of the product, or the purchase object. Referring to FIG3 for illustration, within a fixed distance from the three-dimensional virtual object 310, the delivery type data 381 can be displayed. The delivery type is the type of delivered product, and may include, for example, ordinary delivery, express delivery, fresh food delivery, overseas delivery, etc. Express delivery may be a delivery type that is shorter than the average delivery time of ordinary delivery and is completed within the promised time (for example, 1 day). Fresh food delivery may be a type that, for the purpose of fresh food delivery, delivers the product with packaging that maintains the freshness of the food. Overseas delivery can involve importing goods traded overseas and delivering them domestically to customers. The delivery type directly influences a user's decision to purchase a product. Therefore, delivery type data 381 can be displayed alongside a 3D virtual object, allowing the user to visually identify the delivery type and make a purchase decision. Furthermore, price discount information 380 and review data that influence the user's decision to purchase the product can also be displayed on the screen. This allows the user to comprehensively grasp the information necessary for a product purchase through a single screen 300. The processor 220 of one embodiment can also display the purchased item on screen 300. When selecting an item to purchase, the user can purchase the product corresponding to the three-dimensional virtual object 310 . This simplifies the user's product purchase process. The present invention is not limited to displaying only a portion of the aforementioned data or displaying additional data on screen 300 .

In one embodiment, the processor 220 can display not only the product corresponding to the 3D virtual object in the real image (or screen), but also other products in the real image (or screen). In one embodiment, the processor 220 can display product sales catalog data for a seller's account that sells products. For example, there may be a seller account A that sells the product corresponding to the 3D virtual object 310. Furthermore, seller account A may also sell other products. The processor 220 can also display other products sold by seller account A in the real image. For example, referring to FIG3 , the processor 220 may display other products 391, 392, and 393 sold by seller account A in the area displaying other products 390. This allows the user to compare the product corresponding to the three-dimensional virtual object with other products sold by a specific seller account. In another embodiment, the processor 220 may display recommended product data for products of the same product category. The recommended product data may be data determined based on user preference data. The user preference data may be generated based on the user's product purchase log data. For example, if the product corresponding to the three-dimensional virtual data is milk, the recommended product data may include other products included in the milk category. For example, if the user preference data includes that user A prefers to purchase milk in bottles less than 1L, the recommended product data may include other milks in bottles less than 1L. Referring to FIG. 3 , processor 220 may display products 391, 392, and 393 included in the recommended product data within the area displaying other products 390. This simplifies the online shopping process because users can view other products within the actual image without having to switch to other screens to view other products.

FIG4 is a diagram illustrating a screen for generating a three-dimensional virtual object according to various embodiments of the present invention.

Screen 400 in FIG. 4 may include a three-dimensional virtual object display area 410. The three-dimensional virtual object display area 410 may display a generated three-dimensional virtual object. The three-dimensional virtual object displayed in the three-dimensional virtual object display area 410 may be generated based on user input to at least one of a scan object 430, an image upload object 450, or a size input object 470.

FIG5 illustrates a method for obtaining lidar sensor data for generating a three-dimensional virtual object according to various embodiments of the present invention. The detailed description of FIG5 has been described above in FIG3 and is therefore omitted in this figure.

FIG6 is a diagram illustrating a method for obtaining images of multiple compositions for generating a three-dimensional virtual object according to various embodiments of the present invention.

In one embodiment, the processor 220 can generate a three-dimensional virtual object based on a set of images obtained from two or more different compositions. For example, to generate a three-dimensional virtual object of product 610, the processor 220 can use a camera to obtain a set of images 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, and 622 captured in two or more different compositions. The set of images shown in FIG6 is merely an example; the number of images included in the set can be fewer or more than that shown in FIG6. The processor 220 can generate a three-dimensional virtual object of the product 610 based on the image sets 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, and 622.

Figures 7a and 7b illustrate three-dimensional virtual objects, illustrating how the actual size or shape of the three-dimensional virtual object is determined based on user input in various embodiments of the present invention. The detailed description of Figures 7a and 7b has been described above in connection with Figure 3 and is therefore omitted in this figure.

Figure 8 is a flowchart of the operation of the electronic device according to various embodiments of the present invention.

In one embodiment, the electronic device 200 can generate a three-dimensional virtual object (810) based on user input. The user input can be input for at least one of the following inputs: a scanned object 430, an image upload object 450, or a size input object 470. The electronic device 200 can generate the three-dimensional virtual object based on at least one of lidar sensor data, an image set, or the actual size (and/or shape) of the three-dimensional virtual object. Some or all of the actions in the process of generating the three-dimensional virtual object can also be implemented in the server.

In one embodiment, the electronic device 200 can combine a 3D virtual object with a real image and display it in augmented reality 820. The displayed size of the 3D virtual object can be determined based on the actual size of the 3D virtual object and the actual size of the real object within the real image. This allows the user to intuitively compare the size of the real object with the size of the 3D virtual object within the real image.

Although the above method is described using specific embodiments, the above method can also be implemented as computer-readable code in a computer-readable recording medium. Computer-readable recording media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM (Read Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tapes, floppy disks, optical data storage devices, and the like. Furthermore, computer-readable recording media can be distributed across computer systems connected via a network to store and execute computer-readable code in a distributed manner. Furthermore, the functional programs, codes, and code segments used to implement the above embodiments can be easily derived by programmers in the technical field to which the present invention belongs.

300: Screen 310: 3D virtual object 330: Base object A 370: Base object B 380: Price discount data 381: Delivery type data 390: Other products 391: Product 392: Product 393: Product

Claims (13)

A method for displaying a three-dimensional virtual object is implemented by an electronic device and includes the following steps: generating a three-dimensional virtual object based on user input; determining whether a real image includes a reference object with a predetermined real size and shape; and synthesizing the three-dimensional virtual object with the real image to display the object in an augmented reality manner, wherein the three-dimensional virtual object is synthesized with the real image to display the object in an augmented reality manner. The displaying step includes the following steps: if the reference object is not included in the actual image, determining the display size of the three-dimensional virtual object based on the actual size of the three-dimensional virtual object and the actual size of the actual object in the actual image; and if the actual image includes the reference object, determining the display size of the three-dimensional virtual object based on the actual size of the three-dimensional virtual object and the actual size of the reference object. The method of claim 1, wherein the step of generating the three-dimensional virtual object comprises the steps of: generating at least one of an image set obtained from two or more different compositions, lidar sensor data, or the actual size of the three-dimensional virtual object based on the user input; and generating the three-dimensional virtual object based on at least one of the image set, lidar sensor data, or the actual size of the three-dimensional virtual object. In the method of claim 1, the step of synthesizing the three-dimensional virtual object with a real image and displaying the image in an augmented reality manner includes the following step: displaying the texture of the three-dimensional virtual object on the surface of the three-dimensional virtual object. The method of claim 1, wherein the step of generating the three-dimensional virtual object comprises the following step: determining at least one of the actual size or shape of the three-dimensional virtual object based on user input. The method of claim 1, wherein the step of generating the three-dimensional virtual object comprises the steps of: transmitting at least one of an image set obtained from two or more different compositions generated based on the user input, lidar sensor data, or the actual size of the three-dimensional virtual object to a server; and receiving from the server the three-dimensional virtual object generated based on at least one of the image set, lidar sensor data, or the actual size of the three-dimensional virtual object. In the method of claim 1, the step of synthesizing the three-dimensional virtual object with the real image and displaying it in an augmented reality manner includes the following step: determining the position of the three-dimensional virtual object based on the position of the reference object. The method of claim 1, wherein the step of combining the three-dimensional virtual object with the real image and displaying the object in an augmented reality manner includes the following step: moving or rotating the position of the three-dimensional virtual object in the real image based on user input. The method of claim 1, wherein the step of combining the three-dimensional virtual object with a real image to display the object in an augmented reality manner comprises the steps of: adjusting the magnification of a camera included in the electronic device based on user input; and changing the display size of the three-dimensional virtual object based on the adjusted magnification. The method of claim 1 further comprises the step of displaying at least one of delivery type information, price discount information, review information, or purchase items for a product corresponding to the three-dimensional virtual object within a fixed distance from the three-dimensional virtual object. The method of claim 9, wherein the delivery type data is data related to a delivery type, and the delivery type is at least one of regular delivery, express delivery, fresh food delivery, or overseas delivery. The method of claim 1 further comprises the steps of: displaying, within the actual image, product sales catalog information from a seller's account that sells products corresponding to the three-dimensional virtual object; or displaying, within the actual image, recommended product information of the same product category as the product. An electronic device comprising: a processor; and a memory storing instructions configured to be executed by the processor; the processor being configured to execute the method of any one of claims 1 to 11. A non-transitory computer-readable recording medium recording a computer program for execution by a processor, wherein the computer program is configured to cause the processor to execute the method of any one of claims 1 to 11.