KR20130134273A - System for supplying virtual menu conneted with object and method thereof - Google Patents

Abstract

The present invention relates to a system and method for providing a virtual menu associated with an object, and more particularly, to an augmented reality technology having an association with a morphological and functional aspect of a real object.
To this end, the system for providing a virtual menu associated with an object according to the present invention includes an object association information database, a live image receiving unit, an object recognition unit, an object identification information comparison unit, a virtual menu enhancer, and a controller. The object association information database stores object identification information of the object and information defined to associate one or more virtual menus with the object identification information. The live image receiving unit receives a live image captured by a camera. The object recognizer recognizes an object included in the live image. The object identification information comparison unit compares the object identification information of the recognized object with the object identification information stored in the object association information database. When the object identification information is identified through the comparison, the virtual menu enhancer augments the virtual menu associated with the identified object identification information on the object. If the enhanced virtual menu is selected by the user, the controller processes a corresponding function.

Description

System and method for providing virtual menu associated with object {SYSTEM FOR SUPPLYING VIRTUAL MENU CONNETED WITH OBJECT AND METHOD THEREOF}

The present invention relates to a system and method for providing a virtual menu associated with an object, and more particularly, to an augmented reality technology having an association with a morphological and functional aspect of a real object.

Augmented Reality (AR) is a technology that superimposes a virtual image on the real world that the user sees, and is also called Mixed Reality (MR) because it shows the virtual world with additional information in the real world as a single image. . Such augmented reality is contrasted with virtual reality. In other words, VR technology generally allows users to immerse themselves in the virtual environment, while the user cannot see the real environment, whereas in AR technology, the user can see the real environment, and the real environment and the virtual objects are mixed. Takes shape.

Since augmented reality technology has a large amount of computational processing, it has generally been used in a system having a high performance specification. However, recently released mobile communication terminals are equipped with a camera as a basic specification, and can handle a large amount of computation. It is equipped with high specifications, and provides various services such as 3D rendering or camera image processing to users, and also provides virtual reality technology and augmented reality technology that were possible in high-performance desktops.

Currently, augmented reality technology is applied to handsets such as "hot code" and "visual code recognition system." First, "hot code" is a technology using a conventional bar code, the user simply illuminates the bar code with a terminal equipped with a camera (aka camera phone), the camera of the camera phone captures the bar code and the Internet with the corresponding content Augmented reality technology that moves to the space of the image. The "visual code recognition system" is an augmented reality technology for recognizing a barcode and augmenting information (ie, a virtual object such as text, voice, or image) stored in the storage unit of the terminal on the display of the terminal.

Existing augmented reality examples are mostly composed of a virtual object synthesized in a real image, or a virtual object shows a simple animation. For example, Korean Patent Laid-Open Publication No. 10-2012-0032336 "Mobile terminal and its display method of augmented reality information" in the preview image taken by the camera of the terminal, the AR information icon of each object is displayed superimposed on the object In this case, a technique of providing an AR information screen in which an AR information icon of a remote object is displayed in front of the preview image is disclosed.

In addition, at present, even if the user interacts with the augmented virtual object, the degree of association with the morphological and functional aspects of the real object reflected by the camera is inferior. For example, when a visual tag is illuminated with a camera, a 3D character comes out, and when the user touches the character, the character reacts, and the interaction is not related to the visual tag itself.

Korean Patent Publication No. 10-2012-0032336

An object of the present invention is to provide an augmented reality technology having an association with the morphological and functional aspects of the real object.

In addition, the present invention is designed so that the appearance of a real object (for example, a speaker, etc.) is designed to be very concisely designed mainly on the core function of the object, that is, an object without an operation unit or a generally inoperable object can be manipulated. It aims to do it.

In addition, the present invention stores information displayed on a business card or poster printed on paper in the user's mobile communication terminal, link to a web page of the URL, call a phone number written on the business card, send a text or email It aims to be able to perform such functions.

In order to achieve this object, a virtual menu providing system associated with an object according to the present invention includes an object association information database, a live image receiving unit, an object recognition unit, an object identification information comparing unit, a virtual menu enhancer, and a controller. The object association information database stores object identification information of the object and information defined to associate one or more virtual menus with the object identification information. The live image receiving unit receives a live image captured by a camera. The object recognizer recognizes an object included in the live image. The object identification information comparison unit compares the object identification information of the recognized object with the object identification information stored in the object association information database. When the object identification information is identified through the comparison, the virtual menu enhancer augments the virtual menu associated with the identified object identification information on the object. If the enhanced virtual menu is selected by the user, the controller processes a corresponding function.

As described above, according to the present invention, when the appearance of the real object is designed very concisely, mainly on the core function of the object, that is, a virtual menu associated with an object on an object without an operation unit or an object generally inoperable. By augmenting, there is an effect to perform the function. For example, the appearance of the speaker can be designed to have a natural function of emitting sound without an operation button such as volume control, and even this design can operate all functions of the speaker.

In addition, information displayed on a business card or poster printed on paper can be stored in the user's mobile terminal through an augmented virtual menu, linked to a web page of a corresponding URL, a telephone number written on the business card, a text or By providing a function such as sending an email, user convenience can be maximized.

In addition, the virtual guide is augmented in the live-action image, and the user can move the virtual guide onto the desired virtual menu by using the reference point, thereby making it easier to find and touch the virtual menu on the screen, thereby maximizing user satisfaction. Can be.

1 is a diagram illustrating a schematic configuration of a mobile communication terminal to which a system for providing a virtual menu associated with an object according to an embodiment of the present invention is applied.
2 is a diagram illustrating an example in which virtual menus associated with each object are augmented in a preview image of a camera.
3 illustrates various virtual menus that may be applied to an object.
4 illustrates an example in which virtual menus associated with a speaker are augmented in a preview image of a camera.
5 is a diagram illustrating an example in which a virtual guide is augmented in a preview image of a camera.
6 is a diagram illustrating a schematic flow of a method of providing a virtual menu associated with an object according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. Further, in the description of the embodiments of the present invention, specific values are merely examples, and exaggerated values may be presented for convenience and understanding of the present invention.

<Description of System>

1 is a diagram illustrating a schematic configuration of a mobile communication terminal to which a system for providing a virtual menu associated with an object according to an embodiment of the present invention is applied. The mobile communication terminal 20 may be implemented in various forms such as a mobile phone, a smart phone, a notebook computer, a digital broadcasting terminal, a PDA, and a PMP.

Referring to FIG. 1, the virtual menu providing system 100 associated with an object according to the present invention includes an object association information database 110, a live-action image receiving unit 120, an object recognition unit 130, and an object identification information comparing unit ( 140, the virtual menu enhancer 150, and the controller 160.

The object association information database 110 receives and stores object identification information of the object identified in the real world image received from the server 10 and information defining one or more virtual menus in association with the object identification information. The information refers to information defined to correspond to the virtual menu associated with each object identification information, the object is a real-life object identified by the image, the identification information of the object is information for identifying the object (for example, marker information, Or non-marker information such as feature point information), and a virtual menu (virtual button) are menus (buttons) that are augmented on an object, and a UI component (user interface component) capable of manipulating an object function or a function associated with the object. )to be.

The live image receiving unit 120 receives a live image of the real object that the user wants to see with the camera 21, and receives the live image captured by the camera 21.

The object recognizing unit 130 serves to find and recognize the location of the object included in the live-action image. The object recognizing unit 130 recognizes a business card, a movie poster, a food menu board, and the like, when it is difficult to immediately recognize an object. You can also use the resulting visual tag. The visual tag may be a rectangular black border used in augmented reality toolkit or an overlapping rectangle used in QRCode, and the present invention is not limited by such a configuration. In addition, the object recognition unit 130 tracks the position and posture of the object in the preview image as the user moves the camera 21, calculates the position and posture of the object, and then virtually displays it on the object. It is desirable to be able to place the menu in the correct position.

The object identification information comparing unit 140 compares the object identification information of the recognized object with the object identification information stored in the object association information database 110. That is, when the user operates the camera 21 of the mobile communication terminal 20, the virtual menu providing system 100 associated with the object extracts object identification information of the object from the input live image and extracts the extracted object identification information. Is compared with object identification information stored in the object association information database 110.

When the object identification information is identified through the comparison, the virtual menu enhancer 150 augments the virtual menu associated with the identified object identification information on the object. In other words, the virtual menu enhancer 150 augments and displays a predefined UI component at an object position in the live action image.

When the user selects the virtual menu augmented on the object, the controller 160 processes a corresponding function.

In one embodiment, when the object is an electronic device, the virtual menu providing system 100 associated with the object according to the present invention further includes a wireless communication unit 170 for wireless communication with the electronic device so as to control the operation of the electronic device. It is desirable to.

If the user illuminates the electronic device with the camera 21, the virtual menu enhancer 150 augments the UI component that can operate the electronic device in association with the electronic device on the electronic device, and the controller 160 controls the electronic device. When the augmented UI component is selected by the user, the corresponding function is processed by wireless communication with the electronic device. For example, when a user illuminates a speaker (there is no physically operable button on the speaker) with the camera 21, the virtual menu enhancer 150 operates to listen to music such as volume control and play control on the speaker surface. A button is virtually augmented and displayed. When a user presses an operation button, the controller 160 adjusts the volume and playback of music. In another application, when the electronic device is a digital door lock, the mechanical part of the door lock is designed to be hidden, and an application of opening and closing the door by providing a virtual door lock UI component only when the door itself is projected by the camera is also possible.

Basically, the UI component that can be applied to the virtual menu may be used in all of the UI components used in a general PC or smartphone, etc. The present invention is not limited thereto. However, since the virtual menu is synthesized and displayed on the object, a UI component having a strong feeling of direct manipulation may be applied more principally as shown in FIG. 3. In addition, such a UI component may be augmented to the speaker as shown in FIG. 4, and various modifications such as this may be applied to the present invention, and the present invention is not limited by the specific implementation method.

In another embodiment, when the object is a printed matter, when the user illuminates the printed matter with the camera 21, the virtual menu enhancer 150 augments the UI component associated with each information included in the printed matter on the printed matter, If the augmented UI component is selected by the user, the controller 160 processes a corresponding function. For example, as shown in (a) of FIG. 2, when a business card is illuminated by the camera 21, a virtual menu (button) having a call function on the phone number and an e-mail sending function is displayed on the business card. If the buttons are clicked on the screen of the mobile communication terminal 20, the corresponding function is actually operated. Alternatively, when the movie poster is illuminated by the camera 21 as shown in FIG. 2 (b), the virtual menu is connected to the information or ticketing site related to the movie on the movie poster, or by the camera 21 as shown in (c). When illuminating the food menu plate, augment a virtual menu for placing an order on the food menu plate or viewing information on recipes, ingredients, etc., and performing a corresponding function as the user selects the enhanced virtual menu.

According to this configuration, when the appearance of the real object is designed very concisely, mainly around the core function of the object, that is, by augmenting a virtual menu associated with the object on an object without an operation unit or a generally inoperable object, It has the effect of being able to perform a function. For example, the appearance of the speaker can be designed to have a natural function of emitting sound without an operation button such as volume control, and even this design can operate all functions of the speaker. In addition, the information displayed on a business card or poster printed on paper is stored in the user's mobile communication terminal 20 through an augmented virtual menu, connected to a web page of the corresponding URL, or made a call to a telephone number written on a business card. By providing a function such as sending a text message or an email, user convenience can be maximized.

Meanwhile, the general UI of a conventional PC or smartphone is a method in which a user directly clicks or touches a UI component on a screen, which is familiar and easy through a mouse on a PC, or a device itself in a mobile communication terminal such as a smartphone. If you do not use both hands, it is often difficult to find and touch UI components on the screen by lifting the device with one hand. That is, in the case of a software that manipulates a camera based on a preview image, such as augmented reality, it may be more difficult to find and touch a UI component on a screen because a task of targeting an object with a camera is required.

Accordingly, in the present invention, the virtual menu enhancer 150 augments a virtual guide that acts as a reference point on an object so that a virtual menu, that is, a UI component element can be selected by a user, and the position of the virtual guide is a camera ( 21, the virtual menu is selected so that the selection of the virtual menu is made when the virtual guide is positioned on the virtual menu or after a specific button is placed on the virtual menu. For example, as shown in FIG. 5, when the user wants to click the call button on the preview image of the camera 21, the camera 21 is operated by lowering the camera 21 downward so that the virtual guide in the crosshairs shape downwards. Make sure the virtual guide is above the button. Subsequently, the dialing button is operated by pressing a designated H / W button or a button arranged at a predetermined position on the screen.

In this way, the virtual guide is augmented in the live-action image, and the user can move the virtual guide onto the desired virtual menu using the reference point, thereby facilitating the task of finding and touching the virtual menu on the screen. Can be maximized.

<Description of Method>

A method of providing a virtual menu associated with an object according to an embodiment of the present invention will be described with reference to the flowchart shown in FIG. 6, but will be described with the order of convenience.

1. Storing information defining the virtual menu to be linked to the object identification information and the object identification information of the object <S610>

The object identification information and the object identification information of the object identified in the real-world image received from the server 10 are stored in the database of the mobile communication terminal 20. The information refers to information defined to correspond to the virtual menu associated with each object identification information, the object is a real-life object identified by the image, the identification information of the object is information for identifying the object (for example, marker information, Or non-marker information such as feature point information), and the virtual menu is a menu augmented on an object, and may operate a function of an object object or a function associated with the object.

2. Receiving the live image captured by the camera <S620>

When the user photographs the real image of the real object to be viewed by the camera, the real image captured by the camera is received.

3. Recognizing an object included in the live-action image <S630>

As a function of finding and recognizing the position of an object included in the live-action image, a business card, a poster, a menu board, and the like are recognized as shown in FIG. 2, and when a target is not immediately recognized, a visual tag that is easy to recognize may be used. It may be. The visual tag may be a rectangular black border used in augmented reality toolkit or an overlapping rectangle used in QRCode, and the present invention is not limited by such a configuration. In addition, the object recognition unit tracks when the position and posture of the object in the preview image changes as the user moves the camera, and calculates the position and posture of the object, thereby allowing the virtual menu to be placed on the correct position. It is preferable.

4. Comparing the object identification information of the recognized object with the object identification information stored in the database <S640>

The object identification information of the object recognized in the step S630 is compared with the object identification information stored in the database in the step S610. That is, when the user operates the camera of the mobile communication terminal 20, the virtual menu providing system 100 associated with the object extracts the object identification information of the object from the input live image, and extracts the extracted object identification information into the database. Compare with the stored object identification information.

5. Augmenting a virtual menu associated with the identified object identification information on the object <S650>

If the object identification information is identified through the comparison in step S640, the virtual menu associated with the identified object identification information is augmented on the object. In other words, a predefined UI component (User Interface Component) is augmented and displayed at the object position in the live image.

6. Step of processing the function corresponding to the selected virtual menu <S660>

If the augmented virtual menu is selected by the user in step S650, the corresponding function is processed.

In an embodiment, when the object is an electronic device, in step S650, a UI component capable of manipulating the electronic device in association with the electronic device is augmented on the electronic device. In step S660, the augmented UI component is provided to the user. If selected, the corresponding function is processed by wireless communication with the electronic device. For example, when a user illuminates a speaker with a camera (there is no physically operable button), a virtually augmented display of operation buttons for listening to music, such as volume control and play control, is displayed on the speaker surface. Control volume and playback of music via control buttons.

In another embodiment, when the object is a printed matter, when the user illuminates the printed matter with the camera, in step S650, the UI component associated with each information included in the printed matter is augmented on the printed matter, and in step S660, the enhanced UI is printed. When the component is selected by the user, the corresponding function is processed. For example, as shown in (a) of FIG. 2, when a business card is illuminated by a camera, a virtual menu (button) having a call function on the phone number and an e-mail sending function is displayed on the business card. If you click them on the screen of the mobile communication terminal 20, the actual corresponding function operates. Alternatively, when the movie poster is illuminated with a camera as shown in FIG. 2 (b), the augmented virtual menu leading to information related to a movie or a booking site is displayed on the movie poster, or when the food menu is illuminated with a camera as shown in (c). In addition, augment a virtual menu for placing an order on a food menu or viewing information such as recipes and ingredients, and performing a corresponding function as a user selects an enhanced virtual menu.

According to this configuration, when the appearance of the real object is designed very concisely, mainly around the core function of the object, that is, by augmenting a virtual menu associated with the object on an object without an operation unit or a generally inoperable object, It has the effect of being able to perform a function. For example, the appearance of the speaker can be designed to have a natural function of emitting sound without an operation button such as volume control, and even this design can operate all functions of the speaker. In addition, the information displayed on a business card or poster printed on paper is stored in the user's mobile communication terminal 20 through an augmented virtual menu, connected to a web page of the corresponding URL, or made a call to a telephone number written on a business card. By providing a function such as sending a text message or an email, user convenience can be maximized.

Meanwhile, the general UI of a conventional PC or smartphone is a method in which a user directly clicks or touches a UI component on a screen, which is familiar and easy through a mouse on a PC, or a device itself in a mobile communication terminal such as a smartphone. If you do not use both hands, it is often difficult to find and touch UI components on the screen with one hand. That is, in the case of a software that manipulates a camera based on a preview image, such as augmented reality, it may be more difficult to find and touch a UI component on a screen because a task of targeting an object with a camera is required.

Accordingly, in the present invention, in step S650, the virtual guide, that is, the UI guide element, may be augmented with a virtual guide serving as a reference point on the object so that the UI component element can be selected by the user, and the position of the virtual guide is moved according to the position of the camera. The virtual menu is selected through the input of a specific button when the virtual guide is positioned on the virtual menu or after being positioned on the virtual menu. For example, as shown in FIG. 5, when the user wants to click the call button on the preview image of the camera, the user moves the camera down so that the virtual guide in the crosshairs form a downward direction so that the virtual guide is placed on the call button. do. Subsequently, the dialing button is operated by pressing a designated H / W button or a button arranged at a predetermined position on the screen.

As described above, the virtual guide is augmented in the live image, and the user can move the virtual guide onto the desired virtual menu by using the reference point, thereby facilitating the task of finding and touching the virtual menu on the screen.

Meanwhile, the method for providing a virtual menu associated with an object according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The above description is merely illustrative of the technical idea of the present invention, and those skilled in the art to which the present invention pertains may make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas falling within the scope of the same shall be construed as falling within the scope of the present invention.

100: virtual menu providing system associated with the object
110: object association information database
120: live image receiving unit
130: object recognition unit
140: object identification information comparison unit
150: virtual menu enhancer
160:
170:

Claims (9)

An object association information database for storing object identification information of the object and information defining one or more virtual menus to be associated with the object identification information;
A live image receiving unit to receive a live image photographed from a camera;
An object recognition unit recognizing an object included in the live image;
An object identification information comparing unit comparing the recognized object identification information of the recognized object with the object identification information stored in the object association information database;
A virtual menu enhancer configured to augment a virtual menu associated with the identified object identification information on the object when the object identification information is identified through the comparison; And
If the augmented virtual menu is selected by the user, and comprises a control unit for processing a corresponding function
Virtual menu providing system associated with the object. The method of claim 1,
If the object is an electronic device, further comprises a wireless communication unit for wireless communication with the electronic device,
The virtual menu enhancer may be configured to augment a UI component (User Interface Component) on the electronic device to operate the electronic device in association with the electronic device.
If the augmented UI component is selected by the user, the control unit processes the corresponding function by wireless communication with the electronic device.
Virtual menu providing system associated with the object characterized in that. The method of claim 1,
If the object is a print,
The virtual menu enhancer enhances the UI component associated with each information included in the printed matter on the printed matter,
If the enhanced UI component is selected by the user, the controller is to process a corresponding function
Virtual menu providing system associated with the object characterized in that. The method of claim 1,
The virtual menu enhancer augments a virtual guide serving as a reference point on the object so that the virtual menu can be selected by the user,
The position of the virtual guide is moved according to the position of the camera,
The virtual menu is selected through the input of a specific button when the virtual guide is positioned on the virtual menu or after being positioned on the virtual menu.
Virtual menu providing system associated with the object characterized in that. (a) storing the object identification information of the object and information defining one or more virtual menus in association with the object identification information in a database;
(b) receiving a live image captured by a camera;
(c) recognizing an object included in the live-action image;
(d) comparing the object identification information of the recognized object with the object identification information stored in the database;
(e) augmenting a virtual menu associated with the identified object identification information on the object when the object identification information is identified through the comparison; And
(f) if the enhanced virtual menu is selected by the user, processing the corresponding function;
How to provide a virtual menu associated with an object. The method of claim 5,
If the object is an electronic device,
In step (e), a UI component (User Interface Component) capable of operating the electronic device in association with the electronic device is augmented on the electronic device,
In the step (f), if the augmented UI component is selected by the user, the corresponding function is processed by wireless communication with the electronic device.
Virtual menu providing method associated with the object characterized in that. The method of claim 5,
If the object is a print,
In the step (e), the UI component associated with each information included in the printed matter is augmented on the printed matter,
In step (f), if the augmented UI component is selected by the user, processing the corresponding function.
Virtual menu providing method associated with the object characterized in that. The method of claim 5,
In step (e), the virtual guide acting as a reference point on the object is augmented so that the virtual menu can be selected by the user,
The position of the virtual guide is moved according to the position of the camera,
The selection of the virtual menu in the step (f) is made through the input of a specific button when the virtual guide is located on the virtual menu or after being placed on the virtual menu.
Virtual menu providing method associated with the object characterized in that. A computer-readable recording medium having recorded thereon a program for executing the method according to any one of claims 5 to 8.

Images