KR102024818B1 - Method, apparatus and computer program for providing search information from video - Google Patents

Abstract

The present invention relates to a method, an apparatus and a computer program for providing image retrieval information. The present invention provides a method for providing image retrieval information, the present invention for achieving this object is that when a predetermined query generation event occurs during the playback of any image, the client queries the screen when the query generation event occurs Generating and transmitting to a server, the server extracting a region including image feature information previously learned as a specific product in the query as a search object, and the server using a plurality of category image products Identifying classification information of the object to be searched using a trained model, retrieving at least one product related information corresponding to the classification information from a product database, and the server transmitting the search result to the client; The client may display the search result on the screen. And in one aspect to hereinafter. According to the present invention, a product similar to a product exposed to an image can be searched without a separate search word input.

Description

METHOD, APPARATUS AND COMPUTER PROGRAM FOR PROVIDING SEARCH INFORMATION FROM VIDEO}

The present invention relates to a method, an apparatus, and a computer program for providing image search information in an image reproducing apparatus, and more particularly, to a method of providing a similar product information by recognizing an object in an image without inputting a separate search word. It is about.

With the recent increase in wired and wireless Internet environments, commerce such as promotions and sales using online has been activated. In this regard, buyers search for magazines, blogs or YouTube videos on desktops or mobile terminals connected to the Internet, and when they find a product they like, they search for a brand name and lead to a purchase. For example, the name of a bag that a famous actress heard at the airport, and the name of a childcare item from an entertainment program, are ranked on the portal's real-time search term ranking. At this time, however, the user must separately open a web page for searching to search for a product name and a sales company, etc., and if a user does not know a name, a user may not be able to search easily.

On the other hand, sellers often sponsor products to video contents such as dramas and movies to promote their products, and most of them do not directly expose brand names due to indirect advertising issues despite promotional expenses.

Therefore, in recent years, sellers have been searching for various promotional methods such as producing video contents tagged with promotional product information in a video frame so that the user can check the product information in the video. However, according to this method, the user can check only the limited information provided by the seller (the product information tagged by the seller), and if the product in the video is not a sponsored product, the information about the product is still searched for. Have difficulty. In addition, when a product exposed to a video is not a price that can be purchased, consumers may want to search for a similar product with a lower price. When a product is identified through a video, it is difficult to identify a brand name, which makes it difficult to search for similar products. have.

One object of the present invention is to provide a method, an apparatus, and a program for providing image retrieval information capable of searching for a product similar to a product exposed to an image without inputting a separate search word.

In addition, the present invention can be exposed to the product information when the original video playback without producing a separate content tagged product information to reduce the cost of product promotion and provide image search information that can promote the product more effectively It is another object to provide a method and apparatus.

According to the present invention for achieving the above object, if a predetermined query generation event occurs while playing a video, the client generates a screen when the query generation event occurs as a query and transmits it to the server. Extracting a region including image feature information previously learned as a specific product from a query as a search object, wherein the server classifies the search object using a model trained using a plurality of category product images; Searching for one or more product related information corresponding to the classification information in a product database, the server transmitting the search result to the client, and the client displaying the search result on the screen. It is characterized by including one.

According to the present invention as described above, it is possible to check the merchandise and merchandise related information similar to the merchandise exposed to the image without a separate search word input.

In addition, according to the present invention, it is possible to reduce the cost of product promotion and more effectively promote the product by allowing the product information to be exposed when playing the image without producing a separate content tagged with the product information.

1 is a conceptual diagram illustrating an embodiment of an electronic device for providing image search information according to the present invention;
2 is a conceptual diagram illustrating an embodiment of a server that provides image search information to a client according to the present invention;
3 is a flowchart illustrating an embodiment of a method for providing image search information of an electronic device according to the present invention;
4 is a flowchart illustrating an embodiment of a query generation method according to the present invention;
5 is a flowchart illustrating an embodiment of an object extraction method according to the present invention;
6 is a flowchart illustrating an embodiment of an object extraction method according to the present invention;
7 is a flowchart illustrating an embodiment of a method for providing image search information of a server according to the present invention;
8 is a view for explaining an embodiment of providing image search information of an electronic device according to the present invention.

The above objects, features, and advantages will be described in detail with reference to the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. In describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings are used to indicate the same or similar components, all combinations described in the specification and claims may be combined in any way. And unless specified otherwise, reference to the singular may include one or more, and reference to the singular may also include the plural expression.

In the present specification, 'object' may be understood to mean an object that can be recognized as a product, or an image of the object, and may be an image of an unspecified object including noise according to object extraction conditions, or may be applied to a predefined label. It may also be an image of a corresponding specific object. 'Object' may correspond to a comprehensive category of products depending on the extraction conditions and the classification depth of the learned label, but it should be understood as a concept that can correspond to detailed product characteristics such as color, pattern, and product name of the product. .

'Label' described herein is the information defining the characteristics of the goods (or objects) as described above, for example, categorical information such as women's clothing, men's clothing, furniture, household goods, kitchen appliances, bottoms, tops, Subclassification information such as dress, color, pattern, brand, and price may include subclassification information. This is merely an example, and labels may be variously classified or defined according to user settings. The label according to an embodiment of the present invention may be learned through machine learning, and a label corresponding to image feature information may be stored in advance in a server or a client (electronic device).

In the present specification, the 'electronic device' includes all types of electronic devices capable of playing an image and displaying search information such as a desktop, a smart phone, a tablet PC, and the like in the present specification. It can be understood to have a meaning.

1 is a conceptual diagram illustrating an embodiment of an electronic device that provides image search information according to the present invention. Referring to FIG. 1, the electronic device 100 according to the present invention may include a controller 130, a communication unit 150, and a display unit 170, and may further display an input unit 190 (not shown).

The controller 130 generates a query using the image when a preset query generation event occurs during image playback.

The controller 130 may determine that a query generation event occurs when a preset time elapses or a search event occurs after a pause event occurs during image playback. For example, if the user pauses the image being played and continues to pause even after a predetermined time (2 seconds) has elapsed, the electronic device 100 determines that the query generation event has occurred and based on the paused image frame. You can create a query with

The search event refers to an event that can recognize that the electronic device 100 searches for a product. For example, a user clicks or touches a search button displayed after a pause, and designates an area of an image frame. And clicking or touching one region or a button recognized as an object in the image frame.

In addition, if a pause event occurs during video playback, the controller 130 displays the remaining time until the generation of the query on the screen, and if there is no user input until the remaining time reaches 0, the query generation event occurs. It can be judged that.

The query generated by the controller 130 when a query generation event occurs is a plurality of image frames satisfying a preset condition based on a paused image frame, an image of one region included in the image frame, or a paused screen, or It may include at least one of object information extracted from one image frame included in the image. The object information may include at least one of image feature information extracted from an image, image feature information extracted from an arbitrary object region, image feature information or label information learned to correspond to a specific object.

Here, the image feature information refers to information extracted locally (local) or globally (global) from an image, an image frame, or an image of one region included in the image. The image feature information can be used for machine learning for label generation. In addition, when a pre-learned label exists, any image feature information may correspond to a specific label or a specific object. The image feature information may include vector information (descriptor) of a predetermined range of images (image fragments or patches) located around the entire image or an object area or a feature point in the image, and the vector information may include brightness variation, color, and pattern ( Frequency, vertical edge, horizontal edge distribution), the tilt of the image fragments, and may include information about the change in brightness over a predetermined threshold. In addition, the image feature information may not only have a unique range of values for each object, but may also have a unique range of values for each label according to a learning model.

An image of one region included in the image frame may be designated as a polygon (B) as shown in FIG. 8. When the pause event occurs, the controller 130 may provide a guide including a region recognized as an object for the user's convenience, by adjusting the size or shape of the polygon provided as the guide. Finally, you can select the area to be included in the query.

In addition, the controller 130 may set a preset condition based on the paused image frame when the paused image frame is blurred, when it is difficult to detect an object, or when the object extraction accuracy is increased in the server. One or more satisfying image frames may be included in the query.

The preset condition for selecting one or more image frames included in the query may be set by using time, the number of frames, whether a caption is present, and / or the sharpness. For example, the controller 130 may generate, as a query, image frames within 1 second before and after the paused image frame based on a preset condition. In addition, the controller 130 may add three front and rear image frames one by one based on the paused image frame and generate three image frames as one query. As another embodiment, when the sharpness of the paused image frame is inferior, the controller 130 may search for the high-definition image frame among the before / after image frame and include it in the query. As another example, when the caption is included in the paused image, the controller 130 may include only the image frame from which the caption is removed in the query.

In addition, the controller 130 may include an object extractor 135 that extracts an object from an image frame, and a query generator 137 that generates a query including the object.

The object extractor 135 extracts a region including image feature information learned to correspond to a specific object as an object by using a previously stored learning model, and attaches a label corresponding to the image feature information to the object. Can be. The object in the present embodiment means an object that can be defined as a specific thing, and can be variously defined according to its label or category.

The image feature information learned to correspond to a specific object using a previously stored learning model may be a global descriptor or a local descriptor according to its setting. That is, the object extractor 135 may extract an object including image feature information learned to correspond to a car from an image frame, or may extract an object including image feature information learned to correspond to a garment. Not only what the object is, but also its characteristics (product type, brand of product, color, material, form, pattern, differences in some designs, and feelings that can be defined by people-cute, chic, vintage, etc.) You may. Therefore, using global and local descriptors to suit the purpose, it is possible to label more specific product names, product features.

In another embodiment, the object extractor 135 extracts image feature information from an image frame, partitions one region in the image frame according to a preset condition using the extracted feature information, and uses the divided region as a search object. Can be set. Here, the object extracted by the electronic device according to the present invention may be determined as an arbitrary 'object' regardless of the type of the object, and thus may include noise, but may be extracted at a high speed. The division of the region may be made by using image feature information in an image frame such as a brightness change point, an overall color distribution, edge information, and sharpness information, and an object may be set using coordinate information on the image frame. That is, the object extractor 135 partitions a region where a change in brightness, color distribution, edge information, or sharpness information is greater than or equal to a preset reference value on the screen when a query generation event occurs, and searches for a partitioned region. It can be set as an object and created as a query. A plurality of objects may be included in the image frame, in which case the plurality of objects will be extracted.

When the display unit 170 displays the search result, the controller 130 preferentially displays a search result for an object including a label that is the same as or similar to the object previously searched by the user using the search log of the user. Can be controlled. This is to increase the probability that image search leads to product purchase and increase user satisfaction by first exposing the search results belonging to the user's interest field on the screen. In addition, the controller 130 may increase the accuracy of the search by controlling the display unit 170 to display the search results on the screen in the order of similarity with the image feature information of the object included in the query.

The communicator 150 transmits a query to the server 200 and receives a search result corresponding to the query from the server 200. The communication unit 150 may include a wired / wireless communication module for communicating with the server 200, for example, a mobile communication module supporting GSM, CDMA, WCDMA, HSDPA, and LTE schemes. In addition, the communication unit 190 may include a wireless communication module such as WLAN, WiFi Direct, DLNA, Wibro, Wimax, a communication module for short range communication, for example, Bluetooth, RFID, infrared communication, UWB It may include a communication module supporting ZigBee, NFC, Wi-Fi, Wi-Fi Direct. For reference, the communication unit 150 of the present invention is not limited by the method as long as it is a communication module supporting data transmission and reception with the server 200.

The display unit 170 displays the received search result on the screen, where the search result refers to a search result for any object included in the image. The display unit displays (outputs) information processed by the electronic device 100, and displays execution screen information of an application program driven by the electronic device 100, or a UI (User Interface) or GUI (Graphic User) according to the execution screen information. Interface) information can be displayed.

The display unit 170 may include a display module, and the display module may include a touch screen that forms a layer structure or is integrally formed with the touch sensor. The touch screen may function as an input unit 190 (not shown) that provides an input interface between the electronic device 100 and the user, and may provide an interface for displaying information between the electronic device 100 and the user. . In addition, the display module may be a stereoscopic display module for displaying a stereoscopic image. As the stereoscopic display, a three-dimensional display method such as a stereoscopic method (glasses method), an auto stereoscopic method (glasses type method), a projection method (holographic method), or the like may be applied. The display unit 170 of the present invention is not limited by the type of the display module, and can display a moving image by any type of image reproducing program capable of reproducing an image and display a search result for a query. It includes everything.

delete

2 is a conceptual diagram illustrating an embodiment of a server that provides image search information to a client according to the present invention.

Referring to FIG. 2, the server 200 according to an exemplary embodiment may include a controller 230, a communication unit 250, and a database unit 270.

The controller 230 analyzes the query received from the client 100, searches for the product information corresponding to the label of the object included in the query or the image feature information of the object in the database unit 270, and provides the search result as a search result. . The product information or the product related information is information related to the product, and includes a product name, a manufacturer, a price, a popularity, a category (label), a purchase place information (website, contact information, location information), a related review, an advertisement, and the like.

If the query includes image feature information classified as noise or label information learned with noise, the controller 230 may exclude the corresponding query or object from the search target.

In addition, when the query analysis result, the image frame, an image of a region included in the image frame, or a first object without a label is included, the controller 230 may extract image feature information from the image frame, the image, or the first object. Can be extracted. Here, the first object may be an object without a label, and may be an object recognized by the electronic device as an object regardless of the type of product, and the image may be an image of one region captured or cropped by the electronic device.

The controller 230 includes the image feature information in the image frame, the image, or the first object when the image feature information previously learned as a specific object exists in the image frame, the image, or the first object. One area to be set as a search object and a label corresponding to the image characteristic information may be attached to the search object.

The controller 230 searches for a product having the same / similar label or the same / similar image feature information in the database unit 270 using the label or the image feature information of the search object.

As a result of the query analysis by the controller 230, if the query includes a labeled search target object, the controller 230 may store product information corresponding to a label attached to the search target object or image feature information of the search target object. Search in section 270. The communication unit 250 controls the transmission of the search result to the client.

When the communication unit 250 receives the user's identification information and the search log of the user from the client, the control unit 230 uses the identification information and the search log received from the client product information corresponding to the label included in the search log May be provided to the user through the communication unit 250. More specifically, the controller 230 may assign higher importance so that product information corresponding to a label included in the search log is exposed in the client more preferentially.

The communication unit 250 receives a query generated using the image from the client, and transmits a search result for the query to the client 100. In addition, the communication unit 250 may receive both the user's identification information and the user's search log from the client. An example of a communication module included in the communication unit 250 is the same as the communication module included in the communication unit 150 described above, and thus will be omitted.

The database unit 170 generates and stores a product database including image feature information of a product and a label corresponding to the feature information of the product image for any product.

Hereinafter, an embodiment of a method for providing image search information according to the present invention will be described with reference to FIGS. 3 to 6.

First, referring to FIG. 3, according to the method for providing image retrieval information according to the present invention, when a predetermined query generation event occurs during image playback (S100), the electronic device generates a query using the image (S200). Send to the server (S300). The query generated in step S200 is extracted from a paused image frame, a region image included in the image frame, a plurality of image frames satisfying a preset condition based on the paused screen, or one image frame included in the image. It may include at least one of the object information. The object information may include at least one of image feature information extracted from an image, image feature information extracted from an arbitrary object region, image feature information or label information learned to correspond to a specific object.

Next, the electronic device receives the search result corresponding to the query from the server (S400), and displays the received search result on the screen (S500) to provide image search information. In operation S500 of displaying the received search result on the screen, the electronic device preferentially displays a search result for an object including a label that is the same as or similar to the object previously searched by the user using the search log of the user. Can be. This is because the electronic device preferentially exposes the search results belonging to the user's field of interest on the screen to increase the probability of image search leading to product purchase and to increase user satisfaction. In operation S500, the electronic device may display the search results on the screen in order of high similarity to the image feature information of the object included in the query, thereby increasing the accuracy of the search.

The query generation step S200 using the above-described image may be performed by extracting an object from an image frame (S230) and generating a query including the object (S250) as shown in FIG. 4.

In addition, in the query generation step (S200), after the pause event occurs during image playback, the electronic device may determine that the query generation event occurs when a preset time elapses or a search event occurs.

In addition, in the query generation step (S200), when a pause event occurs during image playback, the electronic device of the present invention displays the remaining time until the generation of the query on the screen, and the user input until the remaining time reaches zero. If not, it can be determined that a query generation event has occurred.

Referring to FIG. 5, in operation S230, the electronic device extracts image feature information from an image frame (S232), and partitions a region within an image frame according to a preset condition using the extracted image feature information (S232). In operation S234, the partitioned area may be set as a search target object in operation S236.

The division of the region in step S234 may be performed using feature information in an image frame such as brightness change point, overall color distribution, edge information, and sharpness information. The setting of the search object in step S236 may include coordinate information on the image frame. It can be made using. A plurality of objects may be included in the image frame, in which case the plurality of objects will be extracted.

In another embodiment of the object extraction (S230), the object extraction (S230) may search for a region including image feature information learned as corresponding to a specific object by using a previously stored learning model as illustrated in FIG. 6. Extraction may be performed (S233), and a label corresponding to the image feature information may be attached to the search object (S237).

7 is a flowchart illustrating an embodiment of a method for providing image search information of a server according to the present invention. Referring to FIG. 7, the server of the present invention generates a product database including image feature information of a product, a label (classification information) corresponding to the feature information of the product image, and product related information.

When the server receives the query generated using the image from the client (S1000), it analyzes the query (S2000), determines whether the query includes image feature information learned with noise (S2500), and if the query is included Can be excluded from search.

As a result of the query analysis (S2000), if the query includes an image frame Case1, an image (not shown) of one region included in the image frame, or a first object Case2 that is not labeled, the server may include an image frame, The image characteristic information previously learned from the image or the first object to the specific object is searched for (S3000). As a result of the search in operation S3000, an area including the image feature information is extracted from the image frame, the image, or the first object as the search target object in operation S3500.

Next, the server attaches a label corresponding to the image feature information to the search object (S3700). That is, the server identifies the label (classification information) of the object to be searched. As described above, the label is generated by learning a feature of a product through machine learning, and thus, the server uses a plurality of product images for each classification information. It can be understood to identify what classification information the searched object has using the trained model. The server searches for product related information having the same / similar label (classification information) or the same / similar image feature information in the product database using the label (classification information) or the image feature information of the search object (S4000).

As a result of the query analysis (S2000), if the query includes a searched object with a label (Case 3), the server displays product information corresponding to a label attached to the searched object or image feature information of the searched object. Search in (S4000). The search result is transmitted to the client (S5000).

Although not shown in the drawing, the server may receive the user's identification information and the user's search log from the client. In this case, in operation S5000, the server may provide the user with product information corresponding to a label included in the search log by using the identification information received from the client and the search log. The server may assign higher importance to the product information and expose it to the client and transmit it to the client.

8 is a view for explaining an embodiment of a method for providing image search information of an electronic device according to the present invention. In FIG. 8, a user plays an image using the electronic device 100. In the image, the woman is wearing a t-shirt A. If the user pauses the image in order to search for product information on the t-shirt A, it is an embodiment of the present invention. When a pause event occurs and there is no user input for two seconds, a query generation event occurs by a preset algorithm.

The electronic device 100 generates a query using the paused image frame according to the occurrence of the query generation event. The electronic device 100 may transmit the image frame itself to the server, or may transmit an image of one region included in the image frame to the server. In addition, an object may be extracted from one image frame included in the image, and the extracted object information may be transmitted to the server.

Suppose a rectangular guide B is displayed so that a user can specify an area when a pause event occurs. The user may select the T-shirt A by adjusting the size and shape of the guide B, or move the position of the guide B, so that the electronic device 100 does not recognize it as an object, but the table C is of interest to the user. ) Can also be specified as an object.

In an embodiment in which the electronic device 100 recognizes a user's click or touch operation as a query generation event, the server 200 (not shown) performs a search using the guide area image B. FIG. When the server 200 receives a query including an image without an additional label from the electronic device 100, the server 200 searches whether the image includes image feature information previously learned as a specific object. If the image has image feature information learned with noise, the server 200 may stop searching for the image, but since the image includes the T-shirt A, the image 200 also includes the image feature information learned with the T-shirt.

When the server 200 finds the image feature information (descriptor corresponding to the T-shirt) learned by the T-shirt, the server 200 sets a region including the corresponding image feature information as the search object. The search object set as described above has a form closest to the T-shirt A, and may be a region in which an area (background), which is not related to a product, is excluded from the image B initially received.

Next, the server 200 may attach labels such as women's clothing (large classification), tops (medium classification), blue, and t-shirts (small classification) to the search object. In addition, a product corresponding to a blue T-shirt among women's tops may be searched in the product database, and a product including similar image feature information may be searched for in the product database using the extracted image feature information (image search). As a result of the search, if the same and / or similar products 20A, 10A, 10B, 10C, and 10D are found, the server 200 transmits the searched identical and / or similar product information to the electronic device 100.

The electronic device 100 receives a search result from the server 200 and displays the search result on the screen. When the electronic device 100 displays the search result on the screen, the electronic device 100 may display product information on an area 10 of the screen, Product information can also be displayed in the area | region B or 20 near the object A which were designated. The product information may be displayed in the descending or ascending order according to the similarity or interest, price, popularity, etc. according to the search log of the user. For example, 10A with the highest similarity may be displayed at the top in order of most similarity, and then 10B and 10C with the highest similarity.

In addition, the electronic device may display a product name 20B, a manufacturer, a price 20C, a popularity, a category (label), purchase information (website, contact information, location information), and related reviews together with the product image 20A. You can also click or touch the information to run another application.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by.

100: electronic device, client
200: server

Claims (7)

If a pause event occurs during a predetermined video playback and a predetermined time elapses, the client determines that the query generation event has occurred and generates a video frame when the query generation event occurs as a query and transmits it to the server;
Extracting, by the server, a region including image feature information previously learned with a specific product in the query as a search object;
Identifying, by the server, classification information of the object to be searched by using a machine learning model trained using a plurality of product images for each classification information, and searching one or more product related information corresponding to the classification information in a product database. ;
The server sending the search results to the client;
And displaying, by the client, a search result on a screen.
The method of claim 1,
The client generating the image frame as a query and transmitting to the server
Extracting at least one product image from the image frame;
And generating the product image as a query and transmitting the query image to the server.
The method of claim 1,
The client generating the image frame as a query and transmitting to the server
Partitioning a region in which the change in brightness, color distribution, edge information, or sharpness information in the image frame is greater than or equal to a preset reference value;
And generating the one area as a query and transmitting the same to the server.
The method of claim 1,
The client generating the image frame as a query and transmitting to the server
And generating and transmitting to the server an image frame having a predetermined time or a predetermined number of image frames before and after the image frame based on the image frame.
The method of claim 1,
The step of displaying the search results on the screen by the client
And displaying, on the screen, a search result for products having the same or similar classification information as the product previously searched by the user by using the user's search log.
An image retrieval information providing program stored in a computer readable medium for executing the method of any one of claims 1 to 5.
delete

Images