WO2014009958A1

WO2014009958A1 - Segmenting and recognizing collectible items in images

Info

Publication number: WO2014009958A1
Application number: PCT/IL2013/050592
Authority: WO
Inventors: Ido Omer
Original assignee: Ido Omer
Priority date: 2012-07-13
Filing date: 2013-07-11
Publication date: 2014-01-16

Abstract

A method of extracting collectible items with their respective identification data from images thereof is provided herein. The method includes the following stages: comprising: capturing one or more images of on array of collectible items of a specified type; segmenting the one or more images into a plurality of segments, each segment containing one or more of the collectible items, wherein the segmenting is carried out based on contour data which is unique for the specified type of the collectible items; determining identification data for the collectible items in the segments by simultaneously applying to the segments: an image recognition decision function that is based on a knowledge base, and the contour data usable for further segmenting the segments into sub segments, each sub segment containing an image of a single collectible item; and extracting a set of single collectible item images, each with its corresponding identification data.

Description

SEGMENTING AND RECOGNIZING COLLECTIBLE ITEMS IN IMAGES

BACKGROUND

1. TECHNICAL FIELD

[0001] The present invention relates generally to image recognition techniques and more specifically to segmenting and recognizing objects.

2. DISCUSSION OF THE RELATED ART

[0002] Collectible items such as coins or stamps require cataloging and managing. Although traditionally the cataloging and managing of collections have been carried out manually by collectors (herein referred to as users), digitations has been used recent years in order to facilitate the classification and managing of such collections.

[0003] It would be therefore advantageous to provide a manner according to which a user may automatically or semi-automatically identify his or her plurality of collectible items and generate a digital catalog thereof.

BRIEF SUMMARY

[0004] One aspect of the invention provides a method of extracting collectible items with their respective identification data from images thereof. The method includes the following stages: comprising: capturing one or more images of on array of collectible items of a specified type; segmenting the one or more images into a plurality of segments, each segment containing one or more of the collectible items, wherein the segmenting is carried out based on contour data which is unique for the specified type of the collectible items; determining identification data for the collectible items in the segments by simultaneously applying to the segments: an image recognition decision function that is based on a knowledge base, and the contour data usable for further segmenting the segments into sub segments, each sub segment containing an image of a single collectible item; and extracting a set of single collectible item images, each with its corresponding identification data.

[0005] Other aspects of the invention may include a system arranged to execute the aforementioned method and a computer readable program configured to execute the aforementioned method. These, additional, and/or other aspects and/or advantages of the embodiments of the present invention are set forth in the detailed description which follows; possibly inferable from the detailed description; and/or learnable by practice of the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] For a better understanding of embodiments of the invention and to show how the same may be carried into effect, reference will now be made, purely by way of example, to the accompanying drawings in which like numerals designate corresponding elements or sections throughout.

[0007] In the accompanying drawings:

Figure 1 is a high level schematic block diagram illustrating the system according to some embodiments of the invention;

Figure 2 is another high level schematic block diagram illustrating the system according to other embodiments of the invention;

Figure 3 is a high level flowchart illustrating a method according to some embodiments of the invention; and

Figure 4 is a diagram illustrating an aspect according to some embodiments of the invention.

[0008] The drawings together with the following detailed description make apparent to those skilled in the art how the invention may be embodied in practice.

DETAILED DESCRIPTION

[0009] With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of the preferred embodiments of the present invention only, and are presented in the cause of providing what is believed to be the most useful and readily understood description of the principles and conceptual aspects of the invention. In this regard, no attempt is made to show structural details of the invention in more detail than is necessary for a fundamental understanding of the invention, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention may be embodied in practice.

[0010] Before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of the components set forth in the following description or illustrated in the drawings. The invention is applicable to other embodiments or of being practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.

[0011] Figure 1 is a high level schematic block diagram illustrating the system according to some embodiments of the invention. The system receives an image from a capturing device 100. Then, in one embodiment, the image may go through photometric calibration/correction component 201. The image may then go through an image segmentation component 202 that may have a user feedback/assistance component 401. The image/image segments then go through a feature extraction /digitization component 204. Digital signatures are then sent to the recognition engine 301, and compared against a catalog 302. Recognition results may get sent to a user feedback component 402 that may update both the catalog and the recognition engine. The recognition engine then updates the virtual representation of the collection 501.

[0012] More specifically, capturing device 100 may be an integral part of the system as in the case of a mobile device software application. In that case the software application may use the device's camera for capturing images. The system may also process images captured by an external capturing device such as a digital image or document scanner or a camera or a 3D capturing device. An optional calibration/correction component 201 may correct or enhance photometric properties such as noise, white balance, blurring, barrel effect, hallowing etc. It may also try to remove flash or inconsistent illumination artifacts, geometric artifacts due to perspective projection wrapping, and the like.

[0013] Optional segmentation component 202 may divide the image into one or more segments, each containing one or more items. The optional user feedback component 401 allows users to "help" the system by backing up the automatic module.

[0014] Digitization component 204 may extract one or more features for each segment and computes descriptors for those features. These descriptors are the digital representation of the segment (that contains one or more collectible items).

[0015] Recognition engine 301 may then compare the descriptors computed by 204 with a pre-computed catalogue of descriptors 302. A list of zero or more candidate catalog items is provided to the user. The recognition feedback component 402 allows the user to validate the recognition results. When the correct catalog items is not among the list of candidate matches, the user may either manually edit the item information, help the visual recognition engine by providing additional information such as keywords descriptors, or send the item as a query to a Mechanical Turk or a community of experts or semi-experts that may identify the item. The feedback may be fed back to the system and may update both the recognition engine and the catalog.

[0016] According to some embodiments of the present invention, the items are then fed to the virtual collection of the user 501, where he or she may maintain and assess the collection, share information with other collectors or trade with them. [0017] The system may be used for the digitization, assessment, management information sharing, virtual exhibitions and or trade. It may provide additional information such as metadata about the collectibles and suggest ways for improving the collection. It may help the user create different views of the collection.

[0018] In the context of the present invention, a collection may refer both to regular, long lasting collections and to ad hoc collections assembled for a specific occasion such as a sale catalog.

[0019] The system may be used to manage collection of two dimensional objects such as stamps, coins, bank notes or baseball cards, to name a few. It may be extended to the recognition of objects such as wine bottles, watches, porcelain sets and more. It may also be extended and used for recognizing different styles in items like furniture and art.

[0020] Figure 2 is another high level schematic block diagram illustrating the system according to other embodiments of the invention. The system illustrated herein implements the invention in client-server architecture. The system includes one or more user terminals 20A-20D associated with respective users/collectors lOA-lOD, wherein user terminals 20A- 20D are configured to capture one or more images 30 of on array of collectible items of a specified type. User terminals 20A-20D are in communication with an application server 110 via a network 40, wherein application server 110 is executed by one or more computer processors; (not shown here).

[0021] Application server 110 is connected to a segmentation module configured to segment the one or more images 30 into a plurality of segments 30", each segment containing one or more of the collectible items, wherein the segmenting is carried out based on contour data 150 which is unique for the specified type of the collectible items.

[0022] The system further includes an image recognition module 160 configured to: determine identification data for the collectible items in the segments by simultaneously applying to the segments: an image recognition decision function that is based on a knowledge base 140, and the contour data 150 usable for further segmenting the segments into sub segments, each sub segment containing an image of a single collectible item, image recognition module 160 is further configured to extract a set of single collectible item images, each with its corresponding identification data.

[0023] According to some embodiments of the present invention, the system further includes a photometric enhancement module 120 configured to apply photometric enhancement image processing algorithms to the captured images, to facilitate the segmenting and the determination of the identification data.

[0024] According to some embodiments of the present invention, the knowledge base 150 is stored on a networked database and is repeatedly updated by a community of human users 10A-10D.

[0025] According to some embodiments of the present invention, the recognition decision function receives as an at least one of its inputs, feedback from one or more human users. Thus the decision function applies well known crowd sourcing techniques for aiding the decision.

[0026] According to some embodiments of the present invention the contour data is repeatedly updated by a community of human users. Expert user may provide their metrics for better distinguishing between collectible item and the background.

[0027] According to some embodiments of the present invention, the contour data comprises a pattern or an edge indicative of a contour of the collectible items of a specified type.

[0028] According to some embodiments of the present invention, the system may further include a catalog generator, which may be implemented by application server 110, configured to generate a catalog based on the extracted set of single collectible item images, wherein the catalog presents the identified collectible items with its respective identification data according to a user defined classification or a predefined layout.

[0029] According to some embodiments of the present invention, the system may further include a sharing module, which may be implemented by application server 110, configured to share the generated catalog with a community of human users.

[0030] Figure 3 is a high level flowchart illustrating an aspect according to some embodiments of the invention. Method 300 may be implemented by any hardware architecture and is not limited to the aforementioned architectures of either Figure 1 or Figure 2. Method 300 starts off with the stage of capturing 310 one or more images of on array of collectible items of a specified type. Then the method proceed with the stage of segmenting 320 the one or more images into a plurality of segments, each segment containing one or more of the collectible items, wherein the segmenting is carried out based on contour data which is unique for the specified type of the collectible items. The method goes on to a stage of determining 330 identification data for the collectible items in the segments by simultaneously applying to the segments: an image recognition decision function that is based on a knowledge base, and the contour data usable for further segmenting the segments into sub segments, each sub segment containing an image of a single collectible item. Finally, the method proceeds to the stage of extracting a set of single collectible item images, each with its corresponding identification data.

[0031] Figure 4 is a diagram illustrating an aspect according to some embodiments of the invention. An image of coins 30 and an image of stamps 30A are shown. The images are shown after segmenting them into segments each having one or more coins 30' ' or stamps 30" A. The segments are carried out both based on type-specific contours and also based on the collectible items-specific knowledge base. Both may be user-aided implementing crowd sourcing techniques. Finally respective sets of identified single images of coins 162 and stamps 162A.

[0032] As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system." Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0033] Any combination of one or more computer readable medium(s) may be utilized. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0034] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire-line, optical fiber cable, RF, etc., or any suitable combination of the foregoing. [0035] Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0036] Aspects of the present invention are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0037] These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

[0038] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0039] The aforementioned flowchart and diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0040] In the above description, an embodiment is an example or implementation of the inventions. The various appearances of "one embodiment," "an embodiment" or "some embodiments" do not necessarily all refer to the same embodiments.

[0041] Although various features of the invention may be described in the context of a single embodiment, the features may also be provided separately or in any suitable combination. Conversely, although the invention may be described herein in the context of separate embodiments for clarity, the invention may also be implemented in a single embodiment.

[0042] Reference in the specification to "some embodiments", "an embodiment", "one embodiment" or "other embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments, but not necessarily all embodiments, of the inventions.

[0043] It is to be understood that the phraseology and terminology employed herein is not to be construed as limiting and are for descriptive purpose only.

[0044] The principles and uses of the teachings of the present invention may be better understood with reference to the accompanying description, figures and examples.

[0045] It is to be understood that the details set forth herein do not construe a limitation to an application of the invention.

[0046] Furthermore, it is to be understood that the invention can be carried out or practiced in various ways and that the invention can be implemented in embodiments other than the ones outlined in the description above.

[0047] It is to be understood that the terms "including", "comprising", "consisting" and grammatical variants thereof do not preclude the addition of one or more components, features, steps, or integers or groups thereof and that the terms are to be construed as specifying components, features, steps or integers.

[0048] If the specification or claims refer to "an additional" element, that does not preclude there being more than one of the additional element.

[0049] It is to be understood that where the claims or specification refer to "a" or "an" element, such reference is not be construed that there is only one of that element.

[0050] It is to be understood that where the specification states that a component, feature, structure, or characteristic "may", "might", "can" or "could" be included, that particular component, feature, structure, or characteristic is not required to be included.

[0051] Where applicable, although state diagrams, flow diagrams or both may be used to describe embodiments, the invention is not limited to those diagrams or to the corresponding descriptions. For example, flow need not move through each illustrated box or state, or in exactly the same order as illustrated and described.

[0052] Methods of the present invention may be implemented by performing or completing manually, automatically, or a combination thereof, selected steps or tasks.

[0053] The descriptions, examples, methods and materials presented in the claims and the specification are not to be construed as limiting but rather as illustrative only.

[0054] Meanings of technical and scientific terms used herein are to be commonly understood as by one of ordinary skill in the art to which the invention belongs, unless otherwise defined.

[0055] The present invention may be implemented in the testing or practice with methods and materials equivalent or similar to those described herein.

[0056] Any publications, including patents, patent applications and articles, referenced or mentioned in this specification are herein incorporated in their entirety into the specification, to the same extent as if each individual publication was specifically and individually indicated to be incorporated herein. In addition, citation or identification of any reference in the description of some embodiments of the invention shall not be construed as an admission that such reference is available as prior art to the present invention.

[0057] While the invention has been described with respect to a limited number of embodiments, these should not be construed as limitations on the scope of the invention, but rather as exemplifications of some of the preferred embodiments. Other possible variations, modifications, and applications are also within the scope of the invention. Accordingly, the scope of the invention should not be limited by what has thus far been described, but by the appended claims and their legal equivalents.

Claims

CLAIMS What is claimed is:

1. A method comprising:

capturing one or more images of on array of collectible items of a specified type; segmenting the one or more images into a plurality of segments, each segment containing one or more of the collectible items, wherein the segmenting is carried out based on contour data which is unique for the specified type of the collectible items;

determining identification data for the collectible items in the segments by simultaneously applying to the segments:

(i) an image recognition decision function that is based on a knowledge base, and

(ii) the contour data usable for further segmenting the segments into sub segments, each sub segment containing an image of a single collectible item; and extracting a set of single collectible item images, each with its corresponding identification data.

2. The method according to claim 1, further comprising applying photometric enhancement image processing algorithms to the captured images, to facilitate the segmenting and the determination of the identification data.

3. The method according to claim 1, wherein the knowledge base is repeatedly updated by a community of human users.

4. The method according to claim 1, wherein the recognition decision function receives as an at least one of its inputs, feedback from one or more human users.

5. The method according to claim 1, wherein the contour data is repeatedly updated by a community of human users.

6. The method according to claim 1, wherein the contour data comprises a pattern or an edge indicative of a contour of the collectible items of a specified type.

7. The method according to claim 1, further comprising generating a catalog based on the extracted set of single collectible item images, wherein the catalog presents the identified collectible items with its respective identification data according to a user defined classification or a predefined layout.

8. The method according to claim 6, further comprising sharing the generated catalog with a community of human users.

9. A system comprising:

one or more user terminals configured to capture one or more images of on array of collectible items of a specified type;

one or more computer processors;

a segmentation module configured to segment the one or more images into a plurality of segments, each segment containing one or more of the collectible items, wherein the segmenting is carried out based on contour data which is unique for the specified type of the collectible items; and

an image recognition module configured to:

(i) determine identification data for the collectible items in the segments by simultaneously applying to the segments:

(1) an image recognition decision function that is based on a knowledge base,

(2) the contour data usable for further segmenting the segments into sub segments, each sub segment containing an image of a single collectible item; and

(ii) extract a set of single collectible item images, each with its corresponding identification data,

wherein the segmentation module and the image recognition module are executed by the computer processor.

10. The system according to claim 9, further comprising a photometric enhancement module configured to apply photometric enhancement image processing algorithms to the captured images, to facilitate the segmenting and the determination of the identification data.

11. The system according to claim 9, wherein the knowledge base is repeatedly updated by a community of human users.

12. The system according to claim 9, wherein the recognition decision function receives as an at least one of its inputs, feedback from one or more human users.

13. The system according to claim 9, wherein the contour data is repeatedly updated by a community of human users.

14. The system according to claim 9, wherein the contour data comprises a pattern or an edge indicative of a contour of^" the collectible items of a specified type.

15. The system according to claim 9, further comprising a catalog generator configured to generate a catalog based on the extracted set of single collectible item images, wherein the catalog presents the identified collectible items with its respective identification data according to a user defined classification or a predefined layout.

16. The system according to claim 15, further comprising a sharing module configured to share the generated catalog with a community of human users.

17. The system according to claim 15, wherein the system is implemented on a communication device.

18. The system according to claim 15, wherein the system is implemented over a computer network.

19. A computer program product comprising:

a computer readable storage medium having computer readable program embodied therewith, the computer readable program comprising:

computer readable program configured to capture one or more images of on array of collectible items of a specified type;

computer readable program configured to segment the one or more images into a plurality of segments, each segment containing one or more of the collectible items, wherein the segmenting is carried out based on contour data which is unique for the specified type of the collectible items; and

computer readable program configured to determine identification data for the collectible items in the segments by simultaneously applying to the segments: an image recognition decision function that is based on a knowledge base, the contour data usable for further segmenting the segments into sub segments, each sub segment containing an image of a single collectible item; and computer readable program configured to extract a set of single collectible item I, each with its corresponding identification data.