TWI894057B - Intelligent goods tracking system - Google Patents

Abstract

The present disclosure provides an intelligent goods tracking system comprising an imaging device and a processing terminal. The imaging device is installed around shelves and captures real-time dynamic images via a shooting module, transmitting the images to the processing terminal. The processing terminal applies deep learning techniques for object detection and image segmentation on the real-time dynamic images to extract features of shelves, goods, and shopping baskets. Using polygonal framing technology, the system marks sensing area frames and product bounding boxes, as well as marks a central point within each product bounding box. When a product is moved, the processing terminal records the movement trajectory of its central point. If the product bounding box overlaps with the sensing area frame and the movement trajectory reaches a purchase distance, a purchase signal is generated. This system enables automatic tracking of goods and detection of purchasing behavior, thereby effectively improving mall management efficiency.

Description

Smart Product Tracking System

This disclosure relates to the field of merchandise management-related technologies, particularly an intelligent product tracking system used in retail stores to intelligently track the status of merchandise.

Today's advances in information and technology have gradually shifted the focus of the retail service industry toward automation, ushering in a period of rapid development. This has led to a more frequent use of artificial intelligence technology by the public. As the scale of artificial intelligence continues to expand, businesses are also pursuing the goals of low error rates, low labor costs, and high efficiency.

While existing technologies can leverage deep learning to improve product recognition accuracy, they still face limitations. For example, AI technology requires high-performance imaging equipment and processors, increasing hardware costs. Furthermore, AI technology is highly sensitive to environmental variables such as lighting and viewing angle during the recognition process, potentially leading to misjudgments or missed detections.

In existing retail environments, tracking customer interactions with products remains challenging. This is especially true when items are removed, moved, or placed in a shopping basket, making it difficult for the system to instantly and accurately determine whether an item has been purchased. Furthermore, existing automated recognition technology struggles to cope with overlapping, obscured, or misplaced items, resulting in reduced recognition accuracy.

To address the aforementioned issues, this disclosure provides an intelligent product tracking system that uses a camera to capture real-time dynamic images and a processing terminal to capture the product's movement status from these images. This system can automatically track products and detect purchasing behavior, effectively improving mall management efficiency.

One embodiment of the present disclosure provides an intelligent product tracking system, which includes: a camera device, which is mounted around a shelf, the camera device having a shooting module and a transmission module coupled to each other, the shooting module can shoot dynamic images of the products on the shelf and the surrounding area of the shelf to generate a real-time dynamic image, and the transmission module transmits the real-time dynamic image to the outside; and a processing terminal, which is coupled to the camera device, the processing terminal having a mutual coupling. A database, a communication module and a computing module are coupled. The database stores a plurality of shelf models, a plurality of product models and a plurality of shopping basket models. The communication module is coupled to the transmission module of the camera device and receives real-time dynamic images. The communication module transmits the real-time dynamic images to the computing module. The computing module performs object detection and image segmentation on the real-time dynamic images through a deep learning algorithm and compares the real-time dynamic images with the shelf models, the product models and the shopping basket models in the database. The product model and each shopping basket model are used to extract a shelf feature, a product feature and a shopping basket feature contained in the real-time dynamic image. The computing module uses a region annotation technology to select a sensing area frame around the shelf feature and between the shopping basket features in a polygonal frame selection method, and marks a product frame around the product feature. The computing module analyzes and processes the product frame to generate a product center point. When the product on the shelf is moved, the product frame corresponding to the product and the center point of the product will move continuously. During the process of the product being moved, the computing module will record the center points of the product generated by the continuous movement and connect them in series to generate a movement trajectory; when the product frame overlaps with the sensing area frame to a set range, and the length of the movement trajectory reaches a purchase length, the computing module determines that the product has been purchased to generate a purchase signal.

In one embodiment, the processing terminal further includes a recording module coupled to the computing module. The computing module transmits a purchase signal along with product characteristics to the recording module. The recording module accumulates the purchase signals corresponding to the product characteristics to generate a single product purchase quantity.

In one embodiment, a communication module is coupled to a recording module and an inventory management system. The recording module obtains inventory quantities of different products from the inventory management system through the communication module and sets a unique notification threshold for each product. When the recording module determines that the purchase quantity of a single product has reached the notification threshold, the recording module generates a replenishment signal.

In one embodiment, the communication module is coupled to a social communication platform; the communication module receives the replenishment signal and product characteristics transmitted by the recording module, and the communication module transmits the replenishment signal and product characteristics to the message interface of the social communication platform through a service dissemination technology.

In one embodiment, the processing terminal has a display module, which is coupled to the communication module and the computing module. The display module can present the real-time dynamic images received by the communication module, and can also present the sensing area frame, product frame, product center point and movement trajectory generated by the computing module in combination with the real-time dynamic images.

In one embodiment, the range is set to an overlap ratio of the product frame and the sensing area frame exceeding 80%.

In one embodiment, the purchase length is the shelf length; the length of the moving track is the distance between the center point of the first product and the center point of the currently generated product.

In one embodiment, as the product is moved, the overlap between the product frame and the sensing area frame gradually increases, and the movement trajectory is a single movement direction. The computing module determines that the product has been purchased.

In one embodiment, each shelf model, each product model, and each shopping basket model are generated from real-time dynamic images generated by a camera module. The computing module uses deep learning algorithms and processes the images through color, brightness, cropping, blurring, noise, scaling, stretching, and rotation to augment the shelf features, product features, and shopping basket features within the real-time dynamic images. Alternatively, each shelf model, each product model, and each shopping basket model are collected and generated using a web crawler.

In one embodiment, the deep learning algorithm technology is the YOLO V8 deep learning model; the region annotation technology is the Roboflow PolygonZone polygon region annotation technology.

Through the above, the present disclosure utilizes a camera and a computing module to detect the movement of goods in real time. This system generates a movement trajectory of the center point of the goods as they are moved, accurately tracking whether the goods have been removed. Furthermore, the system can double-check whether the goods have been purchased by the customer based on the degree of overlap between the outer frame of the goods and the sensing area frame, combined with the length of the movement trajectory. When these conditions are met, a purchase signal is automatically generated, thereby accurately determining the accuracy of the purchase behavior.

Furthermore, the disclosed recording module can accumulate the purchase quantity of the same product, providing merchants with accurate sales data to assist in analyzing product sales trends and customer behavior. Furthermore, when the purchase quantity of a specific product reaches a set notification threshold, the recording module automatically sends a replenishment signal, which helps with inventory management, avoids out-of-stock situations, improves product supply efficiency, and enhances the efficiency of retail management.

In addition, this disclosure can push replenishment signals and related product information to managers or suppliers through social communication platforms to ensure real-time notification and rapid response, thereby improving the efficiency of the overall replenishment process.

Furthermore, the display module disclosed herein can present real-time dynamic images and the product's sensing area frame, product outer frame, product center point, and its movement trajectory, thereby assisting management personnel in intuitively observing product trends and customer behavior patterns.

To facilitate the description of the central concept of the present disclosure in the above invention content column, specific embodiments are presented. Various objects in the embodiments are depicted according to proportions, sizes, deformations or displacements suitable for description, rather than according to the proportions of actual components.

Directional terms mentioned in this disclosure, such as "upper", "lower", "front", "back", "left", "right", "inner", "outer", "side", etc., are merely directions in the drawings; therefore, the directional terms used are for explaining and understanding this disclosure, rather than for limiting this disclosure, as stated above.

1 to 5 , the present disclosure provides a smart product tracking system 100, which includes:

A camera device 10 is mounted around a shelf S in a store (as shown in FIG2 ). The camera device 10 includes a camera module 11 and a transmission module 12 coupled to each other. The camera module 11 can capture the merchandise G on the shelf S and dynamic images of the area surrounding the shelf S to generate a real-time dynamic image P. The transmission module 12 transmits the real-time dynamic image P to the outside. The dynamic image of the area surrounding the shelf S may be the shelf S and a shopping basket B used by a consumer. In the disclosed embodiment, the transmission module 12 transmits the real-time dynamic image P to the outside via wireless transmission.

A processing terminal 20 is coupled to the camera 10. The processing terminal 20 has a database 21, a communication module 22, and a computing module 23 that are coupled to each other. The database 21 stores a plurality of shelf models, a plurality of product models, and a plurality of shopping basket models. The communication module 22 is coupled to the transmission module 12 of the camera 10 and receives real-time dynamic images P. In the disclosed embodiment, the communication module 22 receives the real-time dynamic images P from the transmission module 12 via wireless communication.

The communication module 22 transmits the real-time dynamic image P to the calculation module 23. The calculation module 23 performs object detection and image segmentation on the real-time dynamic image P through a deep learning algorithm, and compares the real-time dynamic image P with the shelf models, product models and shopping basket models in the database 21 to extract the shelf feature F1, product feature F2 and shopping basket feature F3 contained in the real-time dynamic image P. The calculation module 23 uses a region labeling technology to The algorithm uses a polygonal frame selection method to mark a sensing area frame F4 outside the shelf feature F1 and between the shopping basket feature F3. The computing module 23 uses a region annotation technology to mark a product outer frame F5 outside the product feature F2 using a polygonal frame selection method. The computing module 23 analyzes and processes the product outer frame F5 to generate a product center point O, as shown in Figures 1, 3, and 4. The deep learning algorithm is YOLO. V8 deep learning model; Roboflow PolygonZone region annotation technology is used; in the disclosed embodiment, there can be multiple sensing area frames F4, mainly depending on the number of levels of shelf S. Each level of shelf S can generate a sensing area frame F4 between it and shopping basket B.

It should be noted that each shelf model, each product model, and each shopping basket model is generated by augmenting the real-time dynamic image P generated by the camera module 11 with the shelf features F1, product features F2, and shopping basket features F3 within the real-time dynamic image P using deep learning algorithmic techniques in the computing module 23, and through image processing such as color, brightness, cropping, blurring, noise, scaling, stretching, and rotation. Alternatively, each shelf model, each product model, and each shopping basket model is generated by collecting data using a web crawler.

Please refer to Figures 1 to 4. When the product G on the shelf S is moved, the product frame F5 corresponding to the product G and the product center O will produce continuous movement. During the process of the product G being moved, the calculation module 23 will record the product center O generated by the continuous movement and concatenate them to generate a movement trajectory T. When the product frame F5 and the sensing area frame F4 overlap to a set range, and the length of the movement trajectory T reaches a purchase length, the calculation module 23 determines that the product G is purchased and generates a purchase signal. The set range is the overlap between the product frame F5 and the sensing area frame F4. The overlap ratio of F4 exceeds 80 percent; the purchase length is the length of shelf S; the length of the movement trajectory T is the distance between the center point O of the first product and the center point O of the currently generated product. In the disclosed embodiment, as product G is moved, the overlap range between the product outer frame F5 and the sensing area frame F4 gradually increases, and the movement trajectory T moves in a single direction but not in a straight line. The computing module 23 determines that the product G has been purchased. This restriction is to ensure that the product G is actually placed in shopping basket B and not returned to shelf S, thereby achieving a more accurate judgment.

Furthermore, the processing terminal 20 has a recording module 24, which is coupled to the communication module 22, the calculation module 23 and the inventory management system. The calculation module 23 can transmit the purchase signal together with the product feature F2 to the recording module 24. The recording module 24 accumulates the purchase signal corresponding to the product feature F2 to generate a single product purchase quantity. Module 22 obtains inventory quantities of different products G from the inventory management system and sets a unique notification threshold for each product G. When recording module 24 determines that the purchase quantity of a single product has reached the notification threshold, recording module 24 generates a replenishment signal. The notification threshold is the quantity of product G. The notification threshold allows managers (merchants) to set it based on the product G's purchasing process or the characteristics of the product G.

1 and 5 , the communication module 22 is coupled to a social communication platform 1. When the communication module 22 receives the replenishment signal and the product feature F2 transmitted by the recording module 24, the communication module 22 can transmit the replenishment signal and the product feature F2 to the message interface of the social communication platform 1 through a service communication technology. The manager (merchant) can learn from the replenishment signal that the product G needs to be replenished, and can quickly learn the product G that needs to be replenished through the product feature F2 displayed on the message interface, thereby achieving a fast replenishment effect. In the embodiment of the present disclosure, the social communication platform 1 is LINE; the service communication technology is LINE. Notify; and the social communication platform 1 will be displayed on the terminal device (for example: computer, smartphone, tablet, smart wearable device, etc.).

In addition, the processing terminal 20 has a display module 25, which is coupled to the communication module 22 and the computing module 23. The display module 25 can present the real-time dynamic image P received by the communication module 22, and can also present the sensing area frame F4, product outer frame F5, product center point O, and movement trajectory T generated by the computing module 23 in conjunction with the real-time dynamic image P. Among them, the sensing area frame F4 and product outer frame F5 can be presented in different colors or appearances on the display module 25.

In summary, the present disclosure can achieve the following effects:

1. The disclosed intelligent product tracking system 100, through a camera device 10 and a computing module 23, can detect the movement of a product G in real time. As the product G is moved, it generates a movement trajectory T of its center point, accurately tracking whether the product G has been removed. Furthermore, the system can dually determine whether the product G has been purchased by the customer based on the degree of overlap between the product's outer frame F5 and the sensing area frame F4, combined with the length of the movement trajectory T. When these conditions are met, a purchase signal is automatically generated, thereby accurately determining the purchase behavior of the product G.

2. The recording module 24 of the disclosed intelligent product tracking system 100 can accumulate the purchase quantity of the same product G, providing merchants with accurate sales data and assisting in analyzing product G's sales trends and customer behavior. Furthermore, when the purchase quantity of a specific product G reaches a set notification threshold, the recording module 24 automatically issues a replenishment signal, which helps to manage inventory, avoid out-of-stocks, improve the supply efficiency of product G, and enhance the efficiency of retail management.

3. The disclosed intelligent product tracking system 100 can push replenishment signals and related product information to managers or suppliers through the social communication platform 1 to ensure real-time notification and rapid response, thereby improving the efficiency of the overall replenishment process.

4. The display module 25 of the disclosed intelligent product tracking system 100 can present a real-time dynamic image P, along with the sensing area frame F4 of the product G, the product outer frame F5, the product center point O, and its movement trajectory T, thereby helping management personnel to intuitively observe the movement of the product G and customer behavior patterns.

The above embodiments are only used to illustrate the present disclosure and are not intended to limit the scope of the present disclosure. Any modifications or variations that do not violate the spirit of the present disclosure are within the scope of protection intended by the present disclosure.

1: Social communication platform 100: Smart product tracking system 10: Camera 11: Shooting module 12: Transmission module 20: Processing terminal 21: Database 22: Communication module 23: Computing module 24: Recording module 25: Display module S: Shelf G: Product B: Shopping basket P: Real-time dynamic image F1: Shelf features F2: Product features F3: Shopping basket features F4: Sensing area frame F5: Product outer frame O: Product center point T: Movement trajectory

Figure 1 is a block diagram of the disclosed system. Figure 2 is a schematic diagram of an embodiment of the disclosed camera device installed in a store. Figure 3 is a schematic diagram (1) of the disclosed real-time dynamic image analysis, illustrating the selection and setting of a sensing area frame, a product frame, and a product center point. Figure 4 is a schematic diagram (2) of the disclosed real-time dynamic image analysis, illustrating the movement trajectory of a product as it is moved. Figure 5 is a schematic diagram (3) of the disclosed real-time dynamic image analysis, illustrating the transmission of a replenishment signal and product characteristics to a messaging interface on a social communication platform.

1: Social communication platform

100: Intelligent Product Tracking System

10: Photography equipment

11: Shooting Module

12: Transmission module

20: Processing Terminal

21:Database

22: Communication Module

23: Computational Module

24: Recording Module

25: Display Module

Claims (10)

A smart product tracking system includes: A camera device mounted around a shelf, the camera device having a camera module and a transmission module coupled to each other. The camera module can capture dynamic images of the products on the shelf and the surrounding area to generate a real-time dynamic image. The transmission module transmits the real-time dynamic image externally; and A processing terminal is coupled to the photographic device. The processing terminal has a database, a communication module, and a computing module coupled to each other. The database stores a plurality of shelf models, a plurality of product models, and a plurality of shopping basket models. The communication module is coupled to the transmission module of the photographic device and receives the real-time dynamic image. The communication module transmits the real-time dynamic image to the computing module. The computing module performs object detection and image segmentation on the real-time dynamic image through a deep learning algorithm, and compares the real-time dynamic image with each shelf model, each product model, and each shopping basket model in the database to extract a shelf feature, a product feature, and a shopping basket feature contained in the real-time dynamic image. The computing module uses a region labeling The technology uses a polygonal frame selection method to select and mark a sensing area frame outside the shelf feature and between the shopping basket feature, and marks a product frame outside the product feature. The calculation module analyzes and processes the product frame to generate a product center point. When the product on the shelf is moved, the product frame corresponding to the product and the product center point are Continuous movement occurs. As the product is moved, the computing module records the center point of the product and concatenates it to create a movement trajectory. When the product frame and the sensing area overlap within a set range, and the length of the movement trajectory reaches a purchase length, the computing module determines that the product has been purchased and generates a purchase signal. In the intelligent product tracking system of claim 1, the processing terminal further comprises a recording module coupled to the computing module. The computing module transmits the purchase signal together with the product characteristics to the recording module. The recording module accumulates the purchase signal corresponding to the product characteristics to generate a purchase quantity for a single product. In the intelligent product tracking system of claim 2, the communication module is coupled to the recording module and the inventory management system. The recording module obtains inventory quantities of different products from the inventory management system via the communication module and sets a notification threshold specific to each product. When the recording module determines that the purchase quantity of a single product has reached the notification threshold, the recording module generates a replenishment signal. The intelligent product tracking system as described in claim 3, wherein the communication module is coupled to a social communication platform; the communication module receives the replenishment signal and the product characteristics transmitted by the recording module, and the communication module transmits the replenishment signal and the product characteristics to the message interface of the social communication platform through a service dissemination technology. An intelligent product tracking system as described in any one of claims 1 to 4, wherein the processing terminal has a display module, which is coupled to the communication module and the computing module, and the display module is capable of presenting the real-time dynamic image received by the communication module, and presenting the sensing area frame, the product outer frame, the product center point and the movement trajectory generated by the computing module in conjunction with the real-time dynamic image. The smart product tracking system as described in claim 5, wherein the setting range is that the overlap ratio between the product frame and the sensing area frame exceeds 80%. The intelligent product tracking system as described in claim 6, wherein the purchase length is the shelf length; and the length of the moving track is the distance between the center point of the first product and the center point of the currently generated product. In the intelligent product tracking system of claim 7, wherein, as the product is moved, the overlap between the product frame and the sensing area frame gradually increases, and the movement trajectory is a single movement direction, and the computing module determines that the product has been purchased. The intelligent product tracking system as described in claim 5, wherein each of the shelf models, each of the product models, and each of the shopping basket models is generated by augmenting the real-time dynamic imagery generated by the camera module using the deep learning algorithm of the computing module, and performing image processing such as color, brightness, cropping, blurring, noise, scaling, stretching, and rotation to obtain the shelf features, product features, and shopping basket features within the real-time dynamic imagery; or each of the shelf models, each of the product models, and each of the shopping basket models is generated by collecting data using a web crawler. The intelligent product tracking system as described in claim 9, wherein the deep learning algorithm technology is a YOLO V8 deep learning model; and the region annotation technology is Roboflow PolygonZone polygon region annotation technology.