KR20250015150A - System and method for detecting object based on uvc camera image - Google Patents

Abstract

A method for object detection based on UVC camera images is provided. The method includes a step of changing the video data from the CPU into a predetermined type and providing it to the GPU; a step of generating a texture readable by a rendering engine based on the provided video data from the GPU; a step of providing graphic data processed by the GPU based on a two-dimensional text and a GPU memory pointer generated by the CPU to the CPU; a step of serializing and converting the graphic data into a matrix form and providing it as an input of an object detection algorithm; and a step of providing an object detection result as an output of the object detection algorithm from the CPU.

Description

{SYSTEM AND METHOD FOR DETECTING OBJECT BASED ON UVC CAMERA IMAGE}

The present invention relates to a UVC camera image-based object detection system and method.

When implementing an object detection algorithm, developers sometimes receive image data through a UVC standard camera in an embedded module that uses the Android operating system.

At this time, a UVC standard camera is a camera that follows the USB Video Class (UVC) standard. As a standard that provides video streaming via a USB interface, a camera that follows this standard can be used immediately on a computer or other host device without installing a driver.

UVC cameras are commonly used in a variety of applications, including webcams, security cameras, endoscopy cameras, and industrial inspection cameras. The UVC standard defines standardized protocols and formats related to video streaming, which can provide compatibility and convenience.

Most UVC standard cameras support the Plug and Play function, and are powered and transmit video data via a USB connection. Therefore, users can easily use the UVC camera by simply connecting it to a computer, usually without installing a separate driver.

However, if you want to implement an object detection algorithm using a rendering engine (or game engine) such as the Unity Engine or create an application that utilizes the algorithm, you need an efficient process that processes the input image in real time.

Publication of Patent Publication No. 10-2023-0022453 (2023.02.15)

An embodiment of the present invention provides a UVC camera image-based object detection system and method, which can perform object detection in real time while effectively utilizing system resources when using a standard UVC device and an embedded system in object detection, which is a type of computer vision.

However, the technical tasks that this embodiment seeks to accomplish are not limited to the technical tasks described above, and other technical tasks may exist.

As a technical means for achieving the above-described technical task, a UVC camera image-based object detection method according to a first aspect of the present invention includes the steps of: receiving video data from a UVC camera in a CPU; changing the video data into a predetermined type in the CPU and providing it to a GPU; generating a texture readable by a rendering engine based on the provided video data in the GPU; providing graphic data processed by the GPU based on the two-dimensional text and GPU memory pointer generated by the CPU to the CPU; serializing and converting the graphic data into a matrix form in the CPU and providing it as an input of an object detection algorithm; and providing an object detection result as an output of the object detection algorithm in the CPU.

In some embodiments of the present invention, the step of receiving video data from the UVC camera in the CPU may include the step of storing the video data received from the UVC camera in the CPU in a video stream buffer; and the step of receiving the video data stored in the buffer when new video data is received from the UVC camera.

In some embodiments of the present invention, the step of receiving video data from the UVC camera may process and receive video data from which brightness information and color difference information are separated from the UVC camera as a single video.

In some embodiments of the present invention, the step of changing the video data from the CPU to a predetermined type and providing it to the GPU may change the color expression type of the video data from YUV to RGBA and provide it to the GPU.

In some embodiments of the present invention, the step of providing graphic data processed by the GPU to the CPU may include the step of generating a two-dimensional texture having the same resolution and color expression type as the generated texture by the CPU; the step of providing the two-dimensional texture generated by the CPU to the GPU; the step of directly copying the texture to a two-dimensional texture of a rendering engine using a GPU memory pointer and storing graphic data processed by the GPU in a frame buffer; and the step of copying the graphic data stored in the frame buffer to a two-dimensional texture of the CPU and providing the same.

In some embodiments of the present invention, the step of converting the graphic data in the CPU into serialized and matrix form and providing it as an input of an object detection algorithm may include the steps of reading pixels of the two-dimensional texture and serializing them into a byte array; converting the serialized byte array into a matrix form; and providing the byte array converted into the matrix form as an input of the object detection algorithm.

In addition, a UVC camera image-based object detection system according to a second aspect of the present invention includes a CPU which receives video data from a UVC camera, changes the video data into a predetermined type, a GPU which generates a texture readable by a rendering engine based on the video data, and provides graphic data processed based on the two-dimensional text and GPU memory pointer generated by the CPU to the CPU, wherein the CPU serializes and converts the graphic data into a matrix form and provides it as an input of an object detection algorithm, and provides an object detection result as an output of the object detection algorithm.

In addition, other methods for implementing the present invention, other systems, and computer-readable recording media recording a computer program for executing the method may be further provided.

According to one embodiment of the present invention described above, by supporting UVC 1.0/1.1/1.5 devices and embedded systems using the standard UVC specification, hardware constraints for creating applications utilizing computer vision can be minimized. That is, according to one embodiment of the present invention, there is an advantage in that a consistent object detection algorithm can be used in various devices and systems.

In addition, the OpenGL|ES Graphics API can be used to effectively utilize the limited system resources of embedded systems, which has the advantage of allowing the creation of object detection applications while minimizing system resource consumption. In other words, efficient and high-performance object detection can be implemented even in embedded environments.

In addition, it is easier to create new content or expand functions by combining existing computer vision algorithms with commercial game engines. In other words, since it is ultimately built through a commercial game engine, the technology can be widely distributed to existing user bases, and it is advantageous for writing applications and expanding functions because it can be integrated with various functions and plug-ins provided by the game engine.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

Figures 1 to 3 are flowcharts of an object detection method according to one embodiment of the present invention.
FIG. 4 is a block diagram of an object detection system according to one embodiment of the present invention.

The advantages and features of the present invention, and the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and the present embodiments are provided only to make the disclosure of the present invention complete and to fully inform a person skilled in the art of the scope of the present invention, and the present invention is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. The terms "comprises" and/or "comprising" as used in the specification do not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like components throughout the specification, and "and/or" includes each and every combination of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, it is to be understood that these components are not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it should be understood that a first component mentioned below may also be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with the meaning commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries shall not be ideally or excessively interpreted unless explicitly specifically defined.

The present invention relates to a UVC camera image-based object detection system and method.

For example, if the image data of the embedded module is output in the form of GL_OES_EGL_image_external, which is an extension of OpenGL|ES, the image data can be easily processed in the Unity engine like a general OpenGL texture. In fact, in one embodiment of the present invention, the texture is converted into a matrix for computer vision processing and used. In addition, if a memory pointer is used for the efficiency of the process of transferring the texture, the parallel operation of the GPU can be actively utilized. That is, if a memory pointer is used, direct data transfer with the GPU is possible without going through the CPU, and since the GPU has excellent parallel operation processing capability, the data transfer and processing speed can be improved.

Through this, one embodiment of the present invention can transmit real-time image data while effectively using system resources and easily execute computer vision processing from pixel data obtained through the process. At this time, one embodiment of the present invention can efficiently process computationally intensive tasks such as computer vision processing.

Hereinafter, a UVC camera image-based object detection method (hereinafter, “object detection method”) according to one embodiment of the present invention will be described with reference to FIGS. 1 to 3.

Figures 1 to 3 are flowcharts of an object detection method according to one embodiment of the present invention.

First, the CPU receives video data from the UVC camera.

In one embodiment, the UVC camera can separate brightness information (Y) and chrominance information (UV) (S105), and the CPU can receive video data in which brightness information and chrominance information are separated. The video data is generally encoded in a YUV (YCbCr) color space, and through the above process, one embodiment of the present invention can independently manipulate or process brightness and color in an image or video processing task.

At this time, the CPU stores the video data received from the UVC camera in the video stream buffer (S110), and when new video data is received from the UVC camera, the video data stored in the buffer can be read out (S115). The video stream buffer can temporarily store video data in a memory area. The video stream buffer is continuously updated, and can be updated whenever new video data arrives.

Next, the CPU changes the video data to a predetermined type and provides it to the GPU (S120). In one embodiment, the CPU can change the color expression type of the video data from YUV to RGBA and provide it to the GPU. Since most graphic systems and applications use the RGBA color space (red, green, blue, alpha (transparency)), one embodiment of the present invention changes YUV to the RGBA type.

Next, the GPU generates a texture that can be read by the rendering engine based on the provided video data (S125).

In one embodiment, the GPU can receive RGBA pixel data and create a texture. In this case, the texture is created in the form of GL_OES_EGL_image_external, an extension of OpenGL|ES, which provides the ability to read texture data from an external image source. The rendering engine can use this Texture to read color values.

Next, the CPU generates a two-dimensional texture with the same resolution and color expression type so that the texture generated by the GPU can be directly imported into the rendering engine via the GPU (S130). Then, the CPU transfers the generated texture to the GPU.

Next, the GPU directly copies the texture to the two-dimensional texture of the rendering engine through the GPU memory pointer and stores the graphic data processed by the GPU in the frame buffer (S135). In one embodiment of the present invention, the GPU can process textures in parallel, so that efficient data transfer and processing is possible using the memory pointer.

Next, the GPU copies the graphic data stored in the frame buffer to a two-dimensional texture of the CPU and provides it to the CPU (S140).

Next, the CPU copies the values in the frame buffer to the CPU's two-dimensional texture (S145). Since editing pixel data of a texture is only possible on the CPU, the CPU copies the values stored in the frame buffer to the CPU's two-dimensional texture so that they can be edited directly.

Next, the CPU reads pixels of the two-dimensional texture and serializes them into a byte array to execute object detection (S150). That is, one embodiment of the present invention can convert pixel values into byte format and store them as an array so that they can be easily applied to various processing algorithms, especially object detection algorithms.

Next, the CPU converts the serialized byte array into a matrix form (S155). That is, one embodiment of the present invention can convert a byte array, which is serialized graphic data, into a matrix form used in OpenCV (or Computer Vision). When data is converted into a matrix form, computer vision processing tasks such as object detection can be performed. Meanwhile, OpenCV is a library that provides various functions and algorithms for computer vision processing.

Next, the CPU converts the RGBA graphic data to achromatic grayscale (Alpha8) (S160). This is because the object detection algorithm is based on the difference in brightness of the image. The image converted to grayscale removes the color information and focuses on the brightness information, making it a good form for applying the object detection algorithm.

Next, the CPU provides the byte array converted into a matrix form as input to a pre-learned object detection algorithm so that the object detection result is output (S165). The learned object detection algorithm is trained to identify and extract a specific object within an image, and by applying the matrix converted into grayscale to the object detection algorithm, the desired object can be identified and detected from the input image.

Meanwhile, in the above description, steps S105 to S165 may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present invention. In addition, some steps may be omitted as needed, and the order between the steps may be changed. In addition, even if other omitted contents are present, the contents described in FIGS. 1 to 3 and FIG. 4 are mutually applicable.

FIG. 4 is a block diagram of an object detection system (100) according to one embodiment of the present invention.

An object detection system according to one embodiment of the present invention includes at least one CPU (110) and a GPU (120).

The CPU (110) receives video data from the UVC camera and changes the video data into a predetermined type.

The GPU (120) generates a texture that can be read by the rendering engine based on video data, and provides graphic data processed based on the two-dimensional text and GPU memory pointer generated by the CPU (110) to the CPU (110).

Thereafter, the CPU (110) serializes the graphic data and converts it into a matrix form and provides it as an input to the object detection algorithm, and provides the object detection result as an output of the object detection algorithm.

The embodiments of the present invention described above may be implemented as a program (or application) to be executed in combination with a hardware server and stored in a medium.

The above-described program may include codes coded in a computer language, such as C, C++, JAVA, or machine language, that can be read by the processor (CPU) of the computer through the device interface of the computer, so that the computer reads the program and executes the methods implemented as a program. Such codes may include functional codes related to functions that define functions necessary for executing the methods, and may include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. In addition, such codes may further include memory reference-related codes regarding which location (address address) of the internal or external memory of the computer should be referenced for additional information or media necessary for the processor of the computer to execute the functions. In addition, if the processor of the computer needs to communicate with another computer or server located remotely in order to execute the functions, the code may further include communication-related code regarding how to communicate with another computer or server located remotely using the communication module of the computer, what information or media to send and receive during communication, etc.

The above storage medium means a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device. That is, the program can be stored in various storage media on various servers that the computer can access or in various storage media on the user's computer. In addition, the medium can be distributed to computer systems connected to a network, so that a computer-readable code can be stored in a distributed manner.

The steps of a method or algorithm described in connection with the embodiments of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination of these. The software module may reside in a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Flash Memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable recording medium well known in the art to which the present invention pertains.

Above, while the embodiments of the present invention have been described with reference to the attached drawings, it will be understood by those skilled in the art that the present invention may be implemented in other specific forms without changing the technical idea or essential features thereof. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

100: Object Detection System
110: CPU
120: GPU

Claims (7)

In a method performed by a computer,
Step of receiving video data from UVC camera in CPU;
A step of changing the video data into a predetermined type in the CPU and providing it to the GPU;
A step of generating a texture readable by a rendering engine based on the provided video data from the GPU;
A step of providing graphic data processed by a GPU to the CPU based on two-dimensional text generated by the CPU and a GPU memory pointer;
A step of converting the graphic data into serial and matrix form in the CPU and providing it as input to an object detection algorithm; and
Including a step of providing an object detection result as an output of the object detection algorithm in the CPU.
A method for object detection based on UVC camera images.
In the first paragraph,
The step of receiving video data from the UVC camera in the above CPU is:
A step of storing video data received from the UVC camera in the video stream buffer in the CPU; and
Including the step of receiving video data stored in the buffer when receiving new video data from the UVC camera.
A method for object detection based on UVC camera images.
In the first paragraph,
The step of receiving video data from the above UVC camera is:
Processing and receiving video data from which brightness information and color difference information are separated from the above UVC camera as a single video.
A method for object detection based on UVC camera images.
In the third paragraph,
The step of changing the video data from the above CPU to a predetermined type and providing it to the GPU is:
The color expression type of the above video data is changed from YUV to RGBA and provided to the GPU.
A method for object detection based on UVC camera images.
In the first paragraph,
The step of providing the graphic data processed by the GPU to the CPU is:
A step of generating a two-dimensional texture having the same resolution and color expression type as the generated texture in the above CPU;
A step of providing the two-dimensional texture generated by the CPU to the GPU;
A step in which the GPU directly copies the texture to a two-dimensional texture of the rendering engine using a GPU memory pointer and stores the graphic data processed by the GPU in a frame buffer; and
A step of providing the graphic data stored in the frame buffer by copying it to a two-dimensional texture of the CPU.
A method for object detection based on UVC camera images.
In the first paragraph,
The step of converting the graphic data into serial and matrix form in the CPU and providing it as input to the object detection algorithm is as follows.
A step of reading pixels of the above two-dimensional texture and serializing them into a byte array;
A step of converting the above serialized byte array into a matrix form; and
A step of providing a byte array converted into the matrix form as input to an object detection algorithm,
A method for object detection based on UVC camera images.
A CPU that receives video data from a UVC camera and changes the video data into a predetermined type;
A GPU is included that generates a texture that can be read by a rendering engine based on the above video data, and provides graphic data processed based on the two-dimensional text and GPU memory pointer generated by the CPU to the CPU.
The above CPU serializes the graphic data and converts it into a matrix form and provides it as an input to an object detection algorithm, and provides an object detection result as an output of the object detection algorithm.
UVC camera image-based object detection system.