CN110708513B

CN110708513B - 8K video multi-core heterogeneous processing device

Info

Publication number: CN110708513B
Application number: CN201910993073.8A
Authority: CN
Inventors: 孙宏海; 王延杰; 马经纬
Original assignee: Changchun Institute of Optics Fine Mechanics and Physics of CAS
Current assignee: Changchun Institute of Optics Fine Mechanics and Physics of CAS
Priority date: 2019-10-18
Filing date: 2019-10-18
Publication date: 2021-06-01
Anticipated expiration: 2039-10-18
Also published as: CN110708513A

Abstract

An 8K video multi-core heterogeneous processing device relates to the field of ultra-high-definition resolution video imaging and processing technology, and solves the problem that the existing high-resolution monitoring imaging equipment cannot achieve 8K resolution imaging in the same device and complete real-time moving targets at the same time. Problems such as identification and tracking, including ultra-high-resolution CMOS image sensors, multi-core DSP processors, FPGAs and multi-core ARM compression video processors, use FPGA to cache the high-speed image data output by CMOS image sensors, and process them according to different processors. The function requires image cropping and preprocessing, and real-time distribution of images to different processors for real-time processing. The multi-core DSP processor acquires the image of the area of interest, performs image target recognition and tracking processing, and performs automatic dimming control on the image sensor according to the target area. Multi-core ARM processor completes color image restoration, automatic white balance adjustment, real-time 8k video compression, HDMI video display, compression and encoding Gigabit Ethernet port output and other functions.

Description

8K video multi-core heterogeneous processing device

Technical Field

The invention relates to the technical field of ultrahigh-definition resolution video imaging and processing, in particular to the field of ultrahigh-definition large-range scene monitoring imaging, and relates to the technologies of target dynamic processing and 8K video image compression display.

Background

With the development of 5G network technology and ultra-high definition display technology, the ultra-high resolution imaging system is greatly promoted. From digital televisions to high-definition, full-high-definition and ultra-high-definition 4K, and to today's 8K, display pixels are more and more dense, pictures are more and more clear, requirements on imaging systems are more and more high, the requirements of the 2020 Tokyo Olympic Games and the 2022 Beijing Olympic Games are announced, and 8K direct broadcasting is adopted. In the development of the ultra high definition video industry in China, the national broadcast television general administration indicates that the development of the ultra high definition video industry in China is accelerated and promoted again.

The 8K is not only high resolution, but also includes wide color gamut, high data transmission bandwidth, high performance image processing and compression. The acquisition of the content of the ultra-high definition images is a short board of the whole industry of the current 8k videos, the development of the technology plays a crucial role, and higher requirements are put forward on an imaging device and image processing, compression and transmission.

Sony published 2018 that a UHC-8300 8K camera system equipped with three 1.25 inch 8K imagers will be displayed for the first time domestically on a sony BIRTV2018 stand. UHC-8300 opens the door to the 8K world, and its outstanding flexibility will bring 8K, 4K photography and other powerful functions to the user.

At an 8K resolution of 7680x4320(8K UHD,4320p), the total number of pixels reaches 33 Mpix. The requirement of monitoring and acquiring video information in many important scenes puts higher requirements on ultrahigh resolution. The intelligent monitoring equipment has gradually become a facility commonly applied in all social layers, and the related range is wide, including national defense safety, ecological safety, important facility site safety, social order safety, public residence and daily life safety and the like. Therefore, intelligent monitoring products are rapidly developed in more and more fields. In the field of wide-area high-definition monitoring application, a wide-field wide-angle lens is matched, and the 8k imaging processing system can meet the requirements of an ultra-high-definition monitoring scene on a wider monitoring range, clearer image details, stronger image processing capability and faster transmission rate.

Most of the existing high-resolution wide-area monitoring imaging devices are based on coding compression and video recording of 4K resolution images, and can not realize the identification and tracking of real-time moving targets and the long-distance transmission of 8K videos after real-time coding compression processing while 8K resolution imaging in the same device is realized.

Disclosure of Invention

The invention provides an 8K video multi-core heterogeneous processing device, which aims to solve the problems that the existing high-resolution monitoring imaging equipment cannot realize the identification and tracking of real-time moving targets while 8K resolution imaging in the same equipment is realized.

An 8K video multi-core heterogeneous processing device comprises an ultrahigh-resolution CMOS image sensor, a multi-core DSP processor, an FPGA and a multi-core ARM compressed video processor; the method is characterized in that:

the FPGA acquires data and imaging drive control of the CMOS image sensor with ultrahigh resolution, caches the acquired original image data, and cuts and preprocesses the image according to the requirements of different video processors;

the multi-core DSP processor sends a target position to the FPGA, the FPGA is used for windowing an interested target and then sending an image to the multi-core DSP processor, and the multi-core DSP processor identifies and tracks the target according to the interested target area and can simultaneously perform real-time dimming processing on the target area;

the multi-core DSP processor receives the sending of the system control parameters and the state information of the upper computer through a serial port and a network port;

the FPGA sends 8K videos and 2K interesting region images of a central region of a field of view to a multi-core ARM compressed video processor in real time to perform compressed encoding output and storage recording of the images;

the multi-core ARM compressed video processor is used for color image restoration, automatic white balance adjustment, real-time 8K video compression and HDMI video display, and is output through a gigabit network port after being compressed and coded.

The invention has the beneficial effects that: the invention adopts an image sensor with ultrahigh resolution, can carry out ultrahigh-definition imaging of 8K @30fps, segments and extracts a high-definition image from a dynamic target position interested in a visual field, and transmits the compressed image in a long distance by using a standard gigabit Ethernet through an 8K video compression technology. Through a high-speed CXP interface, original ultra-high-definition image data can be transmitted to a close-range acquisition processing storage system.

Drawings

Fig. 1 is a system block diagram of an 8K video multi-core heterogeneous processing apparatus according to the present invention;

fig. 2 is an internal functional block diagram of an FPGA in the 8K video multi-core heterogeneous processing device according to the present invention.

Detailed Description

In a first specific embodiment, the present embodiment is described with reference to fig. 1 and fig. 2, and an 8K video multi-core heterogeneous processing apparatus, which uses an ultra-high resolution CMOS image sensor (GMAX3265 resolution reaches 9344 × 7000 pixels), can meet the 8K video imaging requirement of 7680x4320@30 fps. The method comprises the steps of carrying out imaging control on an original ultrahigh-resolution CMOS image sensor, and simultaneously adopting a multi-core heterogeneous processor architecture to carry out processing on an ultrahigh-resolution image and outputting an original image. Meanwhile, a high-performance FPGA, a multi-core DSP and a multi-core ARM processor architecture are adopted to process the 8k video.

High-speed image data output by the CMOS image sensor is cached by adopting a high-performance FPGA, image cutting and preprocessing are carried out according to the requirements of processing functions of different processors, and the images are distributed to different processors in real time for real-time processing.

The multi-core DSP processor mainly acquires images of the interested areas, performs image target recognition and tracking processing, and can perform automatic dimming control on the image sensor according to the target areas.

The multi-core ARM processor is mainly used for completing the functions of color image restoration, automatic white balance adjustment, real-time 8k video compression, HDMI video display, compressed coding gigabit network port output and the like.

Simultaneously with the 8k image compression, the image compression coding of the region of interest (1920 x 1080) is also completed. The compressed image can be stored and recorded by a USB3.0 interface, a SATA interface and a system EMMC FLASH. Meanwhile, the display output of the HDMI interface can be performed.

The processing device described in this embodiment can output the original image data output by the image sensor through the GTY interface of the FPGA, and convert the original image data into a standard CXP high-speed interface, an optical fiber interface, or a minSAS interface through the high-speed image expansion interface board to output and record the original image data.

In this embodiment, the 8K video image sensor is a GMAX3265 CMOS image sensor, the pixel size is 3.2um, the full-frame effective photosensitive area is 9344(H) x 7000(V), the global electronic shutter, 56 channels of sub-LVDS data output, 10 bits of grayscale output, and imaging at 30fps or more can be achieved.

The multi-core DSP processor adopts a multi-core fixed point/floating point dual-core TMS320C6657 based on TI KeyStone C66x, and has multiply-accumulate capability four times that of a C64x + device under the same frequency; the main frequency is 1.0/1.25GHz, the computing capacity of each core can reach 40GMACS and 20GFLOPS, the system comprises 2 Viterbi coprocessors and 1 Turbo coprocessing decoder, each core comprises 32KByte L1P, 32KByte L1D, 1MByte L2, 1MByte multi-core shared memory and 8192 multipurpose hardware queues, and DMA transmission is supported; the system supports various high-speed interfaces such as SRIO, EMIF16 and gigabit network ports, and supports low-speed interfaces such as SPI, UART and boot;

hi3559A provides efficient and rich computing resources for high-performance multi-core AMR compressed video processors (compressed image gigabit network interface and HDMI display interface), supporting different video processing applications.

The Hi3559AV100 integrates a dual-core A73 and a dual-core A53, a large and small core architecture and a dual operating system, so that power consumption and starting time are balanced. The digital video recording of 8K30/4K120 broadcast-grade image quality is provided, 8K video and 2K video input is supported, H.265 coding output or video-grade RAW data output is supported, high-performance ISP processing is integrated, and meanwhile, an advanced low-power-consumption process and a low-power-consumption architecture design are adopted. 8KP30(7680x4320) +1080P30H.265 coding can be completed simultaneously. The HDMI2.0 is supported, and the output of 4K @60fps is supported at most.

The FPGA is used as a data master control processor, and Xilinx series Kintex UltraScale + FPGA with the model of XC7KU11PFFVE1517-2-I is selected.

UltraScale+^TMThe optimal cost/performance/power consumption balance is provided at a FinFET (16nm) node, and the highest cost performance is realized for high-end functions such as a transceiver, a memory interface linear speed and a 100G connection core. The brand new middle-end series are perfect choices of packet processing and DSP intensive functions, and an internal GTY high-speed data interface is adopted to be suitable for realizing CXP, miniSAS and optical fiber interfaces.

The FPGA is a core control device for imaging control and data exchange, and the maximum imaging resolution of the image sensor of the external expansion interface is as follows: 9344 7000@30fps color bayer format 10 bit.

The 8K compression shows maximum resolution: 7680x4320@30fps (storable after compression);

2K compression shows maximum resolution: 1920x1080@30fps (storable after compression);

DSP interested area resolution: 1920x1080@30fps (target recognition and tracking, dimming and other processing);

gigabit net 1(Hi 3559A): and (5) compressed image transmission.

Gigabit network 2(DSP 6657): and transmitting a full-frame original image, and performing system control and state feedback.

A display interface: HDMI 4K @60 fps.

Controlling the serial port and the GPIO interface: and finishing the system imaging parameter setting and imaging synchronous control.

In the embodiment, 4 DDR4 particles with 16bit width are adopted by the system to form a DDR4 bus with 64bit bus bit width, the bus data rate is designed to be 2400Mbps, and the requirement of high-bandwidth image data caching of the system can be met.

The image sensor interface is designed by adopting a 56-bit sub-LVDS bus, and the transmission data rate is designed to be 1.08 Gbps.

And the image data and the DSP system adopt mature Rapid IO serial buses alternately, so that the high-resolution windowing image transmission can be met.

The FPGA and the multi-core ARM compressed video processor Hi3559A adopt 16 paths of sub-LVDS buses with the transmission rate of 1Gbps, and the transmission requirement of 8K videos is met.

Kintex US + FPGA is mainly responsible for collecting, caching and sending a large amount of high-speed image data, and is responsible for power-on mode control, system reset control, EMIF bus data collection and control, and interconnection and control of UART, I2C, SPI, GPIO and other low-speed buses of the DSP and the AMR processor. And the control of peripheral devices such as FRAM, RTC, clock chip, temperature sensor and the like is carried out.

In this embodiment, the FMC320 interface is a dedicated high-speed interconnect interface with the multi-core ARM compressed video processor Hi 3559A. The QSH-180 interface is a dedicated interface for interconnecting 3265 image sensors.

The universal interface of the device comprises a gigabit network, a USB3.0 and a SATA interface. The low-speed serial port and GPI interface which are expanded externally mainly support the expansion of RS232 interface, 422 interface and TTL GPIO interface; and the high-speed interface for external expansion adopts QSH-40D differential connectors, and high-speed buses such as CXP 4x, miniSAS, QSFP28 and the like can be externally expanded.

In the embodiment, the FPGA mainly finishes the acquisition and imaging drive control of the ultrahigh-resolution CMOS image data, the acquired original image data is quickly cached in a high-capacity DDR4 image cache extended from the FPGA in real time, the data format conversion is carried out by adopting a multi-stage FIFO, the image is cut and preprocessed according to the requirements of different video processors, the DSP sends a target bit, the FPGA is responsible for windowing the interested target and then sending the image to the DSP, the DSP carries out the identification and tracking processing of the target according to the target area, and meanwhile, the real-time dimming processing of the target area can be carried out. The DSP can receive the sending of system control parameters and state information through a serial port and a network port. The FPGA sends the 8K video of the central area of the field of view and the 2K region of interest image to Hi3559A in real time for compression coding output and storage recording of the image. And simultaneously, the FPGA sends the original image data to the high-speed video expansion interface through the GTY high-speed differential serial interface, and the original image data is converted into CXP, miniSAS or optical fiber interface to output the high-speed original video.

The present embodiment is described with reference to fig. 2, and fig. 2 is a main functional block diagram of an FPGA internal data flow.

The 3265Sub-lvds data acquisition module finishes the acquisition of the ultra-high resolution CMOS image data and the generation of a test image. The 3265 imaging drive control module receives imaging control parameters processed by the control serial port and the DSP and then controls working parameters of the 3265 image sensor to complete control of imaging functions such as exposure time, gain, frame frequency and the like.

The SENSOR image data is written into the raw image data of the 3265 image SENSOR acquired by the FIFO and is quickly cached in a high-capacity DDR4 image buffer expanded by the FPGA in real time.

And the DDR4MGT controller completes the switching control of writing and reading DDR4 buses of different data sources.

The DSP system parameter configuration and function control module is in charge of communicating with the DSP6657 through an EMIF bus in a DSP6657 interface, receives configuration parameters of the FPGA and position results processed by the DSP target, and controls various imaging functions.

The FPGA receives the target position sent by the DSP, and the image after windowing the interested target is sent to the DSP through the DSP-rapidIO interface. Image data of a windowing 1920x1080 is converted into YUV image data through a Bayer image data reading FIFO of the BT1120 according to a target position transmitted by the DSP, and then the YUV format color image data is transmitted to Hi3559A through an FMC-BT1120 transmission controller.

The original Bayer image data in the DDR4 buffer is read out through the LVDS image data read-out FIFO, and the original 8K image data is sent to the Hi3559A through the FMC-LVDS sending control module.

And sending the original image data to a high-speed video expansion interface through a GTY high-speed differential serial interface by virtue of the high-speed image data read FIFO, and converting the original image data into CXP, miniSAS or optical fiber interface to output the high-speed original video.

Claims

1. an 8K video multi-core heterogeneous processing device, comprising an ultra-high-resolution CMOS image sensor, a multi-core DSP processor, an FPGA and a multi-core ARM compression video processor; it is characterized in that:

The FPGA collects the data and imaging drive control of the ultra-high-resolution CMOS image sensor, caches the collected original image data, and cuts and preprocesses the image according to the needs of different video processors;

The multi-core DSP processor sends the target position to the FPGA, and the FPGA is used to open the window for the target of interest, and then sends the image to the multi-core DSP processor, and the multi-core DSP processor performs target recognition and tracking processing according to the target area of interest , and can perform real-time dimming processing of the target area;

The multi-core DSP processor receives the control parameters and status information of the host computer system through the serial port and the network port;

The FPGA sends the 8K video and 2K region of interest image in the central area of the field of view to the multi-core ARM compressed video processor in real time for image compression encoding output and storage recording;

The multi-core ARM compressed video processor is used for color image restoration, automatic white balance adjustment, real-time 8K video compression and HDMI video display, and is output through a Gigabit Ethernet port after compression and encoding;

It also includes a DDR4 image buffer. The FPGA completes the collection of ultra-high-resolution CMOS image data and imaging drive control, and real-time buffers the collected original image data into the large-capacity DDR4 image buffer extended by the FPGA. The data format is converted by the first-level FIFO, and then the image is cropped and preprocessed according to the needs of the multi-core ARM compressed video processor or the multi-core DSP processor.

2. The 8K video multi-core heterogeneous processing device according to claim 1, wherein the original image data output by the super-high-resolution CMOS image sensor is output through the GTY high-speed differential serial interface of the FPGA, and then After the high-speed image and video expansion interface board is converted into a standard CXP high-speed interface, optical fiber interface or minSAS interface, the original image data is output.