CN120529163A

CN120529163A - Illumination imaging method and device for mark recognition

Info

Publication number: CN120529163A
Application number: CN202510968892.2A
Authority: CN
Inventors: 张扬; 张苏宁
Original assignee: Ningbo Xinsuan Technology Co ltd
Current assignee: Ningbo Xinsuan Technology Co ltd
Priority date: 2025-07-15
Filing date: 2025-07-15
Publication date: 2025-08-22

Abstract

The invention relates to the technical field of image sensors and discloses an illumination imaging method and equipment for mark recognition, which comprise an image sensor array, a group of imaging lens systems, one or more groups of illumination systems, a group of control and calculation subsystems, a group of power supply systems and a group of substrates, wherein the image sensor array is used for exposing and collecting a target area mark row by row to enable each row of pixel points to be in an exposure state to form a target mark image, the group of imaging lens systems are used for guiding reflected light of the target mark image to be transmitted to the image sensor array and reducing interference of other parasitic lights except illumination of an illumination system of the imaging device, the one or more groups of illumination systems are used for providing specific illumination for the target mark image in a specific window period to enable exposure intensity of the image pixel points in the window period to be higher than that in a non-window period. The invention effectively solves the problem of jelly effect of the rolling shutter image sensor, and greatly expands the further application of the rolling shutter image sensor in the field of mark recognition.

Description

Illumination imaging method and device for mark recognition

Technical Field

The invention relates to the technical fields of image sensors, visual intelligence, video sequence capturing and visual measurement, in particular to an illumination imaging method and equipment for mark recognition.

Background

In the field of modern information processing and automation, machine-readable graphic marks are widely used in numerous scenarios, like logistics management, industrial production quality detection, document identification, product tracing, etc. The method covers various OCR symbols, watermarks LOGO, locators, one-dimensional codes, two-dimensional codes, defect matching and tracking matching symbols and the like, is used as a graphic representation of various information, and is a key medium for realizing rapid and accurate information interaction. The code reader for identifying and extracting the graphic mark information becomes a core device in the related workflow.

The code reader works on the principle that one or more light sources illuminate the graphic marks and reflected light is focused on the image sensor through a lens in the code reader. The pixel units in the image sensor array record the color and brightness information of the corresponding positions of the graphic marks in a charge form, store the color and brightness information as two-dimensional image data, and then process the data by a decoder to extract the information contained in the data.

The image sensors in the code reader are classified into a global shutter and a rolling shutter according to an exposure mode. When the global shutter is exposed, the whole pixel array can be exposed at the same time, all pixels receive incident light in the same time period and store the incident light as electric signals, the shutter is closed during reading, the electric signals are fixed and read out in sequence, and after the processing is finished, the electric signals stored in the pixel points are cleared, so that the next exposure is prepared. The exposure mode has obvious advantages when shooting fast moving objects, can ensure that no time difference exists between the sampling lines, has small image distortion and is easy to realize multi-camera synchronization. However, global shutters have some drawbacks, since there is typically only one analog-to-digital converter (ADC), the more pixels are exposed, the more pixels are transmitted, the lower the total frame rate, and also the higher the ambient noise, the manufacturing cost, and the lower the light sensitivity.

In contrast, rolling shutter image sensors have one analog-to-digital converter (ADC) per column of pixels, with significantly reduced conversion times. And the exposure mode allows each row of pixel points to start the exposure of the next frame immediately after the previous frame is read out, and the method has advantages over the global shutter image sensor in Quantum Efficiency (QE), read-out noise, dynamic range, image frame rate and cost and size. In application scenes with high requirements on image quality and frame rate and sensitive cost, a rolling shutter image sensor can theoretically acquire images with higher quality, so that the rolling shutter image sensor has great application potential in the field of mark recognition.

The special exposure of rolling shutter image sensors presents a serious problem. The method is like a rolling shutter door, and reset, exposure and reading operations are performed on pixel points row by row from top to bottom or from bottom to top. In the reading process, there is an inherent reading time, and only one row of pixel data can be read at the same time. To reduce the total exposure time, the pixel reset start time of the next row is slightly later than that of the previous row. This results in different exposure time points for each row of pixels, and when there is an object in the frame with a movement speed exceeding the exposure movement tolerance of the sensor, the spatial position of the object is continuously changed during the exposure process, and the frame is distorted, tilted, and the like, namely, a "jelly effect" occurs.

In practice, the "jelly effect" is extremely detrimental to the machine recognition of graphic marks in moving scenes. Whether a moving camera is used to take pictures containing graphic marks, a handheld device collects graphic marks in shake, or a logistics conveyor collects graphic marks of fast moving goods, and a production line collects part graphic marks affected by high-frequency mechanical vibration, the problem of "jelly effect" of a rolling shutter image sensor is inevitably encountered. The acquired graphic mark image is severely deformed, so that the identification accuracy of the code reader to the graphic mark is greatly influenced, the read code rate is low, and the actual application requirements cannot be met.

At present, the problem of jelly effect of a rolling shutter image sensor is difficult to effectively solve in the prior art, and further application of the rolling shutter image sensor in the field of mark recognition is limited. Therefore, developing an illumination imaging method and device for identifying marks, which can overcome the jelly effect and fully play the advantages of a rolling shutter image sensor, becomes a key technical problem to be solved in the field.

Disclosure of Invention

The invention aims to provide an illumination imaging device for mark recognition, which is used for solving the problem that the jelly effect of a rolling shutter image sensor is difficult to effectively solve in the prior art.

In order to solve the problems, the invention adopts the following technical scheme:

An illumination imaging apparatus for marker recognition, comprising:

an image sensor array for exposing and collecting the target area mark line by line to make each line of pixel point in an exposure state to form a target mark image;

a group of imaging lens systems is provided, the image sensor array is used for guiding reflected light of the target mark image to be transmitted to the image sensor array and reducing other parasitic light interference, except illumination of an illumination system of the imaging device, received by the image sensor array;

One or more groups of illumination systems for providing specific illumination for the target mark image during a specific window period, so that the exposure intensity of the image pixels during the window period is higher than that during a non-window period;

The control and calculation subsystem is used for acquiring illumination sequence information of the illumination system according to different application scenes, controlling the illumination system to provide illumination for the target mark image, and enabling the image sensor carrying the rolling shutter to simulate the global shutter sensor for imaging;

A set of power supply systems;

A set of substrates is used to mount and connect the image sensor array, imaging lens system, illumination system, control and computing subsystem, and power supply system.

The advantage of this scheme lies in:

The illumination system provides specific illumination for the target mark image in a specific window period, so that the exposure intensity of the image pixels in the window period is higher than that in a non-window period, and the roller shutter image sensor can realize exposure of each row of pixels in a short interval on the premise that the roller shutter image sensor can not simultaneously expose each row of pixels of the image sensor array, namely, the roller shutter image sensor can realize the effect of similar simultaneous exposure although not simultaneously exposing.

The method comprises the steps of collecting a target area mark through line-by-line exposure of an image sensor array, utilizing the advantages of a rolling shutter image sensor, guiding reflected light by an imaging lens system, reducing parasitic light interference, improving image quality, enabling an illumination system to provide specific illumination in a specific window period, enhancing exposure intensity of target mark image pixels, enabling a control and calculation subsystem to acquire illumination sequence information according to different scenes, enabling the rolling shutter image sensor to simulate an effect that a global shutter sensor simulates exposure to each line of pixels approximately simultaneously, and effectively reducing jelly effect in a fast moving identification scene, enabling a power supply system and a substrate to provide power and installation connection basis for all components, and enabling the device to be compact in structure and integrated in functions. The technical scheme fully plays the advantages of the rolling shutter image sensor in the field of mark recognition, solves the problem that the jelly effect of the rolling shutter image sensor is difficult to solve in the prior art, and is a brand-new illumination imaging device design for mark recognition.

Preferably, the image sensor array is a CMOS image sensor array or a CCD image sensor array.

Preferably, the imaging device is used for decoding target marks of the acquired image data, wherein the target marks comprise OCR symbols, watermarks LOGO, locators, various system one-dimensional codes, various system two-dimensional codes, defect matching symbols and tracking matching symbols.

Preferably, the imaging device comprises a portable form and structure suitable for use in a handheld, medical setting for securing and protecting the imaging device.

Preferably, the imaging device is integrally formed and is suitable for industrial production and logistics transportation;

The imaging device comprises a split type appearance and structure which are suitable for industrial production and logistics transportation and are provided with screw interfaces and interchangeable lenses, wherein the split type appearance and structure are used for fixing and protecting the imaging device, and the screw interfaces comprise C, CS, F, M interfaces, M42 interfaces, M58 interfaces and M72 interfaces.

Preferably, the imaging device comprises a scalable profile and structure suitable for sports photography, with both hand-held and fixed to sports equipment and sporting equipment, for securing and protecting the imaging device.

Preferably, the CMOS image sensor array is a CMOS image sensor array using a rolling shutter technique of exposing line by line and resetting line by line;

Or the CMOS image sensor array is a CMOS image sensor array using a rolling shutter technique with a global reset function with a row-by-row exposure.

The imaging LENs system comprises a LENs main body, at least one LENs, and optionally a bandpass filter, a polarization filter, a focal length adjusting device and/or an aperture adjusting device, wherein the imaging LENs system adopts one or more modes including T-LENs, a voice coil motor VCM and a liquid LENs for focusing.

Preferably, the lens body is configured to:

controlling the aperture stop to a minimum extent allowed by the device under a specified scene to reduce the transmission of ambient stray light to the image sensor array;

Or controlling the aperture to be adjusted within the range of F2.3-F16 allowed by the equipment, and accurately guiding and transmitting the reflected light of the target image to the image sensor array;

or an external mechanical shutter is selected and matched in a specified scene, and when the mechanical shutter is closed, reflected light received by the lens can be completely blocked;

or a light shielding plate is selected in a specified scene, and reflected light received by the lens can be completely blocked when the light shielding plate is closed.

Preferably, the illumination system comprises an aiming light source and an auxiliary filter lens, an exposure light source and an auxiliary filter lens, a light source synchronous controller and a communication interface.

Preferably, the light source synchronization controller is configured to generate an illumination switch control signal which is modulated and is coordinated with the exposure action of the rolling shutter image sensor array, wherein the control signal comprises a high level signal, a low level signal, a positive pulse trigger signal, a negative pulse trigger signal and/or a PWM modulation signal.

Preferably, the light source synchronous controller is used for generating an illumination switch control signal which is modulated and linked with the exposure action of the rolling shutter image sensor array so as to precisely control an exposure light source, and comprises a light intensity control module, an illumination time sequence mode module and a circulation and synchronization mechanism module, wherein:

the light intensity control module is used for adjusting the light intensity during illumination;

The illumination time sequence control module controls an illumination switch in a specific window period according to four combination types of an illumination time axis and an exposure time axis, wherein in the specific window period for illuminating a target mark image, the following conditions are required to be met by a1 st line starting exposure time T ₁, an N line starting exposure time T ₂, an illumination starting time T ₃, an illumination closing time T ₄ and a1 st line ending exposure time T ₅:

The time length limit is that T ₄- T₃≤ D_p/V_s,D_p is the maximum pixel offset and V _s is the motion speed;

The opening time limit is T ₃≥ T₂;

closing time limit T ₄≤ T₅;

T ₂= T₁+ N_row*T_gap or T ₂= T₁+ T_readall,N_row is the number of vertical pixel lines, T _gap is the time interval of exposure or reading of two adjacent lines, the unit is seconds, and T _readall is the total time required for completing reading of all lines of a frame of image;

the illumination timing pattern module includes four illumination timing patterns:

The first illumination time sequence mode is that the illumination starting time is earlier than the first line exposure starting time, and the illumination closing time is earlier than the last line exposure ending time;

The second illumination time sequence mode is that the illumination starting time is later than the first row exposure starting time, and the illumination closing time is later than the last row exposure ending time;

The third illumination time sequence mode is that the illumination starting time is earlier than the first row exposure starting time, and the illumination closing time is later than the last row exposure ending time;

A fourth illumination time sequence mode, wherein the illumination starting time is later than the first row exposure starting time, and the illumination closing time is earlier than the last row exposure ending time;

the circulation and synchronization mechanism module is executed in a circulation mode according to a preset parameter sequence corresponding to four illumination time sequence modes, and when illumination parameters change, synchronous modification of exposure parameters of the image sensor is automatically triggered.

Preferably, the light source synchronous controller is used for generating an illumination switch control signal which is modulated and is linked with the exposure action of the rolling shutter image sensor array, the control signal controls the exposure light source to illuminate with different brightness, and the light source synchronous controller is composed of one or more of FPGA, a plate level MCU and a processor unit in a control and calculation subsystem.

Preferably, the light source synchronization controller is configured to generate a modulated control signal that is coordinated with the exposure light source, the control signal precisely controls the on/off of the aiming light source and causes the aiming light source to illuminate and indicate the marking area in a single point, double point, single line, cross line, and/or rectangular frame shape, automatically turn off when the exposure light source is on, and remain on when the exposure light source is on.

Preferably, the illumination system comprises one or more groups of exposure light sources, different groups of exposure light sources allowing different types of light source types to be used, and the different groups of exposure light sources are independently controlled by the light source synchronous controller, wherein each group of exposure light sources can select the same or different specifications, namely illumination types comprising direct light, uniform light and polarized light, and colors comprising red visible light, blue visible light, infrared light and white mixed light.

Preferably, the control and computing subsystem comprises a processor unit, a memory unit, a trigger unit, an internal interface unit and an external interaction unit.

The triggering unit is a group of software and hardware modules which comprise a plurality of digital sensors and are provided with specific algorithms including a shadow compensation algorithm, a position prediction algorithm and a motion artifact quantization algorithm, the plurality of digital sensors comprise a light intensity detection sensor, a speed sensor and an acceleration sensor, the sensors can detect motion characteristics of the device and environmental characteristics which influence image sampling in the environment where the device is positioned, the characteristics are primarily calculated and analyzed, and after being reported to the processor unit, the processor readjust exposure and illumination configuration parameters according to the reporting unit, and then the exposure and illumination flow is started, wherein the specific detection range data of each sensor are as follows:

Light intensity sensors 1-65535lux,

A speed sensor, 0-10m/s,

Acceleration sensor + -1G.

According to the illumination imaging method for identifying the mark, specific illumination is provided for the target mark image in a specific window period, so that the exposure intensity of the pixel point of the target mark image is larger than that of the non-window period, and the image sensor carrying the rolling shutter simulates the global shutter sensor to image.

The advantage of this scheme lies in:

The invention overcomes the technical prejudice in the field, skillfully ensures that the exposure intensity of the target mark image is larger than the exposure intensity of the non-window pixel point through the illumination of a specific window period, ensures that the rolling shutter image sensor can overcome the defects of the rolling shutter image sensor, achieves the effect of simulating the global shutter sensor, has the advantages of the rolling shutter sensor and the global shutter sensor, and realizes the optimization of cost and imaging quality.

Preferably, the method comprises the following steps:

step one, system construction and configuration;

The image sensor array configuration comprises selecting a rolling shutter image sensor array, setting a row-by-row exposure mode, ensuring that a specific window period for exposure exists between each row of pixel points so as to collect a target area mark and prepare for forming a target mark image;

The imaging lens system is configured, wherein the imaging lens system is selected, and parameters of the imaging lens system are configured so that the imaging lens system can guide reflected light of a target mark image to be accurately transmitted to the image sensor array;

the illumination system is configured to provide specific illumination for the target mark image in a specific window period so as to ensure that the exposure intensity of the pixels of the target mark image is greater than that of other pixels;

The control and calculation subsystem is configured, namely, all configurations and using methods of the rolling shutter image sensor array, the imaging lens system and the illumination system are packaged into a program, and the program is scheduled and executed by the control and calculation subsystem;

triggering sampling target marks and transmitting signals to a control and calculation subsystem;

step three, illumination imaging;

The control and calculation subsystem controls the illumination system, and in the process of exposing the rolling shutter image sensor array, the illumination system is started in a specific window period according to the preconfigured illumination sequence information to provide specific illumination for the target mark image, so that the pixel point of the target mark image obtains enough and proper exposure intensity;

step four, image post-processing and decoding;

The control and calculation subsystem performs post-processing on the image information acquired by the image sensor array, including image denoising, contrast enhancement and color correction;

decoding the post-processed image, and identifying the marking information therein.

According to the scheme, the specific window period is skillfully set through the lighting system, so that the pixel points in the window period can be specially illuminated, the pixel points in the specific window period can be in an exposure state together at a certain moment, and the exposure intensity is higher than that in the non-window period. Through system configuration and accurate execution of each step, the problem of jelly effect is effectively solved, and the application range of the method is greatly expanded in the field of mark recognition.

Preferably, in the lighting system configuration of the first step, including using a plurality of sets of lighting preset configurations, the method includes:

Sequentially performing a preset illumination configuration during one exposure period;

Sequentially performing a preset illumination configuration during a plurality of exposure periods;

performing a specified certain preset illumination configuration during one exposure period;

or to perform a specified certain preset illumination configuration within a multiple exposure period.

Preferably, the lighting synchronization controller may be configured and generate the following lighting behavior:

After receiving the illumination trigger signal, the illumination is turned on for 400 microseconds, the illumination intensity is 85% of the maximum power of the illumination light source, and the illumination is turned off after the illumination lasts for 30 microseconds;

or the illumination is started immediately without delay after the illumination triggering signal is received, the illumination intensity is 50% of the maximum power of the illumination light source, and the illumination is stopped after the illumination lasts for 100 microseconds;

Or after the first illumination trigger signal is received, delaying, immediately starting illumination, wherein the illumination intensity is 30% of the maximum power of the illumination light source, and stopping illumination after illumination lasts for 100 microseconds, and then increasing the illumination intensity by 10% of the maximum power every time when the illumination trigger signal is received until the maximum power is reached;

or after the illumination trigger signal is received for the first time, the illumination is started for 20 microseconds, the illumination intensity is 90% of the maximum power of the illumination light source, the illumination is stopped after the illumination lasts for 100 microseconds, the illumination is started for 20 microseconds after each time of the illumination trigger signal is received, and the circulation is stopped after the preset maximum illumination delay starting threshold value is reached;

Or after the first illumination trigger signal is received, the illumination is started in 400 microseconds, the illumination intensity is 85% of the maximum power of the illumination light source, the illumination is stopped after the illumination lasts for 10 microseconds, the illumination duration of 5 microseconds is increased every time the illumination trigger signal is received, and the circulation is stopped after the preset maximum illumination duration threshold value is reached;

or presetting a plurality of illumination sub-sequences which are not more than the limit number of the system, wherein each sub-sequence can have completely or partially different configurations on the starting illumination delay, the illumination power and the illumination duration, and executing one sequence of the preset illumination sub-sequences in sequence after receiving the illumination trigger signal each time.

Preferably, in the controlling and computing subsystem configuration of the first step, the controlling and computing subsystem obtaining the content of each preset configuration includes:

analyzing and calculating the information reported by the triggering unit;

determining parameters of reset, integration, transfer and readout of a rolling shutter image sensor;

Calibrating working environment parameters of an imaging device using the method, wherein the environment parameters comprise a target mark moving speed, a target mark distance, a target mark angle and environment illumination intensity;

Determining an illumination sequence of an illumination system based on the rolling shutter image sensor parameters and the working environment parameters, wherein the illumination sequence comprises illumination intensity, starting time and ending time of a single or a plurality of illumination systems;

And calculating and determining a switching sequence of the global reset or external physical shutter based on the rolling shutter image sensor parameters and the working environment parameters.

Preferably, in the trigger sampling process of the second step:

The method comprises the steps that through a physical key arranged in an imaging device, a key is pressed to trigger sampling;

Or automatically detecting the occurrence of the target mark by utilizing a photoelectric sensor, a distance sensor or an infrared sensor, and automatically triggering a sampling flow when a preset condition is met;

or triggering sampling by an upper computer through an external communication interface;

Or the control and computation subsystem controls the timing to trigger sampling.

Preferably, the triggering unit tracks and collects the position, the moving speed, the height, the angle and the temperature of the target relative to the center of the lens of the imaging device, and the position, the moving speed, the height, the angle and the temperature are preprocessed and then reported to the control and calculation subsystem.

Preferably, the illumination imaging method is applied to industrial production and logistics transportation scenes of display dead spot detection, large-scale plane device scratch detection and package tracking;

the illumination imaging method is applied to daily outdoor scenes of aviation shooting and panoramic motion shooting of the small multi-axis aircraft;

The illumination imaging method is applied to the acquisition and identification scenes of various types of one-dimensional code images and two-dimensional code images in the fields of industry, commerce and logistics motion;

The illumination imaging method is applied to scientific research scenes shot by a digital microscope for marker tracking.

The third scheme is a code reader comprising the imaging device according to the first scheme, and target reading and identification are performed through the imaging device.

The advantage of this scheme lies in:

On the basis of the existing code reader, an imaging device capable of overcoming the jelly effect of the rolling shutter image sensor is introduced, the structure and the function of the code reader are innovatively improved, the performance of the code reader is improved, and the code reader can better finish the mark recognition task in various application scenes.

Different from the traditional code reader working mode, the scheme applies the illumination imaging method specially designed for the rolling shutter image sensor to the code reader, solves the 'jelly effect' problem existing in the prior code reader when the rolling shutter image sensor is used, is an innovation on the code reader working method, and is beneficial to promoting the development of the code reader technology in the related field.

Compared with the prior art, the illumination imaging device and the illumination imaging method for the mark recognition, which are suitable for the roller shutter, have remarkable advantages in the field of mark recognition, and can provide efficient and accurate mark recognition solutions for multiple fields of industrial production, logistics transportation, daily outdoor, scientific research and the like.

The mark recognition accuracy is improved, namely, the parasitic light interference is reduced through the imaging lens system, the illumination system provides accurate specific illumination, and the control and calculation subsystem processes and analyzes the image efficiently, so that various target marks can be recognized accurately, and the mark recognition accuracy is improved effectively. In the logistics transportation scene, the recognition accuracy of the two-dimensional code or the one-dimensional code on the package is greatly improved, the conditions of misreading and misreading are reduced, and the accuracy of package tracking is ensured.

The device has various shapes and structures, such as portable shapes and structures suitable for handheld and medical scenes, integrated fixed shapes and structures suitable for industrial production and logistics transportation, split shapes and structures with screw interfaces and changeable lenses, and expandable shapes and structures suitable for sports shooting, and can meet the requirements of different scenes. In the industrial production, the device with the integrated fixed type and split type structure can be stably arranged on a production line to efficiently detect products, and in daily outdoor scenes, the portable and expandable structure is convenient for users to carry and use, so that panoramic shooting and marking identification are realized.

The flexible system configuration comprises various configuration options of an image sensor array, an imaging lens system, an illumination system and the like, and a user can flexibly select and adjust the image sensor array, the imaging lens system, the illumination system and the like according to actual requirements. For example, the type of the light source, the illumination intensity, the illumination time and the like in the illumination system can be accurately controlled, so that the device can adapt to different shooting environments and target marking characteristics, and the universality and the adaptability of the device are improved.

The high-efficiency data processing and transmission comprises the steps of processing, identifying and decoding the collected images rapidly by the strong processing capacity of the control and calculation subsystem, and transmitting the identification result to other systems in time through various external interaction interfaces, so that the high-efficiency sharing and application of the data are realized. In industrial production, the product detection result can be rapidly transmitted to a production management system, and a basis is provided for production decision.

Drawings

Fig. 1, fig. 2, fig. 3, fig. 4, fig. 5, fig. 6, fig. 7 each show an exposure process of the rolling shutter image sensor and a cause of the "jelly effect".

Fig. 8 and 9 are schematic diagrams showing the effect of adjusting the exposure time of the rolling shutter image sensor on the imaging effect.

Fig. 10, 11, 12 show schematic views of definition of the exposure and illumination windows of the rolling shutter image sensor.

Fig. 13 shows a schematic comparison of the effect of a particular lens and a particular illumination on the imaging effect, including an upper and a lower graph.

Fig. 14 is a schematic diagram of a system architecture of a method according to an embodiment of the invention.

Fig. 15 is a schematic view of an apparatus structure of the method according to the embodiment of the present invention.

Fig. 16 is a flowchart of one of the operations of the apparatus of fig. 15.

Fig. 17 is another workflow diagram of the apparatus of fig. 15.

Fig. 18 is a real-time image of the use of the code reader in an embodiment of the invention.

Fig. 19 is a real image of the use of the conventional code reader in the same use scenario as fig. 18.

Reference numerals in the drawings of the specification include an image sensor array 200, an image sensing unit 202, a first row sensing unit 204, a second row sensing unit 206, a third row sensing unit 208, a fourth row sensing unit 210, a graphic marking 304, an image sensor 402, a control system 404, an illumination system 406, a lens system 408, a target image 412, a mark to be recognized 414, a zoom controller 416, a lens 418, an aperture controller 420, an aiming light filter 422, an illumination light filter 424, an aiming light source 426, an illumination light source 428, an illumination synchronization controller 430, a communication interface 432, a storage unit 434, a processor unit 436, a trigger unit 438, an external interface unit 440, an internal interface unit 442, an aiming mark 444, an illumination light 446, a device housing 448, a metal housing 502, a polarized illumination subsystem 504, a lens 506, a direct illumination subsystem 508, a uniform illumination subsystem 510, a second aviation plug 512, a first aviation plug 514, a knob 516, a fixing plug 518, a status display lamp 520, a debug button 522, and a photonic system 524.

Detailed Description

The following is a further detailed description of the embodiments:

The invention provides an illumination imaging device, a method and a code reader for identifying marks. In the method, calibration configuration parameters are determined according to the working environment of the equipment and the characteristics of the equipment, and then the imaging action of the image sensor system and the illumination action of the flash lamp system are precisely controlled in a linkage mode according to the configuration parameters. The system and the equipment applying the method can obviously lighten the jelly effect of the rolling shutter image sensor, effectively improve the imaging accuracy of the equipment in the mark recognition fields of aviation shooting, display dead point detection and positioning, mark automatic tracking digital microscope, panoramic motion shooting and the like, and enable the details and characteristics of the shot object to be truly and accurately reflected in the recording or reproducing process of the image.

The illumination imaging device for identifying the marks can accurately acquire the marks of the target areas through the cooperative work of the image sensor array, the imaging lens system, the illumination system, the control and calculation subsystem, the power supply system and the substrate, and decode various target marks such as OCR marks, watermarks LOGO, locators, various system one-dimensional codes, various system two-dimensional codes, defect matching marks, tracking matching marks and the like.

An illumination imaging device for marker recognition includes an image sensor array, an imaging lens system, an illumination system, a control and computing subsystem, a power supply system, and a substrate.

The image sensor array is used for exposing and collecting the target area mark row by row so that each row of pixel points are simulated to be in an exposure state approximately at the same time to form a target mark image;

In one embodiment, the CMOS image sensor array is adopted, wherein a rolling shutter technology which is exposed line by line and reset line by line is selected, the technology can be used for efficiently acquiring images and is particularly suitable for capturing dynamic target marks, and a rolling shutter technology which is exposed line by line and has a global reset function is also adopted, so that the image acquisition quality is further optimized, and the image noise and interference are reduced.

In one embodiment a CCD image sensor array is used, which has a high resolution and excellent image quality performance, as an alternative, capable of satisfying very demanding image detail marker recognition scenarios, such as microscopic marker tracking photography in scientific research.

The imaging lens system consists of a lens main body, at least one lens, a bandpass filter which is optionally matched according to actual requirements, a polarization filter which is optionally matched, a focal length adjusting device which is optionally matched and an aperture adjusting device which is optionally matched. The lens main body and at least one lens are necessary accessories, and the bandpass filter, the polarization filter, the focal length adjusting device and the aperture adjusting device are all optional accessories, wherein one or more of the optional accessories are selected according to actual requirements. The main objective of the imaging lens system is to guide the reflected light of the target mark image to be accurately transmitted to the image sensor array, and reduce other parasitic light interference received by the image sensor array except the illumination of the self-illumination system of the imaging device. The focusing technology used by the device comprises a T-LENs, a Voice Coil Motor (VCM) and a liquid LENs, and the focal length can be flexibly adjusted according to different shooting requirements. The lens main body can operate under different scenes, for example, when the ambient stray light needs to be reduced, the aperture is controlled to be reduced to the minimum extent allowed by the device, when the reflected light of a target image needs to be accurately guided, the aperture is adjusted to the proper extent, an external mechanical shutter can be selected under a specific scene, and when the mechanical shutter is closed, the reflected light received by the lens is completely blocked.

The illumination system comprises one or more groups, and comprises an aiming light source, an auxiliary filter lens, an exposure light source, an auxiliary filter lens, a light source synchronous controller and a communication interface.

The control and calculation subsystem consists of a processor unit, a memory unit, a trigger unit, an internal interface unit and an external interaction unit. Processor units include various types such as reduced instruction set processors (RISC), complex instruction set processors (CISC), microprocessors, embedded processors, co-processors, microcontrollers, image processing application specific integrated circuits (ISP), neural Processing Units (NPUs), system-on-chip (SOCs), field Programmable Gate Arrays (FPGAs), digital signal processing units (DSPs), programmable Logic Devices (PLDs), and associated capacitors, resistors, inductors, application specific ICs, and the like. The system can execute a series of instructions to control the illumination system to turn on and off illumination at a designated time, control the lens system to adjust focal length and aperture according to designated configuration, configure working parameters of the image sensor array, control exposure and result reading, perform image processing, identification, decoding and reporting on image information obtained by exposure, sense an indication signal reported by a trigger unit, perform analysis and calculation, and perform other control operations of the whole system.

The power supply system supplies power to the whole device, comprises a 12V alternating current power supply adapter, POE (Power over Ethernet), an amorphous silicon solar cell, a battery and a USB interface, and can select a proper power supply mode according to different use scenes and requirements.

The substrate is used for installing and connecting the CMOS image sensor array (or CCD image sensor array), the lens system, the lighting system, the control and calculation subsystem and the power supply system, and ensures stable connection and cooperative work among all components.

The imaging method of the invention comprises the following steps:

The image sensor array is configured and used, and according to different application scenes and requirements, working parameters of the rolling shutter image sensor array (CMOS or CCD) such as parameters of reset, integration, transfer and readout are configured so as to realize efficient acquisition of the target area marks.

Selecting and configuring a lens system, namely selecting a proper imaging lens system, automatically or manually adjusting a focal length and an aperture according to the working environment of equipment, reducing the interference of other stray light received by a lens, and accurately guiding and transmitting the reflected light of a target mark image to an image sensor array. For the rolling shutter CMOS image sensor with global reset, the reset time and the opening and closing time of the external mechanical shutter can be adjusted according to the calibration in advance or the field test parameters, and the synchronous modification of the exposure parameters of the image sensor and the illumination parameters of the illumination system can be triggered when the time changes.

And configuring and using an illumination system, namely configuring one or more groups of illumination systems, and calculating and determining an illumination sequence of the illumination system according to the information reported by the triggering module, the working environment parameters (such as target mark moving speed, target mark distance, target mark angle and environment illumination intensity) of the imaging device and the parameters of the rolling shutter image sensor, wherein the illumination sequence comprises illumination intensity, starting time and ending time. The illumination system provides specific illumination, and may use multiple sets of illumination preset configurations, sequentially executing or executing a specified certain preset illumination configuration during one or more exposure periods. The illumination control signal is used for precisely controlling the exposure light source to start illumination at appointed time before and after the first row of the rolling shutter image sensor array starts exposure, closing the illumination at appointed time before and after the last row ends exposure, controlling the illumination intensity of the illumination light source, circularly controlling the illumination according to a preset parameter sequence, and triggering synchronous modification of the exposure parameters of the image sensor when the illumination parameters are changed. Specific exposure illumination methods are adopted for different types of target marks, such as OCR symbols, watermarks LOGO, locators, one-dimensional codes of various systems, two-dimensional codes of various systems, defect matching symbols and tracking matching symbols.

Triggering sampling, namely triggering a sampling target mark in an automatic or manual mode. The automatic triggering can detect the motion characteristics of the device and the environmental characteristics of the device by a sensor in the triggering module, the characteristics are calculated and analyzed preliminarily and then are reported to the processor unit, the processor readjust the exposure and illumination configuration parameters according to the reported information and then the exposure and illumination process is started, and the manual triggering can be manually triggered and reported to the processor unit by an operator through a physical key to start the exposure and illumination process. The trigger module can also track and collect data such as the position, the moving speed, the height, the angle, the temperature and the like of the target relative to the lens center of the imaging device, and preprocess and report the data to the control and calculation subsystem.

And the system scheduling execution is performed by encapsulating all configurations and using methods of the rolling shutter image sensor array, the lens system and the lighting system into a program and scheduling execution by the control and computing subsystem so as to ensure the cooperative work among the systems.

And (3) image post-processing and decoding, namely performing post-processing on the acquired image information, performing post-processing and decoding on OCR symbols and watermark LOGO by using a specific algorithm, performing post-processing and decoding on one-dimensional codes of various systems and two-dimensional codes of various systems, and performing post-processing, pattern recognition and tracking on various defect symbols and positioning symbols.

In particular, the method comprises the steps of,

In order to eliminate the influence of the jelly effect of the rolling shutter image sensor on imaging in the field of mark recognition, the rolling shutter image sensor marks a certain target area according to the existing working mode, and the process of forming a target mark image generates obvious jelly effect as shown in fig. 1 to 7.

Fig. 1, 2, 3, and 4 illustrate operation of a pixel array of a rolling shutter image sensor at successive different times, respectively.

It is assumed that the exposure sequence of the image sensor array is from top to bottom, that the time consumed for each row of pixels in the image sensor to perform the reset (state labeled R), exposure (state labeled E) and readout (state labeled T) operations is the same and is one unit time, that the readout state T is a generalized set including multiple steps of charge transfer, ADC conversion, digital signal output, and the like, and that the upper and lower edge portions, i.e., the first row and the last row, are simplified, only the pixel array containing the target image is concerned, and the background where the target image is located is not considered.

The image sensor array 200 in this example is a 13x30 matrix, i.e. 30 image sensing units 202 are arranged in each row for a total of 13 rows. The entire image sensor array 200 includes 390 image sensing units 202, each image sensing unit 202 can sample a pixel, and the image sensor array 200 can sequentially sample an image with a resolution of 13x 30.

The first, second, third, and fourth row sensing units 204, 206, 208, 210 refer to the first, second, third, and fourth rows, respectively, of the image sensor array.

In fig. 1, at this time, it is the time zero of the current sampling of the image sensor array 200, the image sensor rows 204 enter the reset state R, and the reset operation may clear the charge accumulated in each image sensor unit 202 in the image sensor rows 204 in the previously exposed or non-exposed state.

In fig. 2, the image sensor row 206 takes over the image sensor row 204 to enter the reset state R, which is one unit time later than the operating state of the image sensor array 200 of fig. 1. The image sensor rows 204 enter the exposure state E, at which time each image sensing unit 202 in the image sensor rows 204 begins to receive reflected light from the target image and initiate photoelectric conversion.

In fig. 3, the image sensor row 208 takes over the image sensor row 206 to enter the reset state R, which is one unit time later than the operating state of the image sensor array 200 of fig. 2. Image sensor row 206 takes over image sensor row 204 into exposure state E. The image sensor rows 204 enter a readout state, at which time the pixel value of each image sensor unit 202 in the image sensor rows 204 can be read out, i.e., the first row data of the target image can be obtained.

In fig. 4, the image sensor row 210 takes over the image sensor row 208 to enter the reset state R at this time, which is one unit time later than the operating state of the image sensor array 200 in fig. 3. Image sensor row 208 takes over image sensor row 206 into exposure state E. The image sensor rows 206 enter a readout state, at which time the pixel values of each image sensor unit 202 in the image sensor rows 206 can be read out, i.e., second row data of the target image can be obtained.

In this example, the image sensor row 204 enters an idle state, and in the continuous exposure mode, the image sensor row 204 may also enter a reset state again, and then perform exposure and readout therewith, at which time the first row data, which is the next frame target image, is sampled.

Fig. 1 to 4 illustrate a series of reset, exposure, and readout operations for each image sensor row in the image sensor array 200. At the end of this process, complete image data of the target image may be obtained, with each image sensor row capturing image data corresponding to the target image. The reset, exposure and read-out operations are shown as being performed row by row from top to bottom, but may also be performed row by row from bottom to top, column by column from left to right, and column by column from right to left, as preset conditions above.

Fig. 1 to 4 also clearly describe the reason why the "jelly effect" occurs, and the target image is at different positions at the respective exposure times of the first row 204, the second row 206, the third row 208, and the fourth row 210 of the image sensor.

Fig. 5 shows the position of the target image relative to the image sensor array 200 at time zero of the current sample, and fig. 5 shows the position of the target image relative to the image sensor array 200 after three unit time delays of the current sample. The image obtained at the current time is, as shown in fig. 7, formed by "stitching" the first line data D _i of the target image at the zero time, the second line data F _i of the target image after one unit time delay, the third line data G _i of the target image after two unit time delays, and the fourth line data H _i of the target image after three unit time delays, and the phenomenon that the upper image is more "left" and the lower image is more "right" appears as a whole.

Further analysis shows that as the speed of movement of the target image increases, the sampled image becomes more oblique, ultimately leading to failure of the graphical marker recognition.

According to the analysis, the global exposure image sensor has no jelly effect, so that a series of combined measures can be adopted to enable the imaging device carrying the rolling shutter image sensor to simulate the global shutter image sensor in the aspects of exposure mode, characteristics and effect, thereby eliminating the jelly effect and improving the sampling accuracy of the imaging device in the field of mark recognition.

Techniques and examples for illumination imaging to eliminate the "jelly effect" of rolling shutter image sensors in the field of marker identification are disclosed.

Techniques for controlling flash selection and illumination in conjunction with a rolling shutter image sensor sampling graphical indicia in accordance with imaging device linkages specifying lens characteristics are disclosed.

The invention discloses a lens selection method for mark recognition, a rolling shutter image sensor control method, a flash lamp selection and illumination control method are mutually matched and closely related, and the characteristic parameters and the configuration parameters of any one of the above parameters can be independently adjusted to possibly lead to the fact that a target image meeting the mark recognition condition cannot be sampled, so that a linkage method is needed to be used for deciding the selection and the configuration of a lens system, a sensor system and an illumination system. Each row of pixels in the entire pixel array simultaneously begins to receive light signals from the lens system when the global shutter image sensor is exposed, without causing a "jelly effect". It is therefore possible to take some technical means such that the rolling shutter image sensor has characteristics similar to those of a global shutter image sensor in that the respective pixels start exposure at the same time.

As the target image shown in fig. 8, assume that:

the target image is moving rightwards at a uniform speed and the background of the target image is not considered;

the exposure sequence of the image sensor array is from top to bottom;

The time consumed by each row of pixels in the image sensor in the reset (with the state marked as R) and readout (with the state marked as T) operation is the same and is one unit time, the readout state T is a general finger set and comprises a plurality of steps of charge transfer, ADC conversion, digital signal output and the like, and the exposure (with the state marked as E) duration of each row of pixels of the image sensor is thirteen unit times by using the control method of the rolling shutter image sensor in the present disclosure. While simplifying the upper and lower edge portions, i.e., the first and last rows, only the pixel array containing the target image is of interest.

Assuming that the time T ₀ is zero time of the present sampling of the image sensor array 200, the time T ₁ is time delayed by one unit time from the time T ₀, the time T ₂ is time delayed by one unit time from the time T ₁, and so on, the time T ₁₄ is time delayed by one unit time from the time T ₁₃.

From the characteristics of the rolling shutter image sensor, the image sensor row 204 enters a reset state at time T ₀. At time T ₁ image sensor row 204 enters the exposure state and 206 enters the reset state, because the exposure time is thirteen units long, image sensor row 204 remains in the exposure state and image sensor row 206 enters the exposure state at time T ₂, and image sensor row 208 enters the reset state. At time T ₃ the image sensor rows 204, 206, 208 enter the exposure state and the image sensor row 210 enters the reset state. By analogy, at time T ₁₃, all of the image sensor rows 204 and 228 in image sensor array 200 are exposed, and image sensor row 204 has been exposed for twelve units of time and will end exposure after one unit of time and image sensor row 228 has been exposed for one unit of time and will end exposure after twelve units of time.

It is not enough to control only the length of exposure time of the rolling shutter image sensor so that each row of pixels is simultaneously in an exposed state to simulate the behavior of the global shutter, which may cause exposure ghost problems.

As shown in fig. 8, for example, at A, B, C points in the target image, the three pixels are projected onto 204, 214, and 228 rows of the image sensor array 200, respectively.

At time T ₁, point a in the target image is acquired by the pixel points of 204 rows and 230 columns of image sensor array 200.

At time T ₂, since the target image is moved one unit distance to the right and the sensor row 204 is still in the exposed state, point a is acquired by the 204 row 232 column pixel points of the image sensor array 200.

Analysis shows that from time T ₁ to time T ₁₃, the target image A point is acquired once by all pixels between 230 columns and 234 columns in 204 rows without considering the background.

Similarly, from the time T ₁ to the time T ₁₃, the target image B is sequentially collected once by all pixels between 234 columns and 238 columns in 214 rows.

Similarly, the target image C is acquired once by 228 rows 236 columns of pixels.

At the time T ₁₃, as shown in fig. 9, the images acquired by the sensor array 200 are a, B, and C, which are three pixel points A, B, C in the target image acquired by the sensor array at the time T ₁₃, respectively, and a ₁ to a ₁₂ and B ₆ to B ₁₂ are redundant information caused by long-time exposure.

It can be seen that the original target image has not been recognized.

Considering that a background exists in the actual situation, for example, after the pixel point 230 completes the exposure of the point a at the time T ₁, the background on the left side of the point a of the target image is continuously exposed in the period from T ₂ to T ₁₃, and charges are continuously accumulated until the exposure of the pixel line 204 is finished, so that the quality of the finally acquired image is worse.

Therefore, a linkage method based on illumination selection and control of specified lens characteristics is needed to solve the problem of 'motion ghost' of exposure.

In one aspect, in one embodiment using the methods described in the present disclosure, the image acquisition system selects a particular stray light eliminating lens. Such a stray light eliminating lens can greatly reduce the light reaching the image sensor array through the lens, so that even if the image sensor array is in an exposure state without flash illumination, the photoelectric conversion device can only receive weak illumination and generate trace charges, and the interference image acquired by the rolling shutter image sensor at unnecessary time (such as the time period from T ₀ to T ₁₂ in the scene) is reduced.

On the other hand, in the above embodiment, the image acquisition system selects an illumination subsystem and a corresponding control method, wherein the illumination subsystem can independently control the on and off of the flash lamp. Since the use of the stray light removing lens reduces reflected light from the target image received by the image sensor array, the image sensor array may be underexposed. Illumination of the target image using an illumination system during a particular window of the exposure period of the rolling shutter image sensor is required to achieve both simultaneous and sufficient exposure of each row of pixels of the image sensor array and significant elimination of "jelly-effect" and "motion artifacts" to provide a final sampled image signature of sufficient quality for subsequent machine identification, with the illumination window being schematically illustrated in fig. 10, 11, and 12. The illumination window period is an illumination time period for the illumination system to start illumination at a specified time and to turn off illumination at a specified time, wherein the specified time is different according to four combination types between an illumination time axis and an exposure time axis, the four combination types are respectively a first illumination time sequence mode (early), a second illumination time sequence mode (early and late), a third illumination time sequence mode (late) and a fourth illumination time sequence mode (late), and the specified time is for carrying out illumination time sequence control according to the four illumination time sequence modes:

in the specific window period for illuminating the target mark image, the following conditions (condition one) need to be met by the 1 st row starting exposure time T ₁, the N th row starting exposure time T ₂, the illumination starting time T ₃, the illumination turning-off time T ₄ and the 1 st row ending exposure time T ₅:

The opening time limit is T ₃≥ T₂;

closing time limit T ₄≤ T₅;

four illumination timing modes:

and the fourth illumination time sequence mode is that the illumination starting time is later than the first row exposure starting time, and the illumination closing time is earlier than the last row exposure ending time.

In one embodiment of the present disclosure, as shown in fig. 10, the first row of pixels of the image sensor array starts to be exposed at time T1, the last row of pixels of the image sensor array starts to be exposed at time T ₂, the first row of pixels of the image sensor array ends to be exposed at time T ₅, and the last row of pixels of the image sensor array ends to be exposed at time T ₆.

The illumination method disclosed by the invention can accurately calculate and acquire illumination sequence information according to the working environment of the equipment and the characteristics of the equipment, and comprises the time T ₃ of illumination starting 306, the time T ₄ of illumination closing 308 and illumination intensity 310. The acquired target image is free from overexposure and underexposure, and obvious jelly effect and motion ghost.

In another embodiment of the present disclosure, as shown in fig. 11, a rolling shutter image sensor with global reset is selected in this embodiment, the reset actions of each row of pixels in the conventional rolling shutter image sensor are performed in an independent sequence, and the rolling shutter image sensor with global reset can control the reset and reset states of the pixels simultaneously by an external signal. The first row of pixels of the image sensor array begins to be exposed at time T ₁, the last row of pixels of the image sensor array begins to be exposed at time T ₂, the first row of pixels of the image sensor array ends to be exposed at time T ₅, and the last row of pixels of the image sensor array ends to be exposed at time T ₆.

In an embodiment of the present disclosure, if the image sensor supports the external global reset function, the illumination method may also calculate and obtain, according to the working environment of the device and the characteristics of the device, the illumination sequence information in the previous example, and also calculate and obtain the time 312 for releasing global reset of the image sensor array, so that "motion ghost" generated before the global reset 312 may be further eliminated, and the requirement of "illumination on time" is consistent in time (i.e. the requirement of "condition one" is satisfied), that is, the global reset may be released while illumination is turned on.

In another embodiment of the present disclosure, more precise illumination sequence information may be further obtained by pre-calibrating the range of locations where the graphical indicia 304 appear in the target field of view. The invention relates to a method for selecting an illumination system, which comprises a light source type, a light source number and a light source angle, and a control method of the illumination system, wherein the illumination brightness, the opening time, the closing time and the duration period are not isolated, but are closely related to and linked with the working scene, the lens characteristic and the rolling shutter image sensor characteristic of imaging equipment. This is also the key to the present disclosure. The specific selection method comprises the following steps:

When the expected working scene of the imaging device is to sample an object moving at a high speed, since V _s is large, T _f must be extremely small to eliminate the jelly effect, and according to the formula, when the device is required to be selected, a small aperture and a parasitic light filter (such as a narrow-band filter) are selected so that the influence of ambient light on the sampling process is negligible, the formula can be simplified as follows:

Q ≤ L_f*T_f*(F_IL/F²)*α

The minimum illumination intensity of the flash lamp can be obtained by the following steps:

L_f≥ (V_s/D_p)*Q*F²/(F_IL*α)

And the maximum illumination duration of the flash lamp should satisfy:

T_f≤ D_p/V_s

In addition, the exposure time T _exp of the CMOS image sensor should satisfy:

T_exp≥ T_f+ N_row*T_gap

wherein N _row is the number of rows (e.g., 3840) of pixels of the selected CMOS image sensor, and T _gap represents the interval between two adjacent rows of pixels for starting exposure;

then, other optional components are selected to form the lighting system according to actual requirements.

In contrast, when the imaging device is used to sample an object that is stationary or moving at a low speed in the expected working scenario, since V _s is very small, the time range of T _f can be set relatively large, which allows for the selection of a large aperture and a high lens to promote the "contribution" of ambient light to the sampling during the device type selection according to the formula, thereby reducing the flash illumination time T _f, reducing the flash illumination brightness L _f, and achieving the effect of reducing the power consumption and extending the life of the flash. In addition, the exposure time T _exp of the CMOS image sensor should be set to be about equal to or less than the flash illumination time T, i.e

T_exp≤T_f。

Fig. 13 shows a schematic view of a rolling shutter image sensor exposure using only a specific vanishing lens and using both a specific vanishing lens and a interlockable illumination subsystem, respectively.

The image sensor array is provided with a plurality of pixel units, wherein the pixel units are arranged on the image sensor array, the target image is moving rightwards at a uniform speed, the exposure sequence of the image sensor array is from top to bottom, the time consumed by each row of pixels in the image sensor for performing reset (with the state marked as R) and readout (with the state marked as T) operation is the same and is one unit time, and the readout state T is a general finger set and comprises a plurality of steps such as charge transfer, ADC conversion, digital signal output and the like;

The method for controlling the rolling shutter image sensor is used for configuring the exposure (state is marked as E) time length of each row of pixels of the image sensor to be thirteen unit time, the illumination linkage method is used for controlling the illumination subsystem to start illumination at the moment T ₁₃ for one unit time length, the illumination intensity meets the requirement that the image sensor accumulates enough charges in one unit time length, and meanwhile, the upper edge part and the lower edge part, namely the first row and the last row, are simplified, and only the pixel array containing a target image is concerned.

The upper diagram of fig. 13 shows the phenomenon that the image sensor array cannot receive enough reflection of light during exposure using only the stray light removing lens when external illumination is not present, resulting in blurring of the image. The lower diagram of fig. 13 shows the phenomenon when the specific stray light removing lenses and the interlockable illumination subsystem, since it can be seen that since the specific stray light removing lenses are used, even if all the sensor rows between 204 and 226 enter the exposure state in turn in the period between T ₀ and T ₁₂, since the illumination is not turned on, the pixels in all the sensor rows between 204 and 226 receive only weak reflection from the target image and are converted into a very small amount of charge to be stored. At time T ₁₃, the illumination subsystem accurately starts illumination and is accurately turned off after a unit time, at this time, each pixel point in all sensor rows between 204 and 228 is in an exposure state and can receive the most intense radiant light from the target image and convert the radiant light into a corresponding quantity of charges to be accumulated, so that the effect that all pixel points in the image sensor array 200 start exposure to the target image simultaneously is equivalently realized in the remaining exposure time period at time T14 and after, because the illumination is already turned off, each pixel point in all sensor rows between 206 and 228, which have not been exposed yet, can only receive weak reflection light from the target image and convert the weak reflection light into a very small quantity of charges to be accumulated. The charge amount stored in each pixel point in the image sensor array 200 is mainly from exposure at time T ₁₃, and is reflected in the finally acquired image data, namely, the target image data without jelly effect (shown as a dark image in the lower graph of fig. 13) is obtained at time T ₁₃, and meanwhile, the interference of motion ghost to the original image (shown as a light image in the lower graph of fig. 13) is greatly reduced or even basically eliminated.

Fig. 14 depicts a system schematic of an apparatus for illuminating an imaging device or method using the present invention.

The system is configured to detect and identify graphical indicia in the target image. The control system 404 may flexibly calibrate the operating parameters of the rolling shutter image sensor 402 and the illumination system 406 based on the characteristics of the lens system 408, such as the type of illumination source, the on and off times of the illumination source, the intensity of the illumination source, the cycling sequence of the illumination source, the concomitant status of the aiming light source, and the global reset status of the image sensor. The "no jelly effect" window period with the roller shutter image sensor 402 exposure is ensured that illumination is initiated by the illumination light source 428. So that the system can drive the rolling shutter image sensor to clearly capture one or more graphical indicia and perform machine recognition of subsequent graphical indicia.

One or more lighting systems 406 may be included in the system. Each illumination system 406 may or may not include an aiming light source 426 and an aiming light filter 422 to produce a dot, cross-line, rectangular frame illumination pattern visible to the human eye as an aiming mark 444 that identifies the center of field of view of the lens system 408. If the system comprises a plurality of illumination systems 406, typically only one illumination system 406 will comprise an aiming light source 426 and an aiming light filter 422. Each illumination system 406 may include one or more illumination sources 428 and an illumination light filter 424, the illumination sources 428 providing illumination 446 for the target image and the graphic indicia to be detected and identified contained therein under the control of an illumination synchronization controller 430, and flash illumination at specific times during exposure of the image sensor 402, ensuring that the target image area is adequately illuminated. In one embodiment of the present disclosure as shown, illumination source 428 and illumination light filter 424 and its illumination system 406 may be integrated within device housing 448 along with other subsystems, such that the system is presented in an integrated form.

In another embodiment of the present disclosure, the illumination source 428 and the illumination light filter 424 and their illumination system 406 may not be integrated within the device housing 448 and packaged separately, such that the imaging system is presented in a distributed fashion, and the illumination distance and angle of the illumination system 406 relative to the target image 412 may be flexibly adjusted depending on the characteristics of the working scene and the target image 412.

The lighting system 406 also includes a lighting synchronization controller 430.

The functions of the illumination synchronization controller 430 in one embodiment of the present disclosure are implemented by a separate Microprocessor (MCU), and the functions of the illumination synchronization controller 430 in another embodiment of the present disclosure are implemented by a field programmable logic array (FPGA);

In another embodiment of the present disclosure, the functions of the lighting synchronization controller 430 may be implemented by the cooperation of subsystems in the control system 404. The illumination synchronization controller 430 may obtain control signals or control instructions from the control system 404 or the image sensor 402 from the communication interface 432 to generate illumination sequence information, which may directly control the on and off states of the aiming light source 426 and the illumination light source 428, and the illumination intensity in the on illumination state. The control system 404 or the image sensor 402 then enables indirect control of the light sources required for exposure via the illumination synchronization controller 430 so that the illumination system 406 can provide more efficient illumination, including illumination of sufficient intensity at the necessary times of the exposure period of the rolling shutter image sensor 402 and immediate illumination off at unnecessary times of the exposure period.

The illumination sequence information generally comprises two parts, one part describing the specific behavior of this illumination sequence, typically issued by the control system 404 based on the device characteristics of the image sensor 402 and the lens system 408, pre-stored in the illumination synchronization controller 430 in the form of circuitry or instructions.

In one embodiment of the present disclosure, the illumination synchronization controller may be configured to generate an illumination behavior of turning on illumination for 400 microseconds after receiving the illumination trigger signal, the illumination intensity being 85% of the maximum power of the illumination light source, and turning off the illumination after the illumination lasts for 30 microseconds;

In another embodiment of the present disclosure, the illumination synchronization controller may be further configured to generate an illumination behavior of turning on illumination immediately after receiving the illumination trigger signal without delay, the illumination having an illumination intensity of 50% of a maximum power of the illumination light source, and turning off the illumination after the illumination lasts 100 microseconds;

in another embodiment of the present disclosure, the illumination synchronization controller may be further configured to delay and immediately turn on the illumination after the first receipt of the illumination trigger signal, the illumination intensity is 30% of the maximum power of the illumination light source, turn off the illumination after the illumination lasts for 100 microseconds, and subsequently increment the illumination intensity by 10% of the maximum power each time the illumination trigger signal is received until the maximum power is reached, and stop the cycle;

In another embodiment of the present disclosure, the illumination synchronization controller may be further configured to delay turning on illumination for 20 microseconds after the first receipt of the illumination trigger signal, the illumination intensity is 90% of the maximum power of the illumination light source, turn off illumination after the illumination lasts for 100 microseconds, and increase the delay turning on illumination for 20 microseconds each time the illumination trigger signal is received until reaching a preset maximum illumination delay starting threshold value, and stop the cycle;

In another embodiment of the present disclosure, the illumination synchronization controller may be further configured to perform an illumination behavior of turning on illumination for 400 microseconds after the first receipt of the illumination trigger signal, turning off the illumination after the illumination intensity is 85% of the maximum power of the illumination light source for 10 microseconds, and increasing the illumination duration for 5 microseconds each time the illumination trigger signal is received until reaching a preset maximum illumination duration threshold value, and stopping the cycle;

In another embodiment of the disclosure, the illumination synchronization controller may be further configured to generate an illumination behavior of presetting a number of illumination sub-sequences not exceeding a system limit number, wherein each sub-sequence may have a completely or partially different configuration regarding a start illumination delay, an illumination power, an illumination duration;

another part of the illumination sequence information is an illumination trigger signal, which determines the timing of the start-up execution of the aforementioned various illumination behaviors, and which is closely related to the exposure behavior of the image sensor 402, thus requiring extremely high delay characteristics.

In one embodiment of the present disclosure, the illumination trigger signal is derived from a control signal output from the image sensor 402, and may be edge trigger or level trigger;

In another embodiment of the present disclosure, the illumination trigger signal is an edge trigger or a level trigger from a control signal output from the trigger unit 438 or the processor unit 436 in the control system 404.

The lighting system 406 also includes a communication interface 432 for coupling the lighting system 406 to the system so that it can receive control signals from the control system 404 and the image sensor 402. The communication interface 432 is applicable to both embodiments of the integrated form of the system and embodiments of the integrated form of the system. The communication interface 432 includes at least two forms, one is a bus interface based on some encoding protocol, such as PWM, I2C, SPI, and the other is an edge-triggered or level-triggered signal interface.

The system includes a lens system 408, where the lens system 408 is configured to accurately transmit the reflected light of the target image to the image sensor 402 for exposure sampling, and to adapt the characteristics of the illumination system 406 and the image sensor 402 to minimize the transmission of ambient light through the lens 418 to the image sensor 402 except for the illumination from the illumination source 428.

In one embodiment of the present disclosure, a lens system 408 is used that manually focuses and zooms the aperture, in which the focal length and aperture parameters are manually set by the staff on the site of installing the system according to the working environment, the lighting system 406 and the image sensor 402 characteristics to achieve the goal of accurately transmitting the target image and eliminating the stray light;

In another embodiment of the present disclosure, a lens system 408 with configurable focusing and aperture zoom is selected, in this embodiment, a worker or a user can flexibly set the focal length and aperture parameters by using instructions, circuits, logic through the control system 404 to achieve the objective of accurately transmitting the target image and eliminating the flare, wherein the focusing actuator is borne by a Voice Coil Motor (VCM);

In another embodiment of the present disclosure, the focusing device may be a piezoelectric MEMS autofocus actuator (TLens) that is smaller in size, consumes less power, and is faster.

In another embodiment of the present disclosure, a liquid lens may be used to further increase the response speed of auto-focusing;

in another embodiment of the invention, a bandpass filter lens can be added in front of the lens or a bandpass filter coating can be added on the lens to realize the aims of accurately transmitting the target image and eliminating stray light;

In another embodiment of the invention, a polarized lens can be added in front of the lens or a polarized coating film can be added on the lens to realize the aims of accurately transmitting a target image and eliminating stray light;

In another embodiment of the present disclosure, an external mechanical shutter may be additionally integrated into lens system 408, which may be synchronized with the illumination control in illumination system 406, such that when illumination source 428 is on, the mechanical shutter is on, image sensor 402 may receive light from the lens system, and when illumination source 428 is off, the mechanical shutter is off, and image sensor 402 may not receive light from the lens system, the control sequence being as shown in FIG. 12.

An image sensor 402 is included in the system to transfer a lens system 408 to an image of a target on an image sensor array for acquisition and to store the acquired image data in a memory unit 434. The image sensor 402 may be a Complementary Metal Oxide Semiconductor (CMOS) image sensor using a rolling shutter, the parameter configuration of the image sensor 402 may be issued by the processor unit 436 in the control system 404, the action of exposing may be triggered by the processor unit 436 or the trigger unit 438, and the processor unit 436 or the trigger unit 438 may adjust the exposure parameters of the image sensor 402 simultaneously with the adjustment of the illumination parameters.

In one embodiment of the present disclosure, a conventional rolling shutter Complementary Metal Oxide Semiconductor (CMOS) image sensor may be used.

In another embodiment of the present disclosure, an improved Complementary Metal Oxide Semiconductor (CMOS) image sensor with a rolling shutter having a global reset function may be used;

In another embodiment of the present disclosure, a Charge Coupled Device (CCD) image sensor may be used.

The system includes a control system 404, which is complex and includes all of the remaining subsystems for supporting the lens system 408, the illumination system 406, and the image sensor 402, and generally includes a processor unit 436, a memory unit 434, a trigger unit 438, an external interface unit 440, and an internal interface unit 442.

The control system 404 includes a memory unit 434, and the memory unit 434 includes a storage control subunit and a storage medium subunit.

In one embodiment of the present disclosure, the storage unit 434 may be used to store raw data acquired by the image sensor 402, process data and final result data calculated by the processor unit 436, instruction codes executable by the processor unit 436, configuration files required for operating subsystems such as the lighting system 406, and electronic data required for normal operation of any system such as BIT/HEX engineering files that can be loaded by the FPGA in the processor unit 436, and tamper-proof critical security data of the system. The storage unit 434 includes Random Access Memory (RAM), double data rate random access memory (DDR), static Random Access Memory (SRAM), read Only Memory (ROM), electrically erasable programmable read only memory (E2 PROM), one time programmable memory (OTP), magnetic disk storage device (HDD), solid state storage device (SSD/FLASH), and any other type of storage medium suitable for the system to store necessary information.

The control system 404 includes a processor unit 436, the processor unit 436 providing the system with the ability to calculate analysis data and control other subsystems.

The processor units in one embodiment of the present disclosure include reduced instruction set processors (RISC), complex instruction set processors (CISC), microprocessors, embedded processors, co-processors, microcontrollers, image processing application specific integrated circuits (ISPs), neural Processing Units (NPUs), and associated discrete devices such as capacitors, resistors, inductors, and application specific ICs.

In another disclosed embodiment, the processor unit includes a system on a chip (SOC) and associated discrete devices such as capacitors, resistors, inductors, and application specific ICs.

In another embodiment of the present disclosure, the processor unit includes, in addition to the aforementioned devices, devices such as a Field Programmable Gate Array (FPGA), a digital signal processing unit (DSP), and a Programmable Logic Device (PLD). The processor unit 436 includes, in addition to the aforementioned hardware components, software executed by the aforementioned hardware components, including combinations of system programs, application programs, operating systems, firmware, application program interfaces, middleware, code, variables, methods, and data structures. The processor unit 436 may flexibly adjust the specifications according to the performance requirements of the system, the power consumption level, the bus topology, the traffic characteristics, and the different features of the image sensor 402, the lens system 408, and the illumination system 406.

The control system 404 includes one or more trigger units 438, and the trigger unit 438 can sense the basic motion information of the target image and the carrier containing the target image and the motion state information of the code reading device itself in an automatic or manual mode, and report the information to the processor unit 436 in real time, and the latter makes a decision whether to start or not and starts the acquisition process and the mark recognition process once or several times according to what configuration parameters.

In one embodiment of the present disclosure, the trigger unit 438 may include a physical key and a set of speed and acceleration sensors, and an operator generates a trigger signal through the physical key to inform the processor unit 436 to start the collection process and the mark recognition process once, and after receiving the trigger signal, the processor unit 436 determines, according to the motion information of the device reported by the sensor in the trigger unit 438 and the preset configuration information in the memory unit 434 at the same time, which specific configuration is used to start the collection process and the mark recognition process.

In another embodiment of the present disclosure, the trigger unit 438 may include a set of photo-sensors and distance sensors that automatically generate one or more trigger signals to inform the processor unit 436 to initiate one or more acquisition processes and tag identification processes if a pre-configured condition is met.

In another embodiment of the present disclosure, the trigger unit 438 may further include a software and hardware module carrying a specific algorithm, for example, a quadrilateral detection algorithm, a contour detection algorithm, and a pattern matching algorithm, where the trigger unit 438 may periodically start "pre-sampling" to shift the target image, and after the initial determination of the angle information, report the trigger signal and the motion parameter of "formal sampling" to the processor unit 436, and after the processor unit 436 receives the trigger signal, accurately correct the preset configuration information in the memory unit 434 according to the shift, position, and angle information of the target image obtained in the pre-sampling, and then start a standard acquisition procedure and a mark recognition procedure.

The function of the trigger unit 438 in another embodiment of the present disclosure may also be implemented by the processor unit 436, where the processor unit 436 controls the operation of the illumination system 406, the image sensor 402, and the lens system 408 entirely, and the device may be configured to repeat the exposure and illumination process continuously in a certain period.

The control system 404 includes one or more external interface units 440, the external interface units 440 being configured to connect the system to external peripherals,

One or more external interfaces may be supported in one embodiment of the present disclosure, and communication protocols supported by the interfaces should include, but are not limited to, ethernet protocols (IEEE 802.3 series), industrial ethernet protocols, fifth generation mobile communication technology protocols (3 GPP series), wireless local area network protocols (IEEE 802.11 series), bluetooth protocols (Bluetooth), near field communication protocols (NFC), serial universal bus (USB), high Definition Multimedia Interface (HDMI), DP interface, and RS232, I2C, and SPI.

In another embodiment of the present disclosure, the external interface unit further includes a user interface, which may be implemented based on one or more protocols of the foregoing embodiments, through which an operator may write data and configuration information to the system, or read data and status information, and the user interface includes various input and output devices such as a keyboard, buttons, knobs, touch screen, and display screen.

In another embodiment of the present disclosure the user interface also includes the system itself, i.e., an operator can write data or configuration to the system by sampling graphical indicia containing data information or configuration information through the system.

The control system 404 includes one or more internal interface units 442 that provide control and information and image data transfer paths between various subsystems (image sensor 402, control system 404, illumination system 406, lens system 408, power supply system 410) within the system, which may include interfaces for one or more communication protocols.

In one embodiment of the present disclosure, a Network On Chip (NOC) adapted for SOC internal connections and a serial peripheral bus (SPI) adapted for board level circuitry are used together.

The system includes a power supply system 410 to provide power to the overall system,

In one embodiment of the present disclosure, the power supply system 410 supports the use of an external ac power adapter to power the system;

in another embodiment of the present disclosure, power system 410 uses POE (Power over Ethernet) power technology to power the system while transmitting data signals over an ethernet cable;

In another embodiment of the present disclosure, the power supply system 410 uses a USB interface to supply power to the system;

In another embodiment of the present disclosure, the power supply system 410 may use a USB interface and a built-in battery to supply power to the system, when the USB interface is detected to be connected, the USB interface is preferentially used to supply power to the system, and when the USB interface is detected to be unconnected, the built-in battery is used to supply power to the system;

in another embodiment of the present disclosure, the power supply system 410 uses amorphous silicon solar cells and built-in batteries to power the system.

The system shown in fig. 14 is an exemplary system for describing the manner in which one of the devices using the illumination imaging methods described in this disclosure may be constructed, wherein the various subsystems may be implemented in any combination of various general purpose/application specific integrated circuit, discrete device and PCB based hardware and software executing in various processor, microcontroller and field programmable arrays.

The architecture of the system shown in fig. 14 does not refer to the manner in which hardware and software in a real imaging device are partitioned, and the manner in which subsystems in a real imaging device are partitioned, combined, and connected is not limited to the exemplary structure in fig. 14.

The apparatus shown in fig. 15 is a schematic diagram of one application form of the system.

The device can be applied to imaging and identification of various image marks in the scenes of logistics express sorting, automobile complete machine manufacturing, 3C electronic assembly line and the like.

The device is provided with a metal housing 502 and the back of the housing includes four securing pegs 518. The metal housing 502 and the securing pegs 518 may effectively protect and secure the device in an industrial setting.

The lighting system of the device comprises a lighting synchronous controller implemented in a field programmable array (FPGA), a communication interface implemented in an I2C bus and general purpose input output pins (GPIOs), and a plurality of light sources. The light source includes four components, direct illumination subsystem 508, polarized illumination subsystem 504, uniform illumination subsystem 510, and aiming photonic system 524.

The lens system of the device consists of a piece of 12 mm lens 506, an ND reduction lens, and a manual focus knob 516.

The external interface unit of the device includes a first aviation plug 514 supporting 100mbps ethernet protocol, a second aviation plug 512 supporting 24 vdc power, RSR232, polarity-free isolation IO, status display lights 520, debug buttons 522.

The part of the device interior not visible in the figure also includes an SOC based on RISCV instruction set as a processor unit, DDR and NOR-FLASH devices as memory units, and a Complementary Metal Oxide Semiconductor (CMOS) image sensor of an 800 ten thousand pixel rolling shutter.

The workflow shown in fig. 16 is an example of a workflow of the apparatus of fig. 15 that requires an operator to manually calibrate parameters and adjust to optimal conditions depending on the production environment and the equipment's choice. In step 602, the device and the peripheral equipment are installed first, including fixing the device in a proper position and adjusting the focal length through a focusing knob 516, connecting a photoelectric sensor through a nonpolar isolation IO to sense the position of a sampling target and trigger the device to start an exposure sampling and graphic mark recognition process, connecting a master control computer through an RS232 and ethernet interface to issue configuration parameters to the device and receiving the recognition result of the graphic mark reported by the device. In step 602, according to the production environment characteristics (such as the moving speed of the conveyor belt of the assembly line, the environmental illumination type and intensity, the reflection type of the object surface carrying the graphic mark to be identified, the relative position and coding type of the graphic mark to be identified, etc.), and the self characteristics of the device (such as the color of the selected flash lamp, etc.), the adaptive image sensor exposure control parameters and illumination sequence information parameters including the sampling frame number, the exposure time of the image sensor, which illumination light source or illumination light sources are enabled, the starting time and the stopping time of each illumination light source are generated according to the illumination imaging method described above, and the initialization of the device is completed according to the parameters, at this time, the device enters a working state. In step 604, when the external photoelectric sensor detects that the object carrying the graphic mark to be identified reaches a specified position along with the conveyor belt, a trigger signal is immediately reported to the device.

After the device receives the trigger signal, each subsystem inside is controlled to be orderly started according to the configuration parameters debugged in the step 602, one or a plurality of sampling imaging of the target graphic mark is completed, the device comprises starting a rolling shutter of the CMOS image sensor to perform progressive exposure in the step 606, starting the appointed lighting units at the appointed moment in the step 608 according to a preset strategy, closing all the lighting units at the specific moment in the step 610, completing the exposure of the CMOS image sensor in the step 612, and reading the acquired image data to the control and calculation subsystem. In the following step 614, the control and computing subsystem in the device performs image processing and decoding computation on the original image data, and reports the identification result of the graphic mark to the host computer.

The workflow shown in fig. 17 is another workflow of the apparatus of fig. 15, in which the apparatus attempts to combine various image sensor exposure control parameters and illumination sequence information parameters according to a certain policy by itself, finally obtains the optimal configuration parameters in the production environment, records the set of parameters, and completes the initialization of the apparatus. Compared with the flow, the flow can simplify the operation complexity of the device and promote the usability. Step 704 in the process is an operation that may need to be cycled multiple times, each operation adjusts one or more image sensor exposure control parameters and illumination sequence information parameters according to a certain policy, starts a complete sampling and decoding process of the graphic mark according to the parameters, and enters the next sampling and decoding after adjusting the parameters if the decoding fails. If decoding is successful, the group of parameters is recorded and a score is generated, and when all the parameters are traversed or the maximum threshold time of one-key debugging is reached, the device uses the recorded parameters with the highest scores as working parameters in the production environment. Step 708 includes the image sensor starting exposure, the illumination unit turning on the flash, the illumination unit turning off the flash, and the image sensor ending exposure, similar to steps 606, 608, 610, 612 in the flowchart.

The invention skillfully sets each link of system construction and configuration, including the configuration method of the image sensor array, the imaging lens system, the illumination system and the control and calculation subsystem, has strong operability, and has definite steps of triggering sampling, illumination imaging and image post-processing and decoding, thereby being capable of realizing acquisition, processing and identification of target mark images and improving the accuracy of mark identification.

Aiming at the problem of low mark recognition accuracy caused by the jelly effect of a rolling shutter image sensor in the prior art, the scheme provides a complete illumination imaging method, effectively solves the jelly effect problem through system configuration and accurate execution of each step, and greatly expands the application range of the rolling shutter image sensor in the field of mark recognition.

Application scenario one, parcel tracking in logistics transportation scenario

In a large logistics warehouse, a large number of package warehouse-in, sorting and warehouse-out operations are required to be processed every day. In order to realize efficient and accurate parcel tracking, the illumination imaging device for identifying the marks, which is suitable for the rolling shutter, is introduced.

The device is built by selecting an imaging device with an integral fixed shape and structure, and the structure can be firmly arranged at a specific position of a logistics assembly line, thereby playing a good role in fixing and protecting the device. The imaging device comprises a CMOS image sensor array using a rolling shutter technique of exposing line by line and resetting line by line, and can efficiently collect the mark image on the package. The imaging lens system adopts a lens main body to match at least one lens, is provided with a band-pass filter and a polarization filter, can effectively guide reflected light wrapping the marked image to be transmitted to the image sensor array, and simultaneously reduces parasitic light interference. The lighting system is integrated in the device and is presented in an integrated form, and comprises an aiming light source, an accessory filter lens, an exposure light source, an accessory filter lens, a light source synchronous controller and a communication interface. The control and calculation subsystem consists of a processor unit, a memory unit, a trigger unit, an internal interface unit and an external interaction unit, and the power supply system adopts POE (Power over Ethernet) mode to stably supply power to the whole device. The components are mounted and connected by a substrate.

And (3) a working procedure, namely when the package is transported to the visual field of the imaging device on the assembly line, automatically reporting the package to the processor unit and simultaneously starting the exposure and illumination procedure by detecting the package by the light intensity detection sensor in the trigger unit. The processor unit controls the illumination system, and the light source synchronous controller generates a high-level signal which is modulated and is linked with the exposure action of the rolling shutter image sensor array as an illumination switch control signal. The signal precisely controls the exposure light source to start illumination at a designated moment after the first row of the rolling shutter image sensor array starts exposure, and to close illumination at a designated moment before the last row of the rolling shutter image sensor array ends exposure, and the illumination intensity is adjusted according to environmental parameters and package mark characteristics. The imaging lens system automatically adjusts the aperture to accurately guide and transmit reflected light of the package marking image to the image sensor array according to the working environment of the equipment. After the image sensor array completes image acquisition, the control and calculation subsystem performs image processing, identification and decoding on the image information obtained by exposure, identifies various types of two-dimensional code or one-dimensional code information on the package, and reports the identification result to the logistics management system through an Ethernet protocol of an external interaction unit, so that real-time tracking of the package is realized.

Display dead pixel detection in industrial production scene

In a display production factory, defective pixel detection is required to be carried out on a produced display so as to ensure the quality of products. The illumination imaging apparatus and method of the present invention play an important role in this scenario.

The device is selected and configured by adopting the imaging device with a split-type appearance and structure, wherein the imaging device is provided with a threaded interface (such as a C interface) and can be used for replacing a proper lens according to detection requirements of different displays. The image sensor array adopts a CMOS image sensor array which uses row-by-row exposure and has a rolling shutter technology with a global reset function, so that the accuracy of image acquisition is improved. The imaging lens system can flexibly adjust the focal length and the aperture through a focal length adjusting device (adopting Voice Coil Motor (VCM) focusing technology) and an aperture adjusting device, accurately guide reflected light on the surface of a display to an image sensor array, and reduce the interference of ambient stray light. The lighting system is in a distributed form, is packaged separately and communicates with other subsystems through a communication interface. The light source synchronous controller is composed of an FPGA, generates a PWM modulated illumination switch control signal, controls the exposure light source to circularly execute illumination operation according to a preset parameter sequence, and can trigger the synchronous modification of the exposure parameters of the image sensor when the illumination parameters are changed. The processor unit of the control and computing subsystem adopts an image processing application specific integrated circuit (ISP), is matched with related discrete devices, and the memory unit adopts solid state storage equipment (SSD/FLASH) to store original data, process data, result data and the like. The power supply system uses a 12V ac power adapter.

In the detection process, an operator aims the imaging device at a display to be detected, a trigger unit is manually triggered through an entity key, and a report processor unit starts exposure and illumination processes. The aiming light source of the illumination system illuminates and indicates the display detection area in the shape of a rectangular frame, and is automatically turned off when the exposure light source is turned on. The exposure light source adopts uniform light and white mixed light, and the light source synchronous controller precisely controls the light intensity, the starting and closing time of the exposure light source according to the parameters such as the brightness, the contrast and the like of the display. The imaging lens system automatically adjusts the focal length and the aperture according to parameters such as the size and the distance of the display, and accurately focuses the reflected light on the surface of the display onto the image sensor array. After the image acquisition is completed, the control and calculation subsystem performs post-processing on the image, performs recognition and analysis on OCR symbols (such as display model identification) and defect matching symbols (dead pixels) by using a specific algorithm, stores detection results in the memory unit, and outputs the detection results to the production management system through a USB interface of the external interaction unit for judging whether the display is qualified or not.

Application scene three panoramic motion shooting in daily outdoor scene

An outdoor sports fan wishes to record a panoramic view of the way and identify some indicators on the road while riding a mountain bike. The illumination imaging apparatus and method of this patent meets his needs.

The device is assembled and arranged, and an imaging device which is suitable for sports shooting, has the advantages of being handheld and fixed on sports equipment and capable of expanding the appearance and the structure is selected. The device is fixed on the handle bar of the bicycle, and the stability of the device in the riding process is ensured through the expandable structure. The image sensor array employs a CMOS image sensor array of rolling shutter technology. The imaging lens system is provided with a liquid lens, can quickly and automatically adjust the focal length, is suitable for shooting requirements of different distances, and reduces environmental stray light through the aperture adjusting device. The lighting system is integrated in the device, and the light source synchronous controller is composed of a plate-level MCU, and generates a negative pulse trigger signal to control the exposure light source. The exposure light source adopts direct light and red visible light, and can more clearly illuminate the indication marks on the road in the riding process. The processor unit of the control and computing subsystem employs a microcontroller, and the memory unit uses Random Access Memory (RAM) for data caching. The power supply system adopts a battery, and is convenient to carry and use.

In the process of riding, a speed sensor and an angle sensor in the trigger unit detect the motion state change of the bicycle, and when a scene needing to be photographed (such as an indication mark) is detected, the characteristics are calculated and analyzed preliminarily and then reported to the processor unit. The processor unit readjusts exposure and illumination configuration parameters according to the reported information, controls the illumination system to start illumination at a designated moment after the first row of the rolling shutter image sensor array starts exposure, and stops illumination at a designated moment after the last row of the rolling shutter image sensor array ends exposure. After the image sensor array collects images, the control and calculation subsystem performs post-processing on the images, attempts to identify and analyze locators and various standard one-dimensional codes (such as scenic spot guide codes) on a road, and simultaneously performs splicing and processing on panoramic pictures, and the processed images are stored in the memory unit for subsequent viewing and sharing by a user.

Application scenario IV digital microscope shooting of marker tracking in scientific research scenario

In biological laboratories, researchers need to track, photograph and analyze markers on biological samples using digital microscopes. The illumination imaging device and the illumination imaging method provide powerful support for scientific research.

The device is adapted and constructed by adopting the imaging device with portable shape and structure, thereby being convenient for scientific researchers to flexibly use in experimental operation. The image sensor array is a CCD image sensor array, and can provide high-resolution images. The imaging LENs system is configured for the optical characteristics of the microscope, and the focal length is precisely adjusted through a T-LENs focusing technology, so that the markers on the biological sample can be clearly imaged on the image sensor array. The illumination system adopts a group of exposure light sources, the light sources are infrared light, the interference on biological samples can be reduced, and the light source synchronous controller is doubled by a processor unit in the control and calculation subsystem to generate a positive pulse trigger signal to control the exposure light sources. The control and computing subsystem adopts a system on a chip (SOC), integrates multiple functions, and the memory unit adopts an electrically erasable programmable read-only memory (E2 PROM) to store experimental related configuration files and key data. The power supply system adopts a USB interface to supply power, and is convenient to connect with laboratory equipment.

And the experimental operation and data processing are that a scientific research personnel installs the imaging device on the digital microscope, and the exposure and illumination flow is started through the manual operation triggering unit. The aiming light source of the illumination system illuminates the marker area on the biological sample in a single point, the exposure light source turns on illumination at a specified time before the first line of the rolling shutter image sensor array starts exposure, and turns off illumination at a specified time before the last line of the rolling shutter image sensor array ends exposure. After the image sensor array collects the image of the marker on the biological sample, the control and calculation subsystem carries out post-processing on the image, a professional image recognition algorithm is used for recognizing, tracking and analyzing the marker (such as a fluorescent marked cell structure), experimental data are stored in the memory unit, and the experimental data are output to the scientific research data management system through an RS232 interface of the external interaction unit, so that accurate experimental data are provided for scientific research personnel.

Two code readers with the same hardware structure are adopted for comparison experiments, wherein one code reader adopts the marking identification illumination imaging method and the other adopts the existing imaging method. The code reader is rocked at the same frequency, so that the code reader and the opposite mark to be identified generate the same relative motion. After one of the code readers finishes reading and identifying for a period of time, the other code reader is replaced at the same position to execute the same operation.

The field photographed image (fig. 18 and 19) shows the experimental results that in fig. 18, when the barcode reader of the present invention is used, although the barcode image on the display screen is blurred due to the rapid movement, the appearance of the green barcode reading mark frame indicates that the barcode reader has successfully read the barcode content and the result is accurate, while fig. 19 shows that the barcode reader adopting the existing imaging method can clearly present the barcode image, but does not appear the green barcode reading mark frame, indicating that the barcode information cannot be identified. Experimental results show that under the same hardware condition and rapid movement scene, the illumination imaging method can effectively overcome motion ghost and jelly effect, and achieve more accurate and efficient bar code reading.

The foregoing is merely exemplary of the present application, and specific technical solutions and/or features that are well known in the art have not been described in detail herein. It should be noted that, for those skilled in the art, several variations and modifications can be made without departing from the technical solution of the present application, and these should also be regarded as the protection scope of the present application, which does not affect the effect of the implementation of the present application and the practical applicability of the patent. The protection scope of the present application is subject to the content of the claims, and the description of the specific embodiments and the like in the specification can be used for explaining the content of the claims.

Claims

1. A lighting imaging device for mark recognition, comprising:

An image sensor array is used to expose and collect target area marks line by line so that each row of pixels is in an exposed state to form a target mark image;

An imaging lens system is used to guide the reflected light of the target mark image to the image sensor array and reduce the interference of other stray light received by the image sensor array except the light from the imaging device's own illumination system;

One or more lighting systems are used to provide specific lighting for the target mark image during a specific window period, so that the exposure intensity of the image pixels during the window period is higher than that during the non-window period;

A set of control and computing subsystems is used to obtain the lighting sequence information of the lighting system according to different application scenarios, control the lighting system to provide illumination for the target mark image, and enable the image sensor equipped with a rolling shutter to simulate a global shutter sensor for imaging;

A power supply system;

A set of substrates is used to mount and connect the image sensor array, imaging lens system, lighting system, control and computing subsystem and power supply system.

2. The imaging device according to claim 1, wherein the image sensor array is a CMOS image sensor array or a CCD image sensor array.

3. The imaging device according to claim 1 is characterized in that the imaging device is used to decode target marks in the collected image data, and the target marks include: OCR symbols, watermark logos, locators, various types of one-dimensional codes, various types of two-dimensional codes, defect matching symbols, and tracking matching symbols.

4. The imaging device according to claim 1, characterized in that the imaging device includes a portable shape and structure suitable for handheld and medical scenarios, and the portable shape and structure are used to fix and protect the imaging device.

5. The imaging device according to claim 1, wherein the imaging device is integrally formed and is suitable for industrial production and logistics transportation;

The imaging device includes a split appearance and structure with a threaded interface and interchangeable lens suitable for industrial production and logistics transportation. The split appearance and structure are used to fix and protect the imaging device, wherein the threaded interface includes: C, CS, F, M12, M42, M58 and M72 interfaces.

6. The imaging device according to claim 1, characterized in that the imaging device includes an expandable shape and structure suitable for sports photography, capable of being both handheld and fixed to sports equipment and sports equipment, and the expandable shape and structure are used to fix and protect the imaging device.

7. The imaging device according to claim 2, wherein the CMOS image sensor array is a CMOS image sensor array using a rolling shutter technology with row-by-row exposure and row-by-row reset;

Alternatively, the CMOS image sensor array is a CMOS image sensor array using a rolling shutter technology that uses row-by-row exposure and has a global reset function.

8. The imaging device according to claim 1, wherein the imaging lens system comprises a lens body and at least one lens element, and optionally includes a bandpass filter, a polarization filter, a focus adjustment device, and/or an aperture adjustment device; the imaging lens system utilizes one or more of T-LENs, a voice coil motor (VCM), and a liquid lens for focusing.

9. The imaging device according to claim 8, wherein the lens body is used to:

In a specific scenario, the aperture is controlled to be as small as possible within the device's scope to reduce the amount of ambient light transmitted to the image sensor array.

Alternatively, the aperture is controlled and adjusted to the F2.3-F16 range allowed by the device, so as to accurately guide the reflected light of the target image to the image sensor array;

Alternatively, an external mechanical shutter can be used in a specific scenario. When the mechanical shutter is closed, it can completely block the reflected light received by the lens.

Alternatively, an optional sunshade can be added for a specific scenario, which can completely block the reflected light received by the lens when the sunshade is closed.

10. The imaging device according to claim 1, wherein the illumination system comprises an aiming light source and an associated filter, an exposure light source and an associated filter, a light source synchronization controller, and a communication interface.

11. The imaging device according to claim 10, wherein the light source synchronization controller is used to generate a modulated lighting switch control signal that is linked to the exposure action of the rolling shutter image sensor array, and the control signal includes: a high-level signal, a low-level signal, a positive pulse trigger signal, a negative pulse trigger signal and/or a PWM modulation signal.

12. The imaging device according to claim 10, wherein the light source synchronization controller is configured to generate a modulated illumination switch control signal that is linked to the exposure action of the rolling shutter image sensor array, thereby precisely controlling the exposure light source; the light source synchronization controller comprises a light intensity control module, an illumination timing control module, an illumination timing mode module, and a cycle and synchronization mechanism module, wherein the light intensity control module is configured to adjust the light intensity during illumination;

The lighting timing control module controls the lighting switch in a specific window period based on four combinations of the lighting time axis and the exposure time axis. During the specific window period of lighting the target mark image, the exposure start time _T1 of the first row, the exposure start time _T2 of the Nth row, the lighting on time _T3 , the lighting off time _T4 , and the exposure end time _T5 of the first row must meet the following conditions:

Time length limit: T ₄ - T ₃ ≤ D _p /V _s , D _p is the maximum pixel offset, V _s is the motion speed;

Opening time limit: T ₃ ≥ T ₂ ;

Closing time limit: T ₄ ≤ T ₅ ;

Row-column time relationship: T ₂ = T ₁ + N _row *T _gap or T ₂ = T ₁ + T _readall , where N _row is the number of vertical pixel rows, T _gap is the time interval between exposure or reading of two adjacent rows, in seconds, and T _readall is the total time required to complete reading of all rows of a frame of image;

The lighting timing mode module includes four lighting timing modes:

First lighting timing mode: lighting is turned on earlier than the first row exposure start time, and lighting is turned off earlier than the last row exposure end time;

Second lighting timing mode: lighting is turned on later than the first row exposure start time, and lighting is turned off later than the last row exposure end time;

The third lighting timing mode: the lighting is turned on earlier than the first row exposure start time, and the lighting is turned off later than the last row exposure end time;

Fourth lighting timing mode: the lighting is turned on later than the first row exposure start time, and the lighting is turned off earlier than the last row exposure end time;

The loop and synchronization mechanism module executes cyclically according to the parameter sequence corresponding to the four preset lighting timing modes; when the lighting parameters change, it automatically triggers the synchronous modification of the exposure parameters of the image sensor.

13. The imaging device according to any one of claims 10 to 12, characterized in that the light source synchronization controller is used to generate a modulated lighting switch control signal that is linked to the exposure action of the rolling shutter image sensor array, and the control signal controls the exposure light source to illuminate with different brightness; the light source synchronization controller is composed of one or more of the following: an FPGA, a board-level MCU, and a processor unit in the control and computing subsystem.

14. The imaging device according to any one of claims 10 to 12 is characterized in that the light source synchronization controller is used to generate a modulated control signal that is linked to the exposure light source, and the control signal accurately controls the switching of the aiming light source, and enables the aiming light source to: illuminate and indicate the marking area in the shape of a single point, double points, a single straight line, a cross line and/or a rectangular frame; automatically turn off when the exposure light source is turned on; and remain on when the exposure light source is turned on.

15. The imaging device according to claim 10, characterized in that the lighting system includes one or more groups of exposure light sources, different groups of exposure light sources allow the use of different types of light sources, and different groups of exposure light sources are independently controlled by a light source synchronization controller; each group of exposure light sources can select the same or different specifications: the light type includes direct light, uniform light, and polarized light; the color includes red visible light, blue visible light, infrared light, and white mixed light.

16. The imaging device according to claim 1, wherein the control and computing subsystem comprises a processor unit, a memory unit, a trigger unit, an internal interface unit, and an external interaction unit.

17. The imaging device according to claim 16, wherein the trigger unit is a set of hardware and software modules comprising multiple digital sensors and equipped with specific algorithms including a shadow compensation algorithm, a position prediction algorithm, and a motion artifact quantification algorithm; the multiple digital sensors include: a light intensity detection sensor, a velocity sensor, and an acceleration sensor; the sensors can detect the device's own motion characteristics and environmental characteristics of the device's environment that affect image sampling, and after preliminary calculation and analysis of the characteristics, report them to the processor unit. The processor readjusts exposure and lighting configuration parameters based on the reporting unit before restarting the exposure and lighting process; wherein the specific detection range data of each sensor is as follows:

Light intensity sensor: 1-65535lux,

Speed sensor: 0-10m/s,

Accelerometer: ±1G.

18. A lighting imaging method for marker recognition, characterized in that by providing specific lighting to the target marker image during a specific window period, the exposure intensity of the target marker image pixel is greater than the exposure intensity during a non-window period, so that the image sensor equipped with a rolling shutter simulates a global shutter sensor for imaging.

19. The illumination imaging method according to claim 18, characterized in that it comprises the following steps:

Step 1: System construction and configuration;

Image sensor array configuration: Use a rolling shutter image sensor array and set its exposure mode to row by row, ensuring that there is a specific window of exposure between each row of pixels to collect the target area mark and prepare for forming the target mark image;

Imaging lens system configuration: Select an imaging lens system and configure its parameters so that it can accurately guide the reflected light of the target mark image to the image sensor array. At the same time, by selecting a specific stray light elimination lens, adding a bandpass filter lens or a polarizing lens, the image sensor array can be reduced from any stray light interference other than that from the imaging device's own illumination system.

Lighting system configuration: Determine the use of one or more lighting systems based on actual needs; configure the lighting system and control the light source type, light intensity, lighting duration, and lighting synchronization to provide specific lighting for the target mark image during a specific window period to ensure that the exposure intensity of the target mark image pixel is greater than the exposure intensity of other pixels;

Control and computing subsystem configuration: The entire configuration and usage of the rolling shutter image sensor array, imaging lens system, and lighting system are encapsulated into a program, which is scheduled and executed by the control and computing subsystem. The control and computing subsystem has the ability to obtain lighting system lighting sequence information based on different application scenarios and can calculate the time to release the global reset of the image sensor array based on the lighting information.

Step 2: Trigger sampling; trigger the sampling target mark and transmit the signal to the control and calculation subsystem;

Step 3: Illumination imaging;

The control and computing subsystem controls the lighting system. During the exposure process of the rolling shutter image sensor array, the lighting system is turned on according to pre-configured lighting sequence information during a specific window period to provide specific lighting to the target mark image, so that the target mark image pixels obtain sufficient and appropriate exposure intensity. The exposure intensity of the image pixels during the window period is higher than that during the non-window period, thereby forming the target mark image.

Step 4: Image post-processing and decoding;

The control and computing subsystem performs post-processing on the image information collected by the image sensor array, including image denoising, contrast enhancement, and color correction;

Decode the post-processed image and identify the marker information therein.

20. The illumination imaging method according to claim 19, wherein in the illumination system configuration of step 1, multiple sets of preset illumination configurations are used, including:

Sequential execution of preset lighting configurations within one exposure cycle;

Sequentially executing a preset lighting configuration over multiple exposure cycles;

Execute a specified preset lighting configuration within one exposure cycle;

Alternatively, a specific preset lighting configuration may be executed over multiple exposure cycles.

21. The illumination imaging method according to claim 20, wherein the illumination synchronization controller can be configured to generate the following illumination behavior:

After receiving the lighting trigger signal, the lighting is turned on with a delay of 400 microseconds. The lighting intensity is 85% of the maximum power of the lighting source. The lighting lasts for 30 microseconds and then turns off.

Alternatively, upon receiving the lighting trigger signal, the lighting is immediately turned on without delay, the lighting intensity is 50% of the maximum power of the lighting source, the lighting lasts for 100 microseconds, and then the lighting is turned off;

Alternatively, after the first lighting trigger signal is received, the lighting is immediately turned on with a delay, the lighting intensity is 30% of the maximum power of the lighting source, the lighting lasts for 100 microseconds, and then the lighting is turned off. Each subsequent lighting trigger signal is received, the lighting intensity is increased by 10% of the maximum power until the maximum power is reached and the cycle is stopped;

Alternatively, after receiving the lighting trigger signal for the first time, the lighting is turned on with a delay of 20 microseconds, the lighting intensity is 90% of the maximum power of the lighting source, the lighting lasts for 100 microseconds, and then the lighting is turned off. Each subsequent lighting trigger signal is received, the lighting is turned on with an additional 20 microsecond delay until the pre-configured maximum lighting delay start threshold is reached and the cycle stops;

Alternatively, after receiving the lighting trigger signal for the first time, the lighting is turned on with a delay of 400 microseconds, the lighting intensity is 85% of the maximum power of the lighting light source, the lighting is turned off after 10 microseconds, and the lighting duration is increased by 5 microseconds each time the lighting trigger signal is received subsequently until the pre-configured maximum lighting duration threshold is reached and the cycle stops;

Alternatively, multiple lighting subsequences not exceeding the system limit are preset, wherein each subsequence can have completely different or partially different configurations regarding the lighting start delay, lighting power, and lighting duration; and each time a lighting trigger signal is received, one of the preset lighting subsequences is executed in sequence.

22. The illumination imaging method according to claim 19, wherein in step 1, the control and computing subsystem configuration, wherein the control and computing subsystem obtains the contents of each preset configuration, comprises:

Analyze and calculate the information reported by the trigger unit;

Determine parameters for reset, integration, transfer, and readout of rolling shutter image sensors;

Calibrate the working environment parameters of the imaging device using this method, wherein the environmental parameters include target mark moving speed, target mark distance, target mark angle, and ambient lighting intensity;

Calculate and determine the lighting sequence of the lighting system based on rolling shutter image sensor parameters and working environment parameters, including: illumination intensity, start time, and end time of a single or multiple lighting systems;

The global reset or the switching sequence of the external physical shutter is determined by calculation based on the rolling shutter image sensor parameters and the working environment parameters.

23. The illumination imaging method according to claim 19, wherein during the trigger sampling process of step 2:

By pressing a physical button set in the imaging device, sampling is triggered;

Alternatively, a photoelectric sensor, distance sensor, or infrared sensor is used to automatically detect the presence of a target mark, and when pre-set conditions are met, the sampling process is automatically triggered;

Alternatively, sampling is triggered by the host computer via an external communication interface;

Alternatively, the control and computing subsystem controls the timing trigger sampling.

24. The illumination imaging method according to claim 23, wherein the trigger unit tracks and collects the position, speed, height, angle, and temperature of the target relative to the lens center of the imaging device, performs pre-processing, and reports the collected data to the control and computing subsystem.

25. The illumination imaging method according to claim 19, characterized in that the illumination imaging method is applied to industrial production and logistics transportation scenarios such as display bad pixel detection, large-scale flat device scratch detection, and package tracking;

The illumination imaging method is applied to daily outdoor scenes for aerial photography using a small multi-rotor aircraft and panoramic motion photography;

The illumination imaging method is applied to the collection and recognition of various one-dimensional and two-dimensional code images in the fields of industry, commerce, and logistics.

The illumination imaging method is applied to scientific research scenes captured by digital microscopes for marker tracking.

26. A code reader, comprising the imaging device according to claim 1, and performing target reading and identification through the imaging device.