HK1218157B - Ct system for security check and method thereof - Google Patents

Description

Security inspection CT system and method

Technical Field

The present application relates to security inspection, and in particular to a security inspection CT system and method thereof.

Background Art

The multi-energy X-ray security inspection system is a new type of security inspection system developed based on the single-energy X-ray security inspection system. It can not only provide the shape and content of the inspected object, but also provide information reflecting the effective atomic number of the inspected object, thereby distinguishing whether the inspected object is organic or inorganic. The information is displayed in different colors on the color monitor to help operators make judgments.

TIP is a critical requirement in the security inspection field. TIP involves inserting pre-captured images of dangerous goods, also known as fictitious threat images, into images of baggage or parcels. This technology plays a crucial role in training and assessing the efficiency of security inspectors. While mature solutions and widespread application exist for 2D TIP in X-ray security inspection systems, no vendor currently offers this functionality for 3D TIP in security CT systems.

Summary of the Invention

In view of one or more technical problems in the prior art, the present invention proposes a security inspection CT system and method thereof, which can facilitate users to quickly mark suspicious objects in CT images and provide feedback on whether virtual dangerous goods images are included.

In one aspect of the present invention, a method in a security inspection CT system is provided, comprising the steps of: reading inspection data of an inspected object; inserting at least one 3D virtual contraband image (Fictional Threat Image) into a 3D inspection image of the inspected object, the 3D inspection image being derived from the inspection data; receiving a selection of at least one region in a 3D inspection image including the 3D virtual contraband image or at least one region in a 2D inspection image including a 2D virtual contraband image corresponding to the 3D virtual contraband image, the 2D inspection image being derived from the 3D inspection image or the inspection data; and providing feedback related to the inclusion of the at least one 3D virtual contraband image in the 3D inspection image in response to the selection.

According to some embodiments, the step of receiving a selection of at least one area in a 3D inspection image including the 3D virtual contraband image or at least one area in a 2D inspection image including a 2D virtual contraband image corresponding to the 3D virtual contraband image includes: receiving a coordinate position of a portion of the 3D inspection image or the 2D inspection image associated with the selection.

According to some embodiments, the step of giving feedback related to the inclusion of at least one 3D virtual contraband image in the 3D inspection image in response to the selection includes at least one of the following: determining whether the at least one 3D virtual contraband image exists in at least one area of the selection, popping up a dialog box to confirm that the at least one 3D virtual contraband image is included in the 3D inspection image, confirming that the at least one 3D virtual contraband image is included in the 3D inspection image with a text prompt on the interface, highlighting the portion of the 3D inspection image or the 2D inspection image associated with the selection, marking the portion of the 3D inspection image or the 2D inspection image associated with the selection, and filling the portion of the 3D inspection image or the 2D inspection image associated with the selection with a specific color or graphic.

According to some embodiments, at least one spatial characteristic parameter of the inspected object is calculated based on the inspection data, and at least one 3D virtual contraband image is inserted into the 3D inspection image of the inspected object based on the spatial characteristic parameter.

According to some embodiments, the spatial feature parameter is related to at least one of a position, a size, and an orientation of the 3D virtual contraband image to be inserted.

According to some embodiments, the selection of the at least one region includes selection of a portion of the displayed 3D inspection image at a viewing angle.

According to some embodiments, during the 3D rendering process of the 3D inspection image, the step of recording point cloud information representing the inspected object and providing feedback related to the inclusion of at least one 3D virtual contraband image in the 3D inspection image in response to the selection includes: obtaining a sequence of point cloud information clusters of different objects in the inspected object by segmentation; determining at least one selected area from the sequence of point cloud information clusters of different objects based on a predetermined benchmark; and determining whether the at least one 3D virtual contraband image exists in the at least one selected area.

According to some embodiments, the selection of the at least one region comprises selection of a portion of the displayed 3D inspection image at a plurality of different viewing angles.

According to some embodiments, the selection of at least one area includes selecting a portion of the displayed 3D inspection image from two different perspectives, the two different perspectives being substantially orthogonal to each other, wherein transparent areas are removed from the inspection data to obtain a hierarchical bounding box of a non-transparent area in the inspection data, and then scene depth is rendered for the hierarchical bounding box to obtain a front surface depth map and a back surface depth map. In response to the selection, the step of providing feedback related to the inclusion of at least one 3D virtual contraband image in the 3D inspection image includes: searching the front surface depth map and the back surface depth map according to the area selected by the user from the first perspective respectively to generate a first bounding box; performing ray casting using the generated first bounding box as a texture carrier; searching the front surface depth map and the back surface depth map according to the area selected by the user from a second perspective substantially orthogonal to the first perspective to generate a second bounding box; performing a Boolean intersection operation on the first bounding box and the second bounding box in the image space to obtain a marked area in three-dimensional space as the at least one selected area; and determining whether the at least one 3D virtual contraband image exists in the at least one selected area.

According to some embodiments, the step of inserting at least one 3D virtual contraband image into the 3D inspection image of the inspected object includes: segmenting the 3D inspected image to obtain multiple 3D sub-images of the inspected object; calculating distances and positions between the multiple 3D sub-images; and inserting the 3D virtual contraband image based on the calculated distances and positions.

According to some embodiments, the step of inserting at least one 3D virtual contraband image into the 3D inspection image of the inspected object includes: determining transparent parts and non-transparent parts in the volume data of the inspected object based on the opacity value of the voxel; determining the position and size of the luggage of the inspected object from the opaque part of the volume data; determining candidate insertion positions in the transparent area within the luggage range; and selecting at least one position from the candidate insertion positions according to predetermined criteria to insert at least one 3D contraband image.

According to some embodiments, the step of inserting at least one 3D virtual contraband image into the 3D inspection image of the inspected object includes: removing a background image from the 2D inspection image to obtain a 2D foreground image; determining a 2D insertion position of the 2D virtual contraband image in the 2D foreground image; determining a position of the 3D virtual contraband image in the 3D inspection image along a depth direction of the 2D insertion position; and inserting at least one 3D virtual contraband image at the determined position.

According to some embodiments, the method further comprises inserting a 2D virtual contraband image corresponding to the at least one 3D virtual contraband image into the 2D inspection image of the inspected object.

In another aspect of the present invention, a security inspection CT system is provided, comprising: a CT scanning device for obtaining inspection data of an inspected object; a memory for storing the inspection data; a display device for displaying a 3D inspection image and/or a 2D inspection image of the inspected object, wherein the 3D inspection image is obtained from the inspection data, the 2D inspection image is obtained from the 3D inspection image, or is obtained from the inspection data; a data processor for inserting at least one 3D virtual contraband image (Fictional Threat Image) into the 3D inspection image of the inspected object; and an input device for receiving a selection of at least one region in the 3D inspection image including the 3D virtual contraband image or at least one region in the 2D inspection image including a 2D virtual contraband image corresponding to the 3D virtual contraband image; wherein the data processor, in response to the selection, provides feedback related to the inclusion of the at least one 3D virtual contraband image in the 3D inspection image.

According to some embodiments, the data processor calculates at least one spatial feature parameter of the inspected object based on the inspection data, and inserts at least one 3D virtual contraband image into the 3D inspection image of the inspected object based on the spatial feature parameter.

According to some embodiments, the spatial feature parameter is related to at least one of a position, a size, and an orientation of the 3D virtual contraband image to be inserted.

In one aspect of the present invention, a method for marking a suspect in a security inspection CT system is proposed, comprising the steps of: removing transparent areas from CT data obtained by the security inspection CT system to obtain hierarchical bounding boxes of non-transparent areas in the CT data; rendering scene depth for the hierarchical bounding boxes to obtain a front surface depth map and a back surface depth map; using a mark made by a user in the direction of sight to search the front surface depth map and the back surface depth map respectively to generate a first bounding box; using the generated first bounding box as a texture carrier to perform ray projection; using a mark made by the user in a direction orthogonal to the direction of sight to search the front surface depth map and the back surface depth map respectively to generate a second bounding box; performing a Boolean intersection operation on the first bounding box and the second bounding box in image space to obtain a marked area in three-dimensional space; and fusing the marked area in the three-dimensional space for display in the CT data.

According to some embodiments, the step of removing transparent areas includes: sampling CT data along the line of sight; performing volume rendering integration on the line segment between every two sampling points using a pre-integration lookup table based on opacity to obtain the opacity corresponding to the line segment; and using an octree coding algorithm to segment and remove transparent areas to obtain a hierarchical bounding box corresponding to the opaque data area.

According to some embodiments, the step of rendering scene depth includes: removing fragments with larger depth values to obtain a front surface depth map; and removing fragments with smaller depth values to obtain a back surface depth map.

According to some embodiments, the first bounding box and the second bounding box are both arbitrarily oriented bounding boxes.

According to some embodiments, a spatially constrained transfer function is used to fuse the labeled region in the three-dimensional space into the CT data.

In another aspect of the present invention, a device for marking suspicious objects in a security inspection CT system is proposed, comprising: a device for removing transparent areas from CT data obtained by the security inspection CT system to obtain a hierarchical bounding box of non-transparent areas in the CT data; a device for rendering scene depth for the hierarchical bounding box to obtain a front surface depth map and a back surface depth map; a device for searching the front surface depth map and the back surface depth map using a mark made by a user in the direction of sight respectively to generate a first bounding box; a device for performing ray projection using the generated first bounding box as a texture carrier; a device for searching the front surface depth map and the back surface depth map using a mark made by the user in a direction orthogonal to the direction of sight respectively to generate a second bounding box; a device for performing a Boolean intersection operation on the first bounding box and the second bounding box in the image space to obtain a marked area in three-dimensional space; and a device for fusing the marked area in the three-dimensional space and displaying it in the CT data.

According to some embodiments, the device for removing transparent areas includes: a device for sampling CT data along the line of sight; a device for performing volume rendering integral on the line segment between each two sampling points using a lookup table method to obtain the opacity of the corresponding line segment; and a device for removing transparent areas by using an octree coding algorithm to obtain a hierarchical bounding box.

According to some embodiments, the apparatus for rendering scene depth includes: an apparatus for removing fragments with larger depth values compared to obtain a front surface depth map; and an apparatus for removing fragments with smaller depth values compared to obtain a rear surface depth map.

By utilizing the above technical solution, users can quickly mark suspicious objects in CT images and receive feedback on whether the images contain virtual dangerous goods.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to better understand the present invention, the present invention will be described in detail with reference to the following drawings:

FIG1 shows a schematic structural diagram of a security inspection CT system according to an embodiment of the present invention;

FIG2 shows a block diagram of the computer data processor shown in FIG1 ;

FIG3 shows a structural block diagram of a controller according to an embodiment of the present invention;

FIG4A is a schematic flow chart illustrating a method in a security inspection system according to an embodiment of the present invention;

FIG4B is a flowchart illustrating a method for marking a suspect in a CT system according to one embodiment of the present invention;

FIG5 is a schematic diagram illustrating an octree partitioning algorithm;

FIG6 is a schematic diagram of a hierarchical bounding box obtained by using an octree decomposition algorithm in an embodiment of the present invention;

FIG7 is a schematic diagram of a frontal depth map obtained in an embodiment of the present invention;

FIG8 is a schematic diagram of a back surface depth map obtained in an embodiment of the present invention;

FIG9 is a schematic diagram illustrating the radiation transmission process used in an embodiment of the present invention;

FIG10 is a schematic diagram showing a mark drawn by a user in an embodiment of the present invention;

FIG11 is a schematic diagram showing the process of performing forward and reverse face retrieval using user tags;

FIG12 is a schematic diagram showing the results obtained by performing forward face search and backward face search in an embodiment of the present invention;

FIG13 is a schematic diagram showing an OBB bounding box of a marker point sequence obtained in an embodiment of the present invention;

FIG14 is a schematic diagram showing a method of updating a new ray casting range based on the result of the previous marking;

FIG15 is a schematic diagram showing the result of a second marking in an orthogonal direction in an embodiment of the present invention;

FIG16 shows the results of forward and reverse face searches using the second markup in an embodiment of the present invention;

FIG17 is a schematic diagram showing an OBB bounding box of a marker point sequence obtained in an embodiment of the present invention;

FIG18 is a schematic diagram showing a process of performing a Boolean intersection operation on two objects in image space used in an embodiment of the present invention;

FIG19 is a schematic diagram showing a final three-dimensional marked area of a suspect obtained in an embodiment of the present invention; and

FIG. 20 is a schematic diagram showing how the labeled suspects are fused and displayed in the original data in an embodiment of the present invention.

DETAILED DESCRIPTION

Specific embodiments of the present invention will be described in detail below. It should be noted that the embodiments described herein are intended to be illustrative only and are not intended to limit the present invention. In the following description, numerous specific details are set forth to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that these specific details are not necessarily required to practice the present invention. In other instances, well-known structures, materials, or methods are not specifically described to avoid obscuring the present invention.

Throughout this specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present invention. Thus, appearances of the phrases "in one embodiment," "in an embodiment," "an example," or "an example" in various places throughout this specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures, or characteristics may be combined in any suitable combinations and/or subcombinations in one or more embodiments or examples. Furthermore, one of ordinary skill in the art will understand that the term "and/or" as used herein includes any and all combinations of one or more of the associated listed items.

To address the problem that existing technologies cannot quickly insert 3D virtual contraband images, embodiments of the present invention provide a method for reading inspection data of an inspected object. At least one 3D virtual contraband image (Fictional Threat Image) is inserted into the 3D inspection image of the inspected object, the 3D inspection image being derived from the inspection data. A selection is received for at least one area in a 3D inspection image including the 3D virtual contraband image, or at least one area in a 2D inspection image including a 2D virtual contraband image corresponding to the 3D virtual contraband image, the 2D inspection image being derived from the 3D inspection image or the inspection data. In response to the selection, feedback is provided regarding the inclusion of at least one 3D virtual contraband image in the 3D inspection image. Using the above solution, users can quickly mark suspects in CT images and receive feedback on whether a virtual threat image is included.

Figure 1 is a schematic diagram of the structure of a CT system according to an embodiment of the present invention. As shown in Figure 1 , the CT apparatus according to this embodiment includes a gantry 20, a support mechanism 40, a controller 50, a computer data processor 60, and the like. The gantry 20 includes a radiation source 10, such as an X-ray machine, that emits X-rays for inspection, and a detection and acquisition device 30. The support mechanism 40 carries inspected baggage 70 through the scanning area between the radiation source 10 and the detection and acquisition device 30 of the gantry 20. The gantry 20 simultaneously rotates in the direction of travel of the inspected baggage 70, allowing the radiation emitted by the radiation source 10 to penetrate the inspected baggage 70, thereby performing a CT scan on the inspected baggage 70.

The detection and acquisition device 30 is, for example, a detector and data acquisition unit with an integrated modular structure, such as a flat-panel detector. It detects radiation transmitted through the inspected item, generates analog signals, and converts these signals into digital signals, thereby outputting X-ray projection data of the inspected baggage 70. The controller 50 is used to synchronize the operation of various components of the system. The computer data processor 60 processes the data collected by the data acquisition unit, reconstructs the data, and outputs the results.

As shown in Figure 1, a radiation source 10 is placed on one side of the inspected object, while a detection and acquisition device 30 is placed on the other side of the inspected baggage 70. The device comprises a detector and a data acquisition unit, which are used to acquire multi-angle projection data from the inspected baggage 70. The data acquisition unit includes a data amplification and shaping circuit that can operate in either a (current) integration mode or a pulse (counting) mode. The data output cable of the detection and acquisition device 30 is connected to a controller 50 and a computer data processor 60. The acquired data is stored in the computer data processor 60 in response to trigger commands.

FIG2 shows a block diagram of the computer data processor 60 shown in FIG1 . As shown in FIG2 , the data collected by the data collector is stored in the memory 61 via the interface unit 68 and the bus 64 . The read-only memory (ROM) 62 stores configuration information and programs for the computer data processor. The random access memory (RAM) 63 is used to temporarily store various data while the processor 66 is operating. Furthermore, the memory 61 also stores computer programs for data processing. An internal bus 64 connects the memory 61, the read-only memory 62, the random access memory 63, the input device 65, the processor 66, the display device 67, and the interface unit 68.

After the user inputs an operation command through an input device 65 such as a keyboard and a mouse, the instruction code of the computer program commands the processor 66 to execute a predetermined data processing algorithm. After obtaining the data processing result, it is displayed on a display device 67 such as an LCD display, or the processing result is directly output in the form of a hard copy such as a printout.

Figure 3 shows a block diagram of the controller according to an embodiment of the present invention. As shown in Figure 3, the controller 50 includes: a control unit 51, which controls the radiation source 10, the carrier 40, and the detection and acquisition device 30 according to instructions from a computer 60; a trigger signal generating unit 52, which, under the control of the control unit, generates a trigger command to trigger the operation of the radiation source 10, the detection and acquisition device 30, and the carrier 40; a first drive device 53, which drives the carrier 40 to transport the inspected baggage 70 according to a trigger command generated by the trigger signal generating unit 52 under the control of the control unit 51; and a second drive device 54, which rotates the gantry 20 according to a trigger command generated by the trigger signal generating unit 52 under the control of the control unit 51. Projection data obtained by the detection and acquisition device 30 is stored in the computer 60 for CT tomographic image reconstruction, thereby obtaining tomographic image data of the inspected baggage 70. The computer 60 then generates, for example, a DR image of the inspected baggage 70 from the tomographic image data, from at least one viewing angle, and displays it together with the reconstructed 3D image to facilitate security inspection by the image judge. According to other embodiments, the aforementioned CT imaging system may also be a dual-energy CT system, that is, the X-ray source 10 of the gantry 20 is capable of emitting both high-energy and low-energy X-rays. After the detection and acquisition device 30 detects projection data at different energy levels, the computer data processor 60 performs dual-energy CT reconstruction to obtain equivalent atomic number and electron density data for each slice of the inspected baggage 70.

FIG4A is a schematic flow chart illustrating a method in a security inspection system according to an embodiment of the present invention.

As shown in FIG. 4A , in step S401 , inspection data of the inspected object is read.

In step S402, at least one 3D virtual contraband image (Fictional Threat Image) is inserted into the 3D inspection image of the inspected object. The 3D inspection image is obtained from the inspection data. For example, a data processor selects one or more 3D images from a virtual threat image library and inserts them into the 3D inspection image of the inspected object.

In step S403, a selection of at least one area in a 3D inspection image including the 3D virtual contraband image or at least one area in a 2D inspection image including a 2D virtual contraband image corresponding to the 3D virtual contraband image is received, where the 2D inspection image is derived from the 3D inspection image or the inspection data. For example, a user operates an input device to select or circle an area in an image displayed on a screen.

In step S404, feedback related to at least one 3D virtual contraband image being included in the 3D inspection image is given in response to the selection.

In some embodiments, receiving a selection of at least one area in a 3D inspection image including the 3D virtual contraband image or at least one area in a 2D inspection image including a 2D virtual contraband image corresponding to the 3D virtual contraband image includes receiving a coordinate position of a portion of the 3D inspection image or the 2D inspection image associated with the selection.

In some embodiments, the step of providing feedback related to the inclusion of at least one 3D virtual contraband image in the 3D inspection image in response to the selection comprises at least one of the following:

determining whether the at least one 3D virtual contraband image exists in the at least one selected area,

A dialog box pops up to confirm that the 3D inspection image contains at least one 3D virtual contraband image.

Confirm with a text prompt on the interface that the 3D inspection image contains at least one 3D virtual contraband image,

highlighting the portion of the 3D examination image or the 2D examination image associated with the selection,

Marking a portion of the 3D inspection image or the 2D inspection image associated with the selection,

The portion of the 3D inspection image or the 2D inspection image associated with the selection is filled with a specific color or graphic.

For example, at least one spatial characteristic parameter of the inspected object is calculated based on the inspection data, and at least one 3D virtual contraband image is inserted into the 3D inspection image of the inspected object based on the spatial characteristic parameter. In some embodiments, the spatial characteristic parameter is related to at least one of the position, size, and orientation of the 3D virtual contraband image to be inserted. Furthermore, the selection of at least one region includes selecting a portion of the displayed 3D inspection image at a single viewing angle. For example, during 3D rendering of the 3D inspection image, point cloud information representing the inspected object is recorded, and in response to the selection, feedback regarding the inclusion of at least one 3D virtual contraband image in the 3D inspection image may include: obtaining a sequence of point cloud information clusters of different objects in the inspected object by segmentation; determining at least one selected region from the sequence of point cloud information clusters of the different objects based on a predetermined benchmark; and determining whether the at least one 3D virtual contraband image exists in the at least one selected region.

In other embodiments, the selection of the at least one region includes selection of a portion of the displayed 3D inspection image at a plurality of different viewing angles. For example, the selection of at least one area includes selecting a portion of the displayed 3D inspection image from two different perspectives, wherein the two different perspectives are substantially orthogonal to each other, wherein transparent areas are removed from the inspection data to obtain a hierarchical bounding box of a non-transparent area in the inspection data, and then the scene depth is rendered for the hierarchical bounding box to obtain a front surface depth map and a back surface depth map. The step of providing feedback related to the inclusion of at least one 3D virtual contraband image in the 3D inspection image in response to the selection includes: searching the front surface depth map and the back surface depth map according to the area selected by the user at the first perspective respectively to generate a first bounding box; performing ray casting using the generated first bounding box as a texture carrier; searching the front surface depth map and the back surface depth map according to the area selected by the user at a second perspective substantially orthogonal to the first perspective to generate a second bounding box; performing a Boolean intersection operation on the first bounding box and the second bounding box in the image space to obtain a marked area in the three-dimensional space as the at least one selected area; and determining whether the at least one 3D virtual contraband image exists in the at least one selected area.

In some embodiments, the step of inserting at least one 3D virtual contraband image into the 3D inspection image of the inspected object includes: segmenting the 3D inspected image to obtain multiple 3D sub-images of the inspected object; calculating the distances and positions between the multiple 3D sub-images; and inserting the 3D virtual contraband image based on the calculated distances and positions.

In another embodiment, the step of inserting at least one 3D virtual contraband image into the 3D inspection image of the inspected object includes: determining transparent and non-transparent portions in the volume data of the inspected object based on the opacity values of the voxels; determining the position and size of a luggage of the inspected object from the non-transparent portions of the volume data; determining candidate insertion positions in the transparent area within the luggage range; selecting at least one selected area from the candidate insertion positions according to predetermined criteria; and determining whether the at least one 3D virtual contraband image exists in the at least one selected area.

In another embodiment, the step of inserting at least one 3D virtual contraband image into the 3D inspection image of the inspected object includes: removing a background image from the 2D inspection image to obtain a 2D foreground image; determining a 2D insertion position of the 2D virtual contraband image in the 2D foreground image; determining a position of the 3D virtual contraband image in the 3D inspection image along a depth direction of the 2D insertion position; and inserting at least one 3D virtual contraband image at the determined position.

The above description is about inserting a 3D virtual dangerous goods image, but in some embodiments of the present invention, a 2D virtual contraband image corresponding to the at least one 3D virtual contraband image may also be inserted into the 2D inspection image of the inspected object.

Furthermore, to address the challenges of existing technologies, some embodiments of the present invention propose a technique for rapidly marking suspect objects. After rapidly removing transparent areas from the data, the new entry and exit positions of the projected ray are obtained and recorded as a depth map. Based on this, the two-dimensional mark is restored to its depth information in voxel space. A Boolean intersection operation is performed on the two obtained geometric bodies in image space, ultimately yielding the marked region in three-dimensional space.

For example, in some embodiments, transparent areas are first eliminated to quickly obtain a tight hierarchical bounding box of the non-transparent area in the data. Then, the hierarchical bounding box generated above is rendered to obtain the front and back surface depth maps, which are the adjusted entry and exit positions of the projected light. Next, a first pick is performed in the current line of sight direction, and the marker point list is used to search the front and back surface depth maps respectively to generate a bounding box such as an OBB bounding box. Then, based on the OBB bounding box generated above, the projection range of the light is updated, and the user performs a second pick under the automatically rotated orthogonal perspective to generate a new OBB bounding box. The OBB bounding boxes obtained in the first two steps are subjected to a Boolean intersection operation in the image space to obtain the final marked area. Finally, the suspicious area is fused and displayed in the original data using a transfer function based on spatial constraints. Using the marking method of the present invention, transparent areas in CT data can be quickly and accurately eliminated, allowing users to quickly complete the suspicious area marking task in a user-friendly manner.

FIG4B is a flowchart illustrating a method for marking a suspect in a CT system according to an embodiment of the present invention. After the CT device acquires CT data, transparent regions in the CT data are first extracted. After quickly removing the transparent regions from the data, the new entry and exit positions of the light are recorded as a depth map. During the picking process, the depth information of the two-dimensional marker in voxel space is restored by querying the depth map. A Boolean intersection operation is performed on the two obtained geometric bodies in image space to ultimately obtain the marked region in three-dimensional space.

In step S411, transparent areas are eliminated based on pre-integration on the CT data obtained by the security inspection CT system to obtain hierarchical bounding boxes of non-transparent areas in the CT data.

1) Generation of pre-integration lookup table based on opacity

The three-dimensional data field processed by volume rendering is discrete data defined in three-dimensional space, and the entire data field is represented by a discrete three-dimensional matrix. Each small square in the three-dimensional space represents a scalar value, called a voxel. In actual calculations, a voxel can be used as a sampling point of the three-dimensional data field, and the scalar value obtained by sampling is s. For the data field s(x), the volume data must first be classified to specify the color and attenuation coefficient. A transfer function is introduced to map the volume data intensity s to the color I(s) and the attenuation coefficient τ(s). In the implementation example, this transfer function is determined by the grayscale data and material data of the dual-energy CT, and is also called a two-dimensional color table.

In volume rendering, when sampling the three-dimensional scalar field s(x), the Nyquist sampling frequency of the opacity function τ(s(x)) is equal to the product of the maximum Nyquist sampling frequency of τ(s) and the Nyquist sampling frequency of the scalar value s(x). Due to the nonlinear characteristics of the attenuation coefficient, this can cause the Nyquist sampling frequency to increase dramatically. To address this issue, a pre-integration method is employed. This method also allows for rapid determination of transparency in the CT data of a given region.

The pre-integration is mainly divided into two steps. The first step is to sample the continuous scalar field s(x) along the line of sight. At this time, the sampling frequency value is not affected by the transfer function. The second step is to perform volume rendering integration on the line segment between each two sampling points by using a lookup table method.

After sampling s(x), the volume rendering integral is performed on each small line segment. This integral is accomplished using a lookup table. The lookup table has three parameters: the segment start point, the segment end point, and the segment length. If the segment length is set to a constant, only two parameters need to be considered when performing the lookup table calculation: the segment start point and the segment end point.

2) Octree-based transparent area culling

An octree is a tree-like data structure used to describe three-dimensional space. Figure 5 is a schematic diagram illustrating the octree partitioning algorithm. Each node in the octree represents a volume element of a cube. Each node has eight child nodes, and the volume elements represented by these eight child nodes, added together, equal the volume of the parent node. As shown in Figure 5, the octree consists of eight nodes: ulf, urf, ulb, urb, llf, lrf, llb, and lrb. When using the octree encoding algorithm to partition spatial data, assuming that the shape V to be represented can be placed within a sufficiently large cube C, the octree for shape V with respect to cube C can be defined recursively using the following method: Each node in the octree corresponds to a subcube of C, and the root node corresponds to C itself. If V = C, then the octree for V consists only of tree nodes; if V ≠ C, then C is divided into eight subcubes, each corresponding to a child node of the root. As long as a subcube is not completely empty or completely occupied by V, it is divided into eight equal parts, and the corresponding node thus has eight child nodes. This recursive judgment and segmentation process is continued until the cube corresponding to the node is either completely blank, completely occupied by V, or its size is the predefined sub-cube size.

Based on the set leaf node size, the volume data is partitioned layer by layer. As the data field is traversed, the maximum value (s _max) and minimum value (s _min) of all voxels within the sub-block corresponding to the leaf node, as well as the corresponding axial bounding box and volume values of the sub-block, are calculated. Nodes are then merged layer by layer upward to construct an octree. A schematic diagram of the octree partitioning is shown in Figure 5.

According to an embodiment of the present invention, an octree is traversed, and the visibility state of nodes at each level is recursively set. For non-leaf nodes, the states are transparent, partially transparent, and opaque. Their state is determined by the states of the child nodes contained within the node. If all child nodes are transparent, the current node is transparent; if all child nodes are opaque, the current node is opaque; if some child nodes are transparent, the current node is semi-transparent. For leaf nodes, the states are limited to transparent and opaque. The leaf node visibility state is obtained through an opacity query. Specifically, when constructing the octree, the minimum and maximum grayscale values (s _min , s _max ) of each sub-block are stored. The opacity query function α (s _f , s _b ) described above is used to quickly obtain the opacity α of the current sub-block. If α ≥ α _ε , the current leaf node is opaque, where α _ε is the set opacity threshold. Figure 6 shows the remaining opaque portion after removing the transparent block. The large rectangular wireframe represents the original data size.

In step S412, the scene depth is rendered for the hierarchical bounding box to obtain a front-facing depth map and a back-facing depth map. In step S413, the user's gaze mark is used to search the front-facing depth map and the back-facing depth map to generate a first bounding box. Figure 7 is a schematic diagram of a front-facing depth map obtained in an embodiment of the present invention. Figure 8 is a schematic diagram of a back-facing depth map obtained in an embodiment of the present invention.

In volume rendering, a three-dimensional model is required as a carrier of the volume texture. The volume texture is mapped to the model through texture coordinates, and then a ray is drawn from the viewpoint to the point on the model. The ray passing through the model space is equivalent to the ray passing through the volume texture. In this way, the entry and exit positions of the projected ray are determined, which is converted into the intersection problem of the ray and the volume texture carrier. As shown in Figure 7, the scene depth map obtained above is rendered by the hierarchical bounding box, and the fragments with larger depth values are eliminated to obtain the front surface depth map. At this time, the color value of each pixel on the front surface depth map represents the distance to the point closest to the viewpoint in a certain direction. As shown in Figure 8, the fragments with smaller depth values are eliminated, and the scene depth map is rendered to obtain the back surface depth map. The color value of each pixel on the back surface depth map represents the distance to the point farthest from the viewpoint in a certain direction.

Figure 9 is a schematic diagram illustrating the ray projection process used in an embodiment of the present invention. The basic process of ray projection is as follows: a ray is emitted from each pixel in an image along a fixed direction. This ray traverses the entire image sequence. During this process, the image sequence is sampled and classified to obtain color values. These color values are then accumulated according to a ray absorption model until the ray traverses the entire image sequence. The resulting color value represents the color of the rendered image. The projection plane shown in Figure 9 is the aforementioned "image."

The final result of ray casting is a two-dimensional image, which cannot restore the depth information of the voxels passed by the ray cast along the pixel. In order to complete the area picking in the voxel space, as shown in Figure 10, we pick the suspected area on the projection plane, and the marking result is shown in Figure 11. In order to restore the depth information of the voxel space from the marking result, the marking event is discretized into a point column, and searched separately in the front and back surface depth maps to obtain the projection result of the marked area on the depth map. Figure 12 shows a schematic diagram of the process of using the user's mark to perform front surface retrieval and back surface retrieval. At this point, we restore a two-dimensional marking operation on the screen image to a three-dimensional mark in the voxel space.

After completing one marking, the suspected area still covers a large area. In order to continue to crop this suspected area, it is necessary to calculate the OBB hierarchical bounding box corresponding to the marked point sequence in the voxel space.

The basic idea of the bounding box method is to use simple geometric shapes to replace complex and bizarre geometric shapes. First, a rough detection is performed on the object's bounding box. Only when the bounding boxes intersect can the bounding boxes intersect; when the bounding boxes do not intersect, the bounding boxes do not intersect. This eliminates a large number of geometric shapes and parts that are unlikely to intersect, allowing for the rapid identification of intersecting parts. There are several types of bounding boxes: AABBs along the coordinate axes, bounding spheres, OBBs along arbitrary directions, and the more general k-dop bounding box. After weighing the envelope tightness and computational cost of various bounding boxes, OBBs are chosen to calculate the marker point sequence. The key to calculating the OBB bounding box is to find the optimal direction and determine the minimum size of the bounding box that encloses the object in that direction. The first moment (mean) and second moment (covariance matrix) statistics are used to calculate the position and orientation of the bounding box.

The eigenvectors of the covariance matrix can be solved numerically and normalized. Since C is a real symmetric matrix, the eigenvectors of matrix C are perpendicular to each other and can serve as the directional axes of the bounding box. Project the vertices of the geometric body to be enclosed onto the directional axes and find the projection intervals of each directional axis. The length of each projection interval is the corresponding size of the bounding box. Figure 13 shows a schematic diagram of the OBB bounding box of the marker point sequence obtained in an embodiment of the present invention.

In step S414, the generated first bounding box is used as a texture carrier for ray casting; in step S415, the mark made by the user in the direction orthogonal to the viewing direction is used to search in the front depth map and the back depth map respectively to generate a second bounding box.

1) Update the projection range of the light

As shown in FIG14 , after determining the scope of a suspected area, the portion outside the area is removed from the display, and the generated OBB bounding box is used as a new volume texture carrier for ray casting.

2) Second pick after rotating the view

Figure 15 is a schematic diagram showing the results of a second labeling operation in an orthogonal direction in an embodiment of the present invention. Figure 16 shows the results of a forward face search and a reverse face search using the second labeling operation in an embodiment of the present invention. Figure 17 is a schematic diagram showing the OBB bounding box of the marked point sequence obtained in an embodiment of the present invention. In step S416, a Boolean intersection operation is performed on the first bounding box and the second bounding box in image space to obtain a marked area in three-dimensional space. Figure 18 is a schematic diagram showing the process of performing a Boolean intersection operation on two objects in image space used in an embodiment of the present invention.

To quickly obtain the intersection area of two OBB bounding boxes, a CSG method is used for calculation. There are two approaches to rendering CSG models using OpenGL. The first is based on object space: directly converting the CSG model into a set of polygons and then rendering it with OpenGL. Converting it to a B-rep model is a typical approach to this approach, but the model conversion is inefficient and inconvenient for dynamic modification. The second approach is based on image space, which is the method adopted in this paper.

Intersection operations are performed in image space without modifying the model. Dynamic operations are performed on each frame to determine which surfaces should be displayed and which should be hidden or clipped. The OpenGL stencil buffer is used to implement the intersection operation in CSG. Drawing on the idea of ray casting, when a solid surface is projected onto the screen, the number of times its pixels intersect with other surfaces is counted.

Through the previous operations, we've obtained two cubes. Finding the intersection of these two cubes essentially involves finding the portion of one cube's surface that lies within the volume of the other. During the intersection process, any given component entity is rendered separately, with its front and back surfaces rendered in separate passes. Each rendering pass first renders the current surface into the depth buffer. Then, combined with the stencil plane operation, other entities are used to determine the portion of the current surface that lies within the other entity.

Parity check is used here to determine whether a certain point is inside a given entity space. In theory, parity check can be used to determine whether any point in space is inside a given volume. However, since the OpenGL depth buffer can only save one depth value for each pixel, the parity check process for rendering the intersection of entities A and B is: first find the part of A in B and draw it, then find the part of B in A and draw it. At this point, the front of A in B has been rendered. In order to obtain the front of B in A, first re-render the pixels in the depth buffer that are covered by the front of B. This is because after the previous operation, all parts of A are in the depth buffer, and the part of A outside B may obscure the originally visible part of B. After the depth value of B is adjusted correctly in the depth buffer, the part of the front of B in A is found and rendered, which is similar to the above and is omitted. Figure 19 shows a schematic diagram of the final three-dimensional marking area of the suspect obtained in the implementation of the present invention.

In step S417, the marked area in the three-dimensional space is fused and displayed in the CT data. For example, after obtaining the picked suspect area, it is necessary to fuse this suspect area with the original data with a higher visual priority. As shown in Figure 18, the final suspect area may not be a regular rectangular shape. Here, a transfer function based on spatial constraints is used. Using the scan line algorithm, a one-dimensional query texture is generated based on the volume data dimensions. Each texel stores whether the corresponding spatial position is within the suspect area bounding box. The final fused rendering effect is shown in Figure 20.

Furthermore, when inserting a TIP into CT data, it is crucial to ensure that the inserted hazardous material image is within the confines of the luggage and does not overlap the original contents. Furthermore, the algorithm's real-time performance is a key consideration. In some embodiments, transparent and opaque areas of the volume data are determined based on light obscuration. Within the blank area of the luggage, a portion with a volume that meets the required requirements is selected as a candidate insertion location. Finally, a location with a specified concealment level is determined based on the distance from the location to the viewing plane and the number of surrounding objects.

For example, a pre-integrated lookup table based on opacity is first generated to quickly determine transparent and non-transparent areas in the volume data. Next, an opaque octree of the volume data is constructed to determine the location and size of the bag in the CT data. Next, a transparent octree of the volume data is constructed. This octree only counts the transparent portions within the data area, completely eliminating the opaque portions. This identifies the area within the bag that can be inserted. Portions within the transparent area that meet the insertion requirements are selected as candidate insertion locations. The final insertion location is determined based on the specified insertion concealment level.

The solution of the above embodiment can quickly insert a dangerous goods image into CT data and the insertion position can be guaranteed to be inside the luggage; the inserted image does not cover the original items in the luggage; the concealment level of the insertion can be set by parameters; and the real-time performance of the algorithm can be guaranteed.

The above detailed description has described numerous embodiments of methods and apparatus for marking suspects in a security CT system through the use of schematic diagrams, flowcharts, and/or examples. Where such schematic diagrams, flowcharts, and/or examples include one or more functions and/or operations, those skilled in the art will appreciate that each function and/or operation in such schematic diagrams, flowcharts, or examples can be implemented individually and/or collectively through various structures, hardware, software, firmware, or substantially any combination thereof. In one embodiment, several portions of the subject matter described in the embodiments of the present invention may be implemented through an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or other integrated formats. However, those skilled in the art will recognize that some aspects of the embodiments disclosed herein may be equivalently implemented, in whole or in part, in an integrated circuit, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or substantially any combination thereof. Based on this disclosure, those skilled in the art will be equipped with the ability to design circuits and/or write software and/or firmware code. Furthermore, those skilled in the art will recognize that the mechanisms of the subject matter of the present disclosure can be distributed as a program product in a variety of forms, and that the exemplary embodiments of the subject matter of the present disclosure are applicable regardless of the specific type of signal-bearing medium actually used to perform the distribution. Examples of signal-bearing media include, but are not limited to, recordable media such as floppy disks, hard drives, compact disks (CDs), digital versatile disks (DVDs), digital tapes, computer memories, and the like; and transmission media such as digital and/or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and the like).

While the present invention has been described with reference to several exemplary embodiments, it should be understood that the terms used are descriptive and illustrative, rather than restrictive. Since the present invention can be embodied in many forms without departing from the spirit or essence of the invention, it should be understood that the above-described embodiments are not limited to any of the foregoing details, but should be interpreted broadly within the spirit and scope of the appended claims. All changes and modifications that fall within the scope of the claims or their equivalents are intended to be covered by the appended claims.

Claims (15)

1. A method in a security screening CT system, comprising the following steps: Read the inspection data of the object being inspected; Insert at least one 3D virtual threat image into the 3D inspection image of the object being inspected, the 3D inspection image being obtained from the inspection data; Receive selection of at least one region in a 3D inspection image including the 3D virtual contraband image, or selection of at least one region in a 2D inspection image including a 2D virtual contraband image corresponding to the 3D virtual contraband image, wherein the 2D inspection image is obtained from the 3D inspection image or from the inspection data; and In response to the selection, feedback is given relating to the inclusion of at least one 3D virtual image of contraband in the 3D inspection image. The selection of at least one region includes selecting a portion of the displayed 3D inspection image from two different viewpoints, the two different viewpoints being substantially orthogonal to each other, wherein transparent regions are culled from the inspection data to obtain a hierarchical bounding box of the non-transparent regions in the inspection data, and then scene depth is rendered on the hierarchical bounding box to obtain a frontal depth map and a backal depth map, wherein the step of providing feedback in response to the selection as relating to the inclusion of at least one 3D virtual contraband image in the 3D inspection image includes: The first bounding box is generated by searching the frontal and back depth maps respectively based on the area selected by the user from the first-view perspective. Use the generated first bounding box as a texture carrier for light projection; A second bounding box is generated by searching the frontal depth map and the backal depth map respectively based on the area selected by the user in the second view that is substantially orthogonal to the first view. Perform a Boolean intersection operation on the first and second bounding boxes in the image space to obtain the marked regions in the three-dimensional space, which serve as at least one selected region. Determine whether the at least one 3D virtual contraband image exists in the at least one selected area. 2. The method of claim 1, wherein the step of receiving selection of at least one region in a 3D inspection image including the 3D virtual contraband image or at least one region in a 2D inspection image including a 2D virtual contraband image corresponding to the 3D virtual contraband image comprises: Receive the coordinates of the portion of the 3D or 2D inspection image associated with the selection. 3. The method of claim 1, wherein the step of responding to the selection to provide feedback relating to the inclusion of at least one 3D virtual contraband image in the 3D inspection image comprises at least one of the following: Determine whether at least one 3D virtual contraband image exists in at least one selected area. A pop-up dialog box confirms that the 3D inspection image contains at least one 3D virtual image of contraband. The interface displays a text prompt confirming that the 3D inspection image contains at least one 3D virtual image of contraband. Highlight the portion of the 3D or 2D inspection image that is associated with the selection. The portion of the 3D or 2D inspection image associated with the selection is marked. Fill the portion of the 3D or 2D inspection image associated with the selection with a specific color or pattern. 4. The method of claim 1, wherein at least one spatial feature parameter of the object being inspected is calculated based on the inspection data, and at least one 3D virtual contraband image is inserted into a 3D inspection image of the object being inspected based on the spatial feature parameter. 5. The method of claim 4, wherein the spatial feature parameter relates to at least one of the position, size, and orientation of the 3D virtual contraband image to be inserted. 6. The method of claim 1, wherein selecting at least one region includes selecting a portion of the displayed 3D inspection image from a single viewpoint. 7. The method of claim 6, wherein during the 3D rendering of the 3D inspection image, the step of recording point cloud information representing the inspected object, and responding to the selection to provide feedback related to the inclusion of at least one 3D virtual contraband image in the 3D inspection image, comprises: The point cloud information cluster sequence of different objects in the inspected object is obtained by segmentation; At least one selected region is determined from a sequence of point cloud information clusters of different objects based on a predetermined benchmark. Determine whether the at least one 3D virtual contraband image exists in the at least one selected area. 8. The method of claim 1, wherein the step of inserting at least one 3D virtual contraband image into the 3D inspection image of the object being inspected comprises: The 3D image under inspection is segmented to obtain multiple 3D sub-images of the object under inspection; Calculate the distances and positions between the plurality of 3D sub-images; 3D virtual images of contraband are inserted based on the calculated distance and location. 9. The method of claim 1, wherein the step of inserting at least one 3D virtual contraband image into the 3D inspection image of the object being inspected comprises: Based on the opacity values of voxels, the transparent and non-transparent parts in the volume data of the object being inspected are determined. The position and size of the inspected object (bag) are determined from the opaque parts of the volume data. Determine candidate insertion positions within the transparent area of the bag/suit; At least one 3D contraband image is inserted by selecting at least one location from candidate insertion locations according to predetermined criteria. 10. The method of claim 1, wherein the step of inserting at least one 3D virtual image of contraband into a 3D inspection image of the object being inspected comprises: Remove the background image from the 2D inspection image to obtain the 2D foreground image; Determine the 2D insertion position of the 2D virtual contraband image within the 2D foreground image; The position of the 3D virtual contraband image in the 3D inspection image is determined along the depth direction of the 2D insertion location; Insert at least one 3D virtual image of the contraband at the determined location. 11. The method of claim 1, further comprising inserting a 2D virtual contraband image corresponding to the at least one 3D virtual contraband image into a 2D inspection image of the object being inspected. 12. A security screening CT system, comprising: CT scanning equipment obtains examination data of the object being examined; The memory stores the inspection data; A display device displays a 3D inspection image and/or a 2D inspection image of the object being inspected, wherein the 3D inspection image is obtained from the inspection data, and the 2D inspection image is obtained from the 3D inspection image or from the inspection data. The data processor inserts at least one 3D virtual contraband image into the 3D inspection image of the object being inspected; The input device receives a selection of at least one region in a 3D inspection image, including the 3D virtual contraband image, or a selection of at least one region in a 2D inspection image, including a 2D virtual contraband image corresponding to the 3D virtual contraband image. The data processor responds to the selection by providing feedback related to the presence of at least one 3D virtual contraband image in the 3D inspection image. The selection of at least one region includes selecting a portion of the displayed 3D inspection image from two different viewpoints, which are substantially orthogonal to each other. Transparent regions are culled from the inspection data to obtain hierarchical bounding boxes of the non-transparent regions within the inspection data. Scene depth is then rendered onto these hierarchical bounding boxes to obtain a frontal depth map and a backal depth map. Wherein, the data processor: The first bounding box is generated by searching the frontal and back depth maps respectively based on the area selected by the user from the first-view perspective. Use the generated first bounding box as a texture carrier for light projection; A second bounding box is generated by searching the frontal depth map and the backal depth map respectively based on the area selected by the user in the second view that is substantially orthogonal to the first view. Perform a Boolean intersection operation on the first and second bounding boxes in the image space to obtain the marked regions in the three-dimensional space, which serve as at least one selected region. Determine whether the at least one 3D virtual contraband image exists in the at least one selected area. 13. The security inspection CT system of claim 12, wherein the data processor calculates at least one spatial feature parameter of the object being inspected based on the inspection data, and inserts at least one 3D virtual contraband image into the 3D inspection image of the object being inspected based on the spatial feature parameter. 14. The security screening CT system of claim 13, wherein the spatial feature parameter relates to at least one of the position, size, and orientation of the 3D virtual contraband image to be inserted. 15. A storage medium containing computer instructions that, when executed, implement the method as described in any one of claims 1-11.