TWI859061B - Optimization device and optimization method for evaluating infectious arthritis - Google Patents

Abstract

An optimization device and an optimization method for evaluating infectious arthritis are provided. The optimization method includes obtaining a gray knee-tissue image and a doppler knee-tissue image with a blood vessel distribution that the gray knee-tissue image and the doppler knee-tissue image correspond to the same knee area; inputting the gray knee-tissue image to a vision transformer to retrieve multiple first image features of the gray knee-tissue image; inputting the doppler knee-tissue image to the vision transformer to retrieve multiple second image features of the doppler knee-tissue image; and simultaneously using the multiple first image features and the second image features by a machine learning classifier to generate multiple classified results and a probability of each classified result that the multiple classified results and the probabilities correspond to the same knee area to be information on evaluating infectious arthritis.

Description

Optimized device and optimized method for evaluating infectious arthritis

This case involves an optimized device and an optimized method for evaluating infectious arthritis.

The current method for evaluating infectious arthritis is to extract specimens from the joint, such as the synovial fluid, synovium or blood of the knee, culture the specimens for cells over a period of several days to a week, isolate the pathogens from the cultured specimens, and then analyze the bacterial species of the pathogens to prescribe the right medicine.

The culture of specimens takes a long time, and the condition of the affected part may deteriorate rapidly within a few days of the culture. Therefore, in order to wait for the culture results of the specimens, the affected part often cannot receive immediate treatment, resulting in delayed treatment.

As it takes a long time to culture specimens before analyzing pathogens, how to optimize the process is a technical problem that technicians in this technical field want to solve.

The case proposes an optimized device for evaluating infectious arthritis, including an ultrasound sensor, a Doppler sensor, and a controller. The ultrasound sensor is configured to sense a grayscale knee tissue image covering a partial knee area. The Doppler sensor is configured to sense a Doppler knee tissue image having a vascular distribution feature on a partial knee area, wherein the Doppler knee tissue image and the grayscale knee tissue image are tissue images corresponding to the same partial knee area. The controller is coupled to the ultrasound sensor and the Doppler sensor and is configured to perform the following operations: inputting a grayscale knee tissue image to a visual converter to capture a plurality of first image features of the grayscale knee tissue image; inputting a Doppler knee tissue image to a visual converter to capture a plurality of second image features of the Doppler knee tissue image; using the plurality of first image features and the plurality of second image features simultaneously in a machine learning classifier to perform classification operations to generate a plurality of classification results corresponding to tissue images of the same knee region and a probability value of each classification result; and outputting a plurality of classification results with probability values as evaluation information for evaluating infectious arthritis.

Another embodiment of the present invention provides an optimization method for evaluating infectious arthritis, which is performed using an optimization device. The optimization method includes the following steps: obtaining a grayscale knee tissue image corresponding to a portion of the knee region through an ultrasound sensor of the optimization device and obtaining a Doppler knee tissue image with vascular distribution characteristics corresponding to the same portion of the knee region through a Doppler sensor of the optimization device; inputting the grayscale knee tissue image to a visual converter to capture a plurality of first image features of the grayscale knee tissue image; inputting the Doppler knee tissue image to a visual converter to capture a plurality of first image features of the grayscale knee tissue image; An image-to-vision converter is used to capture a plurality of second image features of a Doppler knee tissue image; a plurality of first image features and a plurality of second image features are simultaneously used in a machine learning classifier to perform classification operations to generate a plurality of classification results corresponding to tissue images of the same knee region and a probability value of each classification result; and a plurality of classification results with probability values are output as evaluation information for evaluating infectious arthritis.

The present invention is further described below with reference to the drawings and embodiments, so that persons skilled in the art can better understand the present invention and implement it accordingly. However, the embodiments are not intended to limit the present invention.

In order to reduce the time for evaluating infectious arthritis and optimize the evaluation process of infectious arthritis, this case applies ultrasound images obtained by two ultrasound sensors, and proposes an optimization method and a device for executing the optimization method to achieve the above purpose.

FIG. 1 is a block diagram of an optimized device for evaluating infectious arthritis according to an embodiment of the present invention.

As shown in FIG1 , the optimization device 100 includes an ultrasonic sensor 110 , a Doppler sensor 120 , and a controller 130 . The controller 130 is coupled to the ultrasonic sensor 110 and the Doppler sensor 120 .

The ultrasound sensor 110 is configured to sense a portion of a knee region of a potential patient and generate a grayscale knee tissue image corresponding to the portion of the knee region.

The Doppler sensor 120 is configured to sense a portion of the knee region of the same potential patient and generate a Doppler knee tissue image corresponding to the portion of the knee region and having vascularity characteristics.

In one embodiment, the ultrasound sensor 110 and the Doppler sensor 120 are arranged in the same housing, and through this structure, the two images captured by the ultrasound sensor 110 and the Doppler sensor 120 correspond to the same part of the knee area. For example, when the ultrasound sensor 110 is placed on the skin surface of the knee and transmits a signal to the knee, the ultrasound sensor 110 receives the return signal and captures and stores it as a grayscale knee tissue image. At the same time, based on the structure that the ultrasonic sensor 110 and the Doppler sensor 120 are set in the same housing, when the ultrasonic sensor 110 is placed on the skin surface of the knee, the Doppler sensor 120 will also be placed on the skin surface of the same knee area. Therefore, after the Doppler sensor 120 transmits a signal to the knee, the return signal received and captured and stored as the knee area of the Doppler knee tissue image will be the knee area corresponding to the grayscale knee tissue image.

In one embodiment, the Doppler knee tissue image has the same tissue features as the grayscale knee tissue image, except that the Doppler knee tissue image also has vascular distribution features. The Doppler sensor 120 presents the vascular distribution features in the Doppler knee tissue image based on the relative position of blood vessels to general tissues and media. In other words, the vascular distribution features may obscure the portion of the image that originally presents general tissues and media.

In one embodiment, the grayscale knee tissue image and the Doppler knee tissue image of a set of image pairs are tissue images corresponding to the same partial knee region. Based on the number of times the ultrasound images are generated, a plurality of image pairs are generated, and different image pairs correspond to different knee regions.

The controller 130 is, for example, but not limited to, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Central Processing Unit (CPU), a System Single Chip (System) on Chip (SoC), Field Programmable Gate Array (FPGA), network processor (Network Processor) chip or a combination of the above components.

For easier understanding of the grayscale and Doppler knee tissue images of this case, please refer to Figures 2 and 3.

FIG. 2 is an actual image of a grayscale knee tissue generated by sensing a portion of the knee region with an ultrasound sensor according to an embodiment of the present invention. FIG. 3 is a Doppler knee tissue image with vascular distribution characteristics generated by sensing a portion of the knee region with a Doppler sensor according to an embodiment of the present invention. FIG. 2 and FIG. 3 are two tissue images generated toward the same knee region and can be considered as a set of image pairs. As shown in FIG. 2 , the grayscale knee tissue image 210 presents tissues and media of the human body, such as tissue 231.

On the other hand, FIG3 further shows the vascular distribution characteristics of the human body. As shown in FIG3, the Doppler knee tissue image 220 includes tissue 239 and vascular distribution characteristics 235. The irregular closed shape in FIG3 (please also refer to the orange block in the red frame of the attached file) is the vascular distribution characteristics sensed by the Doppler sensor 120, including the size of the blood vessels and the positional relationship relative to other tissues and media.

Since the grayscale knee tissue image 210 and the Doppler knee tissue image 220 correspond to the same knee region (the same image pair), the tissue 231 of the grayscale knee tissue image 210 and the tissue 239 of the Doppler knee tissue image 220 may be the same tissue feature of the knee region.

The following describes the optimization method for evaluating infectious arthritis using the grayscale knee tissue image 210 and the Doppler knee tissue image 220 in the field of machine learning.

FIG. 4 is a schematic diagram of the data flow of an optimization method for evaluating infectious arthritis executed by an optimization device according to an embodiment of the present invention.

In one embodiment, a set of image pairs of grayscale knee tissue images 210 and Doppler knee tissue images 220 are used as a set of input data, image features are captured by visual converters 310a and 310b, and classification results are generated by machine learning classifiers 320 to provide evaluation information for evaluating whether the corresponding knee area is a lesion site of high-risk infectious arthritis.

In one embodiment, the grayscale knee tissue image 210 and the Doppler knee tissue image 220 are input to the visual converter 310a and the visual converter 310b, respectively.

The vision converter 310a and the vision converter 310b are, for example, converters (ViT, Vision Tramsformer) for vision processing for image recognition.

In the vision converter 310 a that receives the grayscale knee tissue image 210 , the vision converter 310 a performs vision processing to extract a plurality of first image features 216 of the grayscale knee tissue image 210 .

On the other hand, in the visual converter 310 b receiving the Doppler knee tissue image 220 , the visual converter 310 b performs visual processing to extract a plurality of second image features 226 of the Doppler knee tissue image 220 .

Then, the plurality of first image features 216 and the plurality of second image features 226 are simultaneously input into the machine learning classifier 320 .

In one embodiment, the visual converter 310a and the visual converter 310b may be the same visual converter for processing the grayscale knee tissue image 210 and the Doppler knee tissue image 220 to extract a plurality of first image features 216 and a plurality of second image features 226 respectively.

The machine learning classifier 320 is, for example, a classifier constructed by methods such as a support vector classifier, a linear discriminant, a nearest neighbor method, a decision tree, a random forest, or a neural network.

The machine learning classifier 320 processes the plurality of first image features 216 and the plurality of second image features 226 simultaneously to perform correlation operations on these image features to generate an output data 410. The output data 410 includes a plurality of classification results and a probability value of each classification result. Then, these classification results with probability values are used as evaluation information for evaluating infectious arthritis.

The operation details of the visual converter 310a are further described below.

FIG. 5 is a schematic diagram of the detailed data flow of gray-scale knee tissue image processed by the visual converter of FIG. 4 according to an embodiment of the present invention.

The visual converter 310a first divides the grayscale knee tissue image 210 into a plurality of first small blocks 212. As shown in FIG5, the grayscale knee tissue image 210 is divided into nine first small blocks 212.

In one embodiment, the visual converter 310a includes a linear projection module 312 and a conversion encoder 314. After the visual converter 310a cuts out a plurality of first small blocks 212, the plurality of first small blocks 212 are input to the linear projection module 312. After the linear projection module 312 receives the plurality of first small blocks 212, the position information of each first small block 212 in the grayscale knee tissue image 210 is added to each first small block 212. For example, if the grayscale knee tissue image 210 is cut into 9 small blocks, the position information of the small block in the upper left corner is added to the small block in the upper left corner in an additional manner.

In addition to the position information, the linear projection module 312 also adds the category information to which each first small block 212 belongs in the grayscale knee tissue image 210 to each first small block 212. For example, if the grayscale knee tissue image 210 is cut into 9 small blocks and the category information of the small block in the upper left corner is articular membrane, the category information of "articular membrane" will be added to the small block in the upper left corner in an additional manner.

It is worth mentioning that the linear projection module 312 processes each first small block 212 in a vector form, so the position information and the category information are also added to each first small block 212 in a vector form. In the following description, each small block to which the position information and the category information are added is referred to as a first small block 214.

The visual converter 310a further includes a conversion encoder 314. Then, the linear projection module 312 inputs the plurality of first small blocks 214 carrying the position information and the category information to the conversion encoder 314. The conversion encoder 314 uses the plurality of first small blocks 214 carrying the position information and the category information to analyze a plurality of first image features 216. The conversion encoder 314 inputs these first image features 216 to the machine learning classifier 320.

The visual converter 310b also performs the same image processing procedure as that of FIG. 5 on the Doppler knee tissue image 220. In one embodiment, the visual converter 310b is a component independent of the visual converter 310a. The visual converter 310b also includes a linear projection module and a conversion encoder. The linear projection module and the conversion encoder of the visual converter 310b are the same components and functions as the linear projection module 312 and the conversion encoder 314 of FIG. 5, respectively, and implement the same operation as the linear projection module 312 and the conversion encoder 134.

In one embodiment, the visual converter 310b and the visual converter 310a are the same component, and the visual converter 310b includes a linear projection module 312 and a conversion encoder 314. To simplify the description, the following describes the visual converter 310b including the linear projection module 312 and the conversion encoder 314.

In one embodiment, the visual converter 310b first cuts the Doppler knee tissue image 220 into a plurality of second small blocks (not shown). After the linear projection module 312 receives the plurality of second small blocks, the position information of each second small block in the Doppler knee tissue image 220 is added to each second small block. For example, if the Doppler knee tissue image 220 is cut into 9 small blocks, the position information of the small block in the upper right corner is added to the small block in the upper right corner in an additional manner.

The linear projection module 312 adds the category information of each second small block in the Doppler knee tissue image 220 to each second small block. For example, if the Doppler knee tissue image 220 is cut into 9 small blocks and the category information of the small block in the upper right corner is blood vessel, the category information of "blood vessel" is added to the small block in the upper right corner in an additional manner.

It is worth mentioning that the linear projection module 312 processes each second small block in a vector form, so the position information and the category information are also added to each second small block in a vector form.

Next, the linear projection module 312 inputs the plurality of second small blocks carrying the position information and the category information to the conversion encoder 314. The conversion encoder 314 uses the plurality of second small blocks carrying the position information and the category information to analyze a plurality of second image features 226. The conversion encoder 314 inputs these second image features 226 to the machine learning classifier 320.

Then, as described above, the machine learning classifier 320 simultaneously processes the plurality of first image features 216 and the plurality of second image features 226 to perform classification operations on these image features to generate an output data 410. The output data 410 includes a plurality of classification results each having a probability value.

In one embodiment, the controller 130 provides the classification result having the maximum probability value according to the output data 410 as highly relevant information for evaluating infectious arthritis.

It is worth mentioning that the visual converter 310a, the visual converter 310b and the machine learning classifier 320 in Figures 4 and 5 are software modules, which are composed of a plurality of program codes. The controller 130 of the optimization device 100 in Figure 1 executes the program codes of the software module to execute the visual converter 310a, the visual converter 310b and the machine learning classifier 320 to implement each step of the optimization method for evaluating infectious arthritis.

FIG6 is a flow chart of an optimization method for evaluating infectious arthritis according to an embodiment of the present invention. The optimization method for evaluating infectious arthritis can be executed by the optimization device 100 of FIG1 .

In step S610 , a grayscale knee tissue image 210 covering a portion of the knee area is obtained by the ultrasound sensor 110 , and a Doppler knee tissue image 220 having a vascular distribution feature is obtained by the Doppler sensor 120 .

In one embodiment, when the grayscale knee tissue image 210 and the partial knee area covered by the Doppler knee tissue image 220 correspond to the same knee area, the two images are stored as an image pair.

In step S620 , the grayscale knee tissue image 210 is input to the visual converter 310 a via the controller 130 to capture a plurality of first image features 216 of the grayscale knee tissue image 210 .

In step S630, the Doppler knee tissue image 220 is input to the visual converter 310b via the controller 130 to capture the plurality of second image features 226 of the Doppler knee tissue image 220. It is worth mentioning that the controller 130 can simultaneously input the grayscale knee tissue image 210 and the Doppler knee tissue image 220 to the visual converter 310a and the visual converter 310b, respectively, and the execution order of step S620 and step S630 is not necessarily the order shown in FIG. 6 .

In step S640, the controller 130 inputs the plurality of first image features 216 and the plurality of second image features 226 to the machine learning classifier 320. The machine learning classifier 320 simultaneously uses the plurality of first image features 216 and the plurality of second image features 226 to perform classification operations to generate a plurality of classification results and probability values of each classification result.

In one embodiment, the obtained classification results are information of the grayscale knee tissue image 210 and the Doppler knee tissue image 220 corresponding to the same knee region.

In step S650, the controller 130 operates the machine learning classifier 320 to output a plurality of classification results with probability values as evaluation information for evaluating infectious arthritis.

In one embodiment, each classification result has a corresponding probability value, and the controller 130 outputs the classification results from high to low based on the probability value for the user's reference. The classification result with the highest probability value can be used as highly relevant information for evaluating infectious arthritis.

In one embodiment, the vision transformer 310a and the vision transformer 310b of FIG. 4 are pre-trained models. For example, the vision transformer 310a and the vision transformer 310b may be vision transformer models (Vision Transformer) obtained by other computing devices using other images unrelated to the grayscale knee tissue image 210 and the Doppler knee tissue image 220 for training. Since the optimization device 100 does not need to train the vision transformer 310a and the vision transformer 310b first, but uses a general image model to perform image feature extraction, the time for training the vision transformer 310a and the vision transformer 310b can be saved.

In one embodiment, the controller 130 uses a plurality of sets of image pairs to train the machine learning classifier 320 of FIG. 4 . For example, the controller 130 uses a grayscale knee tissue image (or a training grayscale knee tissue image) and a Doppler knee tissue image (or a training Doppler knee tissue image) of the same knee region sensed in advance as a set of image pairs, and obtains a set of image pairs every time the ultrasound sensor 110 and the Doppler sensor 120 move a short distance. Therefore, the controller 130 uses a plurality of sets of image pairs as training data to train the machine learning classifier 320.

In summary, when only one image type is considered to evaluate infectious arthritis, for example, only grayscale knee tissue images are used, because grayscale knee tissue images lack vascular distribution characteristics, there are fewer available image features, resulting in poor evaluation performance; or, when only Doppler knee tissue images are used, although Doppler knee tissue images have vascular distribution characteristics and have more available image features, the vascular distribution characteristics will obscure other tissues and media, which will cause some image features to be discontinuous, thereby distorting the accuracy of the evaluation. In contrast, the optimized device and optimized method for evaluating infectious arthritis proposed in this case refer to both grayscale knee tissue images and Doppler knee tissue images. Grayscale knee tissue images supplement the problem of discontinuous image features caused by using Doppler knee tissue images, while Doppler knee tissue images supplement the problem of fewer image features when using grayscale knee tissue images. Based on this, this case can reduce the time for evaluating infectious arthritis, optimize the evaluation process of infectious arthritis, and improve the efficiency of evaluating infectious arthritis.

The above is only a specific example of this case, and does not limit the scope of the patent application of this case. Therefore, all equivalent changes made by applying the content of this case are also included in the scope of this case and should be stated.

100: optimization device 110: ultrasound sensor 120: Doppler sensor 130: controller 210: grayscale knee tissue image 212: first small block 214: first small block 216: first image feature 220: Doppler knee tissue image 226: second image feature 231: tissue 235: vascular distribution feature 239: tissue 310a: visual converter 310b: visual converter 320: machine learning classifier 410: output data S610~S650: step

FIG. 1 is a block diagram of an optimized device for evaluating infectious arthritis according to an embodiment of the present invention.

FIG. 2 is an actual image of the grayscale knee tissue generated by an ultrasonic sensor sensing a portion of the knee area according to an embodiment of the present invention.

FIG. 3 is a Doppler knee tissue image with vascular distribution characteristics generated by sensing a portion of the knee area with a Doppler sensor according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of the data flow of an optimization method for evaluating infectious arthritis executed by an optimization device according to an embodiment of the present invention.

FIG. 5 is a schematic diagram of the visual converter of FIG. 4 processing the detailed data flow of the grayscale knee tissue image according to an embodiment of the present invention.

FIG6 is a flow chart of an optimized method for evaluating infectious arthritis according to an embodiment of the present invention.

210: Grayscale knee tissue image

216: First image feature

220: Doppler knee tissue image

226: Second image feature

310a: Visual converter

310b: Visual converter

320: Machine Learning Classifier

410: Output data

Claims (10)

An optimized device for evaluating infectious arthritis, comprising: an ultrasound sensor, configured to sense a grayscale knee tissue image covering a portion of a knee area; a Doppler sensor, configured to sense a Doppler knee tissue image having vascular distribution characteristics on the portion of the knee area, wherein the Doppler knee tissue image and the grayscale knee tissue image are tissue images corresponding to the same portion of the knee area; and a controller, coupled to the ultrasound sensor and the Doppler sensor, configured to: input the grayscale knee tissue image to a visual converter to capture a plurality of first image features of the grayscale knee tissue image; Input the Doppler knee tissue image to the visual converter to capture a plurality of second image features of the Doppler knee tissue image; Use the plurality of first image features and the plurality of second image features in a machine learning classifier to perform classification operations to generate a plurality of classification results corresponding to the tissue image of the same knee region and a probability value for each of the classification results; and Output the plurality of classification results with the probability value as evaluation information for evaluating infectious arthritis. The optimized device for evaluating infectious arthritis as described in claim 1, wherein the controller is configured to perform the following operations after inputting the grayscale knee tissue image to the visual converter: Executing the visual converter to cut the grayscale knee tissue image into a plurality of first small blocks; Executing a linear projection module of the visual converter to add a position information of each first small block in the grayscale knee tissue image to each first small block and adding a category information to which each first small block belongs in the grayscale knee tissue image to each first small block; and The plurality of first small blocks carrying the position information and the category information are input to a conversion encoder of the visual converter, and the conversion encoder is executed to analyze the plurality of first image features of the plurality of first small blocks. The optimized device for evaluating infectious arthritis as described in claim 2, wherein the controller is configured to perform the following operations after inputting the Doppler knee tissue image into the visual converter: Executing the visual converter to cut the Doppler knee tissue image into a plurality of second small blocks; Executing the linear projection module to add the position information of each second small block in the Doppler knee tissue image to each second small block and adding the category information to which each second small block belongs in the Doppler knee tissue image to each second small block; and The plurality of second small blocks carrying the position information and the category information are input to the transform encoder, and the transform encoder is executed to analyze the plurality of second image features of the plurality of second small blocks. An optimized device for evaluating infectious arthritis as described in claim 1, wherein the controller uses a plurality of sets of image pairs to train the machine learning classifier, wherein each of the image pairs includes a training grayscale knee tissue image and a training Doppler knee tissue image corresponding to the same knee region. An optimization device for evaluating infectious arthritis as described in claim 1, wherein the controller is configured to provide the classification result having the largest probability value as highly relevant information for evaluating infectious arthritis. An optimization method for evaluating infectious arthritis is performed using an optimization device, and the optimization method includes: Obtaining a grayscale knee tissue image corresponding to a portion of the knee area through an ultrasound sensor of the optimization device and obtaining a Doppler knee tissue image with vascular distribution characteristics corresponding to the same portion of the knee area through a Doppler sensor of the optimization device; Inputting the grayscale knee tissue image to a visual converter to capture a plurality of first image features of the grayscale knee tissue image; Inputting the Doppler knee tissue image to the visual converter to capture a plurality of second image features of the Doppler knee tissue image; Using the plurality of first image features and the plurality of second image features simultaneously in a machine learning classifier to perform classification operations to generate a plurality of classification results corresponding to the tissue image of the same knee region and a probability value for each of the classification results; and outputting the plurality of classification results with the probability value as evaluation information for evaluating infectious arthritis. The optimization method for evaluating infectious arthritis as described in claim 6, wherein after inputting the grayscale knee tissue image into the visual converter, the optimization method includes: Executing the visual converter to cut the grayscale knee tissue image into a plurality of first small blocks; Executing a linear projection module of the visual converter to add a position information of each first small block in the grayscale knee tissue image to each first small block and adding a category information to which each first small block belongs in the grayscale knee tissue image to each first small block; and The plurality of first small blocks carrying the position information and the category information are input to a conversion encoder of the visual converter, and the conversion encoder is executed to analyze the plurality of first image features of the plurality of first small blocks. The optimization method for evaluating infectious arthritis as described in claim 7, wherein after inputting the Doppler knee tissue image into the visual converter, the optimization method includes: Executing the visual converter to cut the Doppler knee tissue image into a plurality of second small blocks; Executing the linear projection module to add the position information of each second small block in the Doppler knee tissue image to each second small block and adding the category information to which each second small block belongs in the Doppler knee tissue image to each second small block; and The plurality of second small blocks carrying the position information and the category information are input to the transform encoder, and the transform encoder is executed to analyze the plurality of second image features of the plurality of second small blocks. The optimization method for evaluating infectious arthritis as described in claim 6 further includes using a plurality of sets of image pairs to train the machine learning classifier, wherein each of the image pairs includes a training grayscale knee tissue image and a training Doppler knee tissue image corresponding to the same knee region. The optimization method for evaluating infectious arthritis as described in claim 6, wherein the step of outputting the evaluation information further includes: Providing the classification result with the maximum probability value as highly relevant information for evaluating infectious arthritis.