TWI855947B - Device and method for analyzing cardiac ultrasound images - Google Patents

Abstract

The invention proposes a method for analyzing cardiac ultrasound images that is executed by an analysis device. This method includes: acquiring multiple cardiac ultrasound images; inputting the ultrasound images into a first machine learning model to generate multiple cycle detection results, which are configured to indicate whether each cardiac ultrasound image belongs to a systolic phase or a diastolic phase; inputting the ultrasound images into a second machine learning model to generate multiple segmentation results, which are configured to indicate atriums, valves, and regurgitation in the ultrasound images; and based on the cycle detection results and the segmentation results, executing a third machine learning model to generate an identification result which is configured to indicate the occurrence of atrioventricular valve regurgitation.

Description

Cardiac ultrasound image analysis device and method

The present disclosure relates to an apparatus and method for automatically analyzing ultrasound images, particularly for detecting atrioventricular valve regurgitation.

According to clinical epidemiological studies in Japan and Taiwan, mitral and tricuspid regurgitation are the first and third most common conditions among valvular heart diseases. However, compared with other developed countries, the proportion of these patients receiving treatment is relatively low, which in turn increases the public's medical burden. A number of clinical observational and interventional studies have confirmed that timely medical intervention, including appropriate drug therapy or valve replacement, can effectively improve patients' clinical prognosis and heart failure-related symptoms. Echocardiography is an important tool for diagnosing heart valvular regurgitation, but the old method requires experienced doctors to diagnose, which is time-consuming and labor-intensive.

The disclosed embodiment provides a cardiac ultrasound image analysis device, including a memory and a processor. The memory is used to store multiple instructions, and the processor is communicatively connected to the memory to execute these instructions to complete the following steps: obtaining multiple cardiac ultrasound images; inputting the ultrasound images into the first machine learning model to generate multiple cycle detection results, and these cycle detection results are used to indicate whether each cardiac ultrasound image belongs to the systolic period or the diastolic period; inputting the ultrasound images into the second machine learning model to generate corresponding multiple cutting results, and these cutting results are used to indicate the atria, valves and regurgitation in the ultrasound images; executing the third machine learning model according to the cycle detection results and the cutting results to generate identification results, and this identification result is used to indicate whether atrioventricular valve regurgitation occurs.

In some embodiments, the first machine learning model includes a recurrent neural network, and the cycle detection results include a plurality of binary images, the binary images include a plurality of values, and the values are the same and are used to indicate whether the corresponding ultrasound image belongs to the systolic period or the diastolic period.

In some embodiments, the step of executing the third machine learning model based on the cycle detection results and the cutting results includes: for one of the cardiac ultrasound images, performing element-wise multiplication operations on the corresponding cycle detection results, the atrium cutting results, and the reverse flow cutting results to obtain an intersection cutting result.

In some embodiments, the step of executing the third machine learning model based on the cycle detection results and the cutting results further includes: connecting the intersection cutting results, the cardiac ultrasound image and the valve cutting results and inputting them into the third machine learning model.

In some embodiments, the third machine learning model is a capsule network, and the intersection cutting result, the cardiac ultrasound image, and the valve cutting result belong to one of a plurality of viewpoints. The third machine learning model includes a plurality of input branches corresponding to the viewpoints, each input branch is used to generate a viewpoint recognition result, and the third machine learning model is used to combine the viewpoint recognition results to generate a recognition result.

From another perspective, the embodiment of the present disclosure proposes a cardiac ultrasound image analysis method, which is executed by an analysis device. The analysis method includes: obtaining multiple cardiac ultrasound images; inputting the ultrasound images into a first machine learning model to generate multiple cycle detection results, which are used to indicate whether each cardiac ultrasound image belongs to the systolic period or the diastolic period; inputting the ultrasound images into a second machine learning model to generate corresponding multiple cutting results, which are used to indicate the atria, valves and regurgitation in the ultrasound images; executing a third machine learning model based on the cycle detection results and the cutting results to generate a recognition result, which is used to indicate whether atrioventricular valve regurgitation occurs.

The terms “first,” “second,” etc. used herein do not particularly refer to order or sequence, but are only used to distinguish elements or operations described with the same technical term.

FIG1 is a diagram of an analysis device for cardiac ultrasound images according to an embodiment. Referring to FIG1 , the analysis device 100 may be a personal computer, a laptop, a server, a distributed computer, a cloud server, an industrial computer, a smart phone, a tablet computer, or various electronic devices with computing capabilities, but the present invention is not limited thereto. The analysis device 100 includes a processor 110 and a memory 120, and the processor 110 is communicatively connected to the memory 120, wherein the communication connection may be achieved through any wired or wireless communication means, or may also be achieved through the Internet. The processor 110 may be a central processing unit, a microprocessor, a microcontroller, an image processing chip, a special application integrated circuit, etc. The memory 120 may be a random access memory, a read-only memory, a flash memory, a floppy disk, a hard disk, an optical disk, a flash drive, a magnetic tape, or a database accessible via the Internet, wherein a plurality of instructions are stored. The processor 110 executes these instructions to complete a cardiac ultrasound image analysis method, which will be described in detail below.

First, the annotations of cardiac ultrasound images are explained. These annotations are provided by professional physicians. First, the annotations include which view angle each cardiac ultrasound image belongs to. These view angles may include parasternal long axis (PLAX), parasternal short axis (PSAX), apical four chambers (A4C), apical three chambers (A3C), apical two chambers (A2C) or subcostal view (SC), etc., but the present disclosure is not limited thereto. Second, the annotations also include each leaflet of the mitral valve (anterior leaflet and posterior leaflet) and the tricuspid valve (septum, anterior leaflet and posterior leaflet), and the four nodes of each leaflet are evenly marked from the valve annulus to the tip. Third, the ventricles and atria, regurgitation, and inflow were labeled in 2D and color mode. Finally, for mitral and tricuspid regurgitation grading, the original structured reports authorized by experienced cardiologists were extracted.

FIG. 2 is a flowchart of a cardiac ultrasound image analysis method according to an embodiment. Referring to FIG. 2 , a plurality of cardiac ultrasound images to be analyzed are first obtained. These cardiac ultrasound images may be color images or grayscale images, but the present disclosure is not limited thereto. The cardiac ultrasound images may be identified to which viewing angle they belong through a machine learning model (not shown). Here, cardiac ultrasound images 201 to 205 are used as an example to illustrate that these cardiac ultrasound images 201 to 205 belong to the same viewing angle. In this embodiment, there are multiple (e.g., 20) cardiac ultrasound images at each viewing angle to form a video, but the present disclosure does not limit the number of cardiac ultrasound images 201 to 205.

Next, the cardiac ultrasound images 201-205 of the same viewing angle are input into the machine learning model 210 to generate a plurality of cycle detection results. These cycle detection results correspond to the cardiac ultrasound images 201-205 respectively, and are used to indicate whether the corresponding cardiac ultrasound images belong to the diastole 211 or the systole 212. For example, the cardiac ultrasound images 201-202 belong to the diastole 211, the cardiac ultrasound images 203-204 belong to the systole 212, and so on. The machine learning model 210 is also called an event detector. FIG3 is a network architecture diagram of a cycle detector according to an embodiment. 3, the machine learning model 210 includes a three-dimensional convolutional neural network 310, recurrent neural networks 320, 330, and a one-dimensional convolutional layer 340. For example, the cardiac ultrasound images 201-205 can be connected to each other to form a three-dimensional feature map, which is then input into the three-dimensional convolutional neural network 310, and then the temporal correlation is processed by the recurrent neural networks 320, 330. In this embodiment, the recurrent neural networks 320, 330 are long short-term memory networks (Long Short-Term Memory, LSTM), but in other embodiments, they can also be suitable networks such as gated recurrent units (GRU), and the present disclosure is not limited thereto. The output of the circulatory neural network 330 passes through the one-dimensional convolution layer 340, and then outputs multiple values 341-343, each of which corresponds to a cardiac ultrasound image. Here, the value "1" represents the systolic period, and the value "0" represents the diastolic period. For the sake of ease of reading, the text description "3D CNN" is added to the three-dimensional convolution neural network 310 in FIG. 3; the text description "LSTM" is added to the circulatory neural networks 320 and 330; and the text description "1D Conv" is added to the one-dimensional convolution layer 340.

Next, the value 341 is converted into a binary image 351, the value 342 is converted into a binary image 352, and so on. Each binary image 351-353 has the same value, which is used to indicate whether the corresponding cardiac ultrasound image belongs to the systolic period or the diastolic period. These binary images 351-353 form the above-mentioned multiple cycle detection results. The binary image (rather than a single value) is generated for use in the subsequent convolution layer, which will be explained in detail below.

Please go back to Figure 2. On the other hand, the cardiac ultrasound images 201~205 will also be input into the machine learning model 220 to generate multiple cutting results 221~223, wherein the cutting result 221 is used to indicate the reverse flow in the cardiac ultrasound image, and the cutting result 222 is used to indicate the ventricle and atrium. The cutting result 223 is used to indicate the position of the valve, for example, several points can be used to represent the leaflets of the valve. Figure 3 is a network architecture diagram of the machine learning model 220 according to an embodiment. Please refer to Figure 3. In this embodiment, the machine learning model 220 adopts a stacked hourglass (SHG) model, as shown in Figure 4, but in other embodiments, a U-network, a full convolution network, or other suitable networks can also be used, and the present disclosure is not limited thereto.

Please return to FIG. 2 . Next, the machine learning model 240 is executed according to the cycle detection results output by the machine learning model 210 and the cutting results output by the machine learning model 220 to generate the identification result 241. The identification result 241 is used to indicate whether atrioventricular regurgitation occurs. For example, the cycle detection results and all the cutting results can be connected together to form a larger feature graph, which is then submitted to the machine learning model 240 for identification. In this embodiment, medical knowledge is also combined to design the identification process. Specifically, if the reflux caused by the mitral valve and the tricuspid valve is to be detected, since these refluxes only occur in the atrium and only occur during the systolic period, for a certain cardiac ultrasound image, the corresponding cycle detection result (binarized image in FIG. 3 ), the atrium cutting result 222, and the reflux cutting result 221 can be element-wise multiplied 230. In other words, the length and width of the binary image, the cutting result 222, and the cutting result 221 are the same. For example, in the cycle detection result, the value "1" indicates the systolic period, in the atrium cutting result 222, the value "1" indicates the atrium, and in the reverse flow cutting result 221, the value "1" indicates the occurrence of reverse flow. Therefore, the element multiplication operation 230 is equivalent to calculating the intersection of the systolic period, the atrium, and the reverse flow. Through such an operation, some errors can be avoided and the recognition accuracy can be improved. The element multiplication operation 230 will generate the intersection cutting result 231. This intersection cutting result 231, the cardiac ultrasound images 201-205, and the valve cutting result 223 are connected and input into the machine learning model 240.

The above operations are for the cardiac ultrasound images 201-205 at a certain viewing angle. In some embodiments, the machine learning model 240 can output the recognition result 241 based on the cardiac ultrasound images at only one viewing angle. In other embodiments, the machine learning model 240 can also consider multiple viewing angles and output the recognition result 241 by combining the images at multiple viewing angles. The cardiac ultrasound images at other viewing angles can also be processed in the same way as the cardiac ultrasound images 201-205. Therefore, if there are n viewing angles, a total of n machine learning models 210 and n machine learning models 220 need to be trained, where n is a positive integer.

In some embodiments, the machine learning model 240 uses a capsule network. FIG. 5 is a schematic diagram of the structure of a capsule network according to an embodiment. Referring to FIG. 5 , the machine learning model 240 includes a plurality of input branches 501-502, each of which corresponds to a viewpoint. For the input branch 501, the intersection cutting result 231, the cardiac ultrasound images 201-205, and the valve cutting result 223 generated above are connected to form a feature map 511. The feature map 511 is input to the feature extractor 521 to generate a feature vector 531, which is then passed through a classifier (not shown) to generate a viewpoint recognition result 541. For example, the view angle recognition result 541 includes two values, which respectively indicate whether the atrioventricular valve regurgitation is significant or not significant at this view angle. In other embodiments, the view angle recognition result 541 may also include a score to indicate the severity of the reflux. Similarly, for the input branch 502, a corresponding feature map 512 may also be generated, and a corresponding feature vector 532 and a view angle recognition result 542 may be generated through the feature extractor 522. The view angle recognition results 541-542 of different view angles are combined into a vector 551, and then output as a recognition result 561 through an identifier (not shown). The recognition result 561 also includes two numerical values, respectively indicating whether the atrioventricular valve regurgitation is significant or not significant.

The above-mentioned atrioventricular regurgitation may represent mitral valve regurgitation and/or tricuspid valve regurgitation, or in other embodiments, other valve regurgitation may also be detected. A person with ordinary knowledge in the field may modify the above-mentioned network architecture according to different detection targets. For example, if a certain valve regurgitation only occurs in the diastole, "1" may be set as the diastole in the cycle detection result; if a certain valve regurgitation only occurs in the ventricle, the corresponding cutting result of the ventricle may be taken to perform the above-mentioned element-by-element multiplication operation.

FIG6 is a schematic diagram showing experimental results according to an embodiment. FIG6 shows the cutting result 601 of the mitral valve, the cutting result 602 of the tricuspid valve, the cutting result 603 of the ventricle and atrium, the cutting result 604 of the reverse flow, and the cutting result 605 of the inflow. In FIG6, the cutting results of the left and right ventricles and atria are displayed in the same figure, but the left and right ventricles and atria can have different labels.

FIG7 is an experimental result after being restricted by an element-wise multiplication operation according to an embodiment. Referring to FIG7 , a reverse flow region 711 is indicated in the cutting result 710, but this reverse flow region 711 covers the ventricle. After the above-mentioned element-wise multiplication operation, the cutting result can be restricted to the atrium to obtain a cutting result 720, which can more accurately indicate the reverse flow region 721.

FIG8 is a flowchart of a cardiac ultrasound image analysis method according to an embodiment. In step 801, a plurality of cardiac ultrasound images are obtained. In step 802, the ultrasound images are input into a first machine learning model to generate a plurality of cycle detection results, which are used to indicate whether each cardiac ultrasound image belongs to the systolic phase or the diastolic phase. In step 803, the ultrasound images are input into a second machine learning model to generate a plurality of corresponding cutting results, which are used to indicate the atria, valves and regurgitation in the ultrasound images. In step 804, the third machine learning model is executed according to the cycle detection result and the cutting result to generate an identification result, and the identification result is used to indicate whether atrioventricular valve regurgitation occurs. The steps in FIG8 have been described in detail above and will not be repeated here. It is worth noting that the steps in FIG8 can be implemented as multiple program codes or circuits, and the present invention is not limited thereto. In addition, the method of FIG8 can be used in conjunction with the above embodiments or can be used alone. In other words, other steps can be added between the steps in FIG8.

From another perspective, the present invention also proposes a computer program product, which can be written in any programming language and/or platform. When the computer program product is loaded into a computer system and executed, the above-mentioned analysis method can be executed.

In addition to the above examples, the machine learning model mentioned in the present disclosure may also be a decision tree, a random forest, a multi-layer neural network, a support vector machine, etc., but the present invention is not limited thereto. When a convolutional neural network is used, in addition to the specific examples provided above, the architecture of these networks may adopt LeNet, AlexNet, VGG, GoogLeNet, ResNet, DenseNet or YOLO (You Only Look Once), etc. The loss function used may be mean square error (MSE), mean absolute error (MAE), cross-entropy, Huber loss function, Log-Cosh loss function, etc. The procedure for updating parameters may adopt gradient descent, backpropagation, etc., but the present disclosure is not limited thereto.

Although the present invention has been disclosed as above by the embodiments, they are not intended to limit the present invention. Any person with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be defined by the scope of the attached patent application.

100:Analysis device

110:Processor

120: Memory

201~205: Cardiac ultrasound images

210,220,240: Machine learning models

211: Diastole

212: Contraction period

221~223: Cutting results

230: Element-wise multiplication

231: Intersection cutting result

241: Identification results

310: 3D Convolutional Neural Network

320,330:Circulatory neural network

340: One-dimensional convolution layer

341~343:Number

351~353: Binarized image

501,502: Input branch

511,512: Feature graph

521,522: Feature Extractor

531,532: Feature vector

541,542: Visual recognition results

551: Vector

561:Identification results

601~605,710,720: cutting results

711,721: Countercurrent area

801~804: Steps

In order to make the above features and advantages of the present invention more clearly understandable, the following embodiments are specifically cited and detailed with the attached figures. FIG. 1 is a diagram of an analysis device for cardiac ultrasound images according to an embodiment. FIG. 2 is a flow chart of a method for analyzing cardiac ultrasound images according to an embodiment. FIG. 3 is a network architecture diagram of a cycle detector according to an embodiment. FIG. 4 is a network architecture diagram of a stacked hourglass model according to an embodiment. FIG. 5 is a schematic diagram of the architecture of a capsule network according to an embodiment. FIG. 6 is a schematic diagram of experimental results according to an embodiment. FIG. 7 is an experimental result after being restricted by element multiplication operation according to an embodiment. FIG8 is a flowchart of a cardiac ultrasound image analysis method according to an embodiment.

201~205: Cardiac ultrasound images

210,220,240: Machine learning model

211: Diastole

212: Contraction period

221~223: Cutting results

230: Element-wise multiplication

231: Intersection cutting result

241: Identification results

Claims (10)

A cardiac ultrasound image analysis device includes: A memory for storing a plurality of instructions; and A processor, communicatively connected to the memory, for executing the instructions to complete a plurality of steps: Acquire a plurality of cardiac ultrasound images; Input the ultrasound images into a first machine learning model to generate a plurality of cycle detection results, the cycle detection results are used to indicate that each of the cardiac ultrasound images belongs to a systolic period or a diastolic period; Input the ultrasound images into a second machine learning model to generate a plurality of corresponding cutting results, the cutting results are used to indicate a plurality of atria, a plurality of valves and a regurgitation in each of the ultrasound images; A third machine learning model is executed based on the cycle detection results and the cutting results to generate a recognition result, and the recognition result is used to indicate whether an atrioventricular valve regurgitation occurs. An analysis device as described in claim 1, wherein the first machine learning model includes a recurrent neural network, the cycle detection results include multiple binary images, one of the binary images includes multiple values, which are the same as each other and are used to indicate whether the corresponding ultrasound image belongs to the systolic period or the diastolic period. The analysis device as described in claim 2, wherein the step of executing the third machine learning model according to the cycle detection results and the cutting results includes: For one of the cardiac ultrasound images, perform an element-wise multiplication operation on the corresponding cycle detection result, the cutting result of the atria, and the cutting result of the reverse flow to obtain an intersection cutting result. The analysis device as described in claim 3, wherein the step of executing the third machine learning model according to the cycle detection results and the cutting results further includes: Connecting the intersection cutting results, the cardiac ultrasound images and the cutting results of the valves and inputting them into the third machine learning model. An analysis device as described in claim 4, wherein the third machine learning model is a capsule network, the intersection cutting result, the cardiac ultrasound images and the cutting result of the valves belong to one of a plurality of viewing angles, the third machine learning model comprises a plurality of input branches corresponding to the viewing angles, each of the input branches is used to generate a viewing angle recognition result, and the third machine learning model is used to combine the viewing angle recognition results to generate the recognition result. A cardiac ultrasound image analysis method is performed by an analysis device, the analysis method comprising: Acquiring a plurality of cardiac ultrasound images; Inputting the ultrasound images into a first machine learning model to generate a plurality of cycle detection results, the cycle detection results are used to indicate whether each of the cardiac ultrasound images belongs to a systolic phase or a diastolic phase; Inputting the ultrasound images into a second machine learning model to generate a plurality of corresponding cutting results, the cutting results are used to indicate a plurality of atria, a plurality of valves and a regurgitation in each of the ultrasound images; and A third machine learning model is executed based on the cycle detection results and the cutting results to generate a recognition result, and the recognition result is used to indicate whether an atrioventricular valve regurgitation occurs. An analysis method as described in claim 6, wherein the first machine learning model comprises a recurrent neural network, the cycle detection results comprise a plurality of binary images, one of the binary images comprises a plurality of numerical values, the numerical values are the same as each other and are used to indicate whether the corresponding ultrasound image belongs to the systolic period or the diastolic period. The analysis method as described in claim 7, wherein the step of executing the third machine learning model according to the cycle detection results and the cutting results includes: For one of the cardiac ultrasound images, perform an element-wise multiplication operation on the corresponding cycle detection result, the cutting result of the atria, and the cutting result of the reverse flow to obtain an intersection cutting result. The analysis method as described in claim 8, wherein the step of executing the third machine learning model based on the cycle detection results and the cutting results also includes: Connecting the intersection cutting results, the cardiac ultrasound images and the cutting results of the valves and inputting them into the third machine learning model. An analysis method as described in claim 9, wherein the third machine learning model is a capsule network, the intersection cutting result, the cardiac ultrasound images and the cutting result of the valves belong to one of a plurality of viewing angles, the third machine learning model comprises a plurality of input branches corresponding to the viewing angles, each of the input branches is used to generate a viewing angle recognition result, and the third machine learning model is used to combine the viewing angle recognition results to generate the recognition result.