

ORIGINAL ARTICLE 

Year : 2021  Volume
: 11
 Issue : 3  Page : 177184 

Speckle tracking accuracy enhancement by temporal superresolution of threedimensional echocardiography images
Mohammad Jalali, Hamid Behnam
Department of Biomedical Engineering, Iran University of Science and Technology, Tehran, Iran
Date of Submission  27Apr2020 
Date of Decision  30May2020 
Date of Acceptance  27Aug2020 
Date of Web Publication  21Jul2021 
Correspondence Address: Hamid Behnam Department of Biomedical Engineering, Iran University of Science and Technology, Tehran Iran
Source of Support: None, Conflict of Interest: None
DOI: 10.4103/jmss.JMSS_26_20
Background: Speckle tracking has always been a challenging issue in echocardiography images due to the lowcontrast and noisy nature of ultrasonic imaging modality. While in ultrasound imaging, framerate is limited by image size and sound speed in tissue, speckle tracking results get worse inthreedimensional imaging due to its lower frame rate. Therefore, numerous techniques have beenreported to overcome this limitation and enhance tracking accuracy. Methods: In this work, we have proposedto increase the frame rate temporally for a sequence of threedimensional (3D) echocardiographyframes to make tracking more accurate. To increase the number of frames, cubic Bspline is usedto interpolate between intensity variation time curves extracted from every single voxel in theimage during the cardiac cycle. We have shown that the frame rate increase will result in trackingaccuracy improvement. Results: To prove the efficiency of the proposed method, numerical evaluation metricsfor tracking are reported to make a comparison between high temporal resolution sequences andlow temporal resolution sequences. Anatomical affine optical flow is selected as the stateoftheartspeckle tracking method, and a 3D echocardiography dataset is used to evaluate the proposedmethod. Conclusion: Results show that it is beneficial for speckle tracking to perform on temporally condensedframes rather than ordinary clinical 3D echocardiography images. Normalized mean enhancementvalues for mean absolute error, Hausdorff distance, and Dice index for all cases and all frames are0.44 ± 0.09, 0.42± 0.09, and 0.36 ± 0.06, respectively.
Keywords: Cubic Bspline interpolation, speckle tracking, temporal superresolution, threedimensional echocardiography
How to cite this article: Jalali M, Behnam H. Speckle tracking accuracy enhancement by temporal superresolution of threedimensional echocardiography images. J Med Signals Sens 2021;11:17784 
How to cite this URL: Jalali M, Behnam H. Speckle tracking accuracy enhancement by temporal superresolution of threedimensional echocardiography images. J Med Signals Sens [serial online] 2021 [cited 2021 Jul 23];11:17784. Available from: https://www.jmssjournal.net/text.asp?2021/11/3/177/322006 
Introduction   
Echocardiography has become the first imaging choice of most cardiologists today because the physical characteristics of sound waves have made it harmless to living tissues and also its portability compared to magnetic resonance imaging (MRI) systems has made it accessible almost everywhere. Besides, it is real time, handy, and inexpensive and no considerable postprocessing is needed after data acquisition to prepare the result. Albeit, from the very 1^{st} days of ultrasound medical diagnostic usage, there have been challenging issues, namely poor spatial resolution and the inherent noise of ultrasound images.
In threedimensional (3D) echocardiography, there are naturally fewer frames compared to its 2D counterpart that is due to the longer time needed to sweep a volume rather than a single plain. This shortcoming has made 3D images unable to follow fastmoving body organs. Hence, many works have been reported to overcome these drawbacks among which speckle tracking is an important topic in 3D echocardiography images specifically for cardiac functional assessment.
Tracking is an image processing procedure under which an image point of interest is pursued frame by frame. In echocardiography literature, speckle tracking is to follow a speckle pattern in an ultrasonic image frame by frame until a complete cardiac cycle. 3D speckle tracking echocardiography is a useful tool for regional and global cardiac assessment. There are three major categories of speckle tracking based on cardiac motion calculation, namely block matching, optical flow, and elastic image registration.^{[1]}
In methods based on Block matching,^{[2],[3]} first, a patch of specific size is extracted around the point of interest in the frame n. Next, based on similarity measure and within a defined search span, the most similar patch in frame n + 1 is considered to be the corresponding patch in the next frame which has moved from the frame n and this procedure continues for subsequent frames. There are different choices for the similarity measure and different results can be obtained based on the selection of this measure and it is completely problem dependent.
Optical flow tracking assumes that pixel/voxel brightness intensity does not change significantly when it is moving from one frame into another. An optimization function is defined which penalizes intensity variation, and by optimizing it, movement vector between frames is obtained. This function can be defined locally or globally on the whole image, leading to different results.^{[4],[5],[6],[7]}
Finally, elastic image deformation computes a deformation field as an alternative to tracking points by defining and optimizing a cost function. Several properties can be exerted to the resulting field as constraints applied to the cost function.^{[8],[9],[10]} To make calculations easier, cardiac deformation can be parameterized by its decomposition on a set of basis functions, for example, Bsplines.^{[9],[11]}
In Queiros et al.'s study,^{[12]} medical image tracking toolbox (MITT) has been developed to facilitate the customization of medical image tracking solutions. It is a software package developed in both C/C++ and MATLAB which offers computationally efficient tracking solutions for 2D and 3D echocardiography image sequences. Currently, its implementation includes an objectbased image tracking module based on both global and localized aanatomical affine optical flow (AAOF)^{[13]} algorithms. In this study, we have used the same software package to track 3D+t echocardiography data (i.e., 3D sequence of echocardiography images acquired through time) using localized AAOF.
In our previous study,^{[14]} we have shown that temporal interpolation of echocardiography frame intensities for every pixel/voxel (i.e., intensity variation time curves [IVTCs]) through cubic Bsplines can help improve medical diagnostics. Bsplines are a class of smooth and continuous functions that can be fit to a set of discrete points (i.e., knots). The degree of Bspline is the degree of the polynomial function used to fit between every two consecutive discrete points. If a Bspline of order n is used, then the whole function and its next (n – 1) derivatives are continuous at the knot points, leading to the smoothness of fitted function. We have also shown that the best degree for Bspline functions to interpolate echocardiography images temporally is three and the set of functions are called cubic Bsplines. One advantage of spline functions is that they can do interpolation efficiently using linear filtering^{[15]} which makes hardware implementation of cubic Bspline interpolation possible.
In the current work, the efficiency of temporal cubic Bspline interpolation on 3D echocardiography speckle tracking will be proved. To this end, anatomical affine optical flow as the stateoftheart speckle tracking method is used to track 3D echocardiography endocardial borders through a cardiac cycle in both the original sequence of frames (we will call them as the lowresolution frames) and temporally interpolated frames (called the highresolution frames) on a synthetic dataset of 3D echocardiography sequences available online,^{[1]} which will be described in detail later on. We have selected this dataset because the endocardium surface in every single volume is delineated exactly which makes tracking evaluation to be done more precisely.
The rest of the study is structured as follows. In the next section, “Materials and Methods,” main concepts are described in brief. Both the interpolation procedure and the dataset used are described and also metrics to evaluate tracking accuracy are defined. In the “Results” section, tracking evaluation results are demonstrated, and finally, in the “Conclusions” section, results are summarized and the study is concluded.
Materials and Methods   
Cubic Bspline interpolation
IVTCs are signals extracted from intensity variations of a single pixel/voxel through time in a sequence of 2D or 3D echocardiography. In [Figure 1], a typical IVTC extracted from a 2D echocardiography frame sequence is demonstrated. If we consider the specific pixel from which IVTC is extracted, every value of IVTC corresponds to that pixel intensity value in the corresponding frame.  Figure 1: Formation of a typical intensity variation time curve in a twodimensional echocardiography sequence
Click here to view 
Bspline fitting^{[15]} of a continuous function is performed by convolving IVTC and Bspline basis function:
In the above equation, c_{i} is the intensity value of IVTC and χn is the Bspline basis function of order n. The resulting χn is the continuous function fitted to discrete IVTC points. Now, if the continuous function is sampled with a frequency greater than the original frame rate, we have interpolated the signal.
Bspline basis functions have the property which helps to create each basis spline of order n from its previous order n–1 and zeroorder basis spline:
where * demonstrates a convolution operator. Thus, using χ^{(0)} (x), all other basis splines can be obtained. Zeroorder basis spline is defined as:
In Jalali et al.'s study,^{[14]} we have shown that the best order for Bspline interpolation is 3, called cubic Bspline. Therefore, all interpolations in this study are calculated using the thirdorder Bsplines. In [Figure 2], the result of cubic Bspline interpolation for a sample IVTC is demonstrated. Consider all IVTCs are interpolated with the same factor, and the number of frames is increased by the same factor.  Figure 2: Cubic Bspline interpolation of a typical intensity variation time curves
Click here to view 
Dataset
In this study, we have used a synthetic dataset^{[1]} which resembles real 3D echocardiography recordings, in which every frame includes myocardial, epicardial, and endocardial borders delineated from which we have used endocardium. This dataset has been used for detail evaluation of tracking methods in the original article and contains 8 sequences of 3D echocardiogram corresponding to different pathophysiological conditions, which are four ischemic cases containing occlusion of proximal left anterior descending coronary artery (LADprox), occlusion of distal left anterior descending coronary artery (LADdist), occlusion of left circumflex coronary artery (LCX), occlusion of right coronary artery (RCA), three cases with cardiomyopathy containing cardiomyopathy with synchronous activation pattern (sync), dyssynchronous due to left branch bundle block with a small delay in activation of the septum and lateral wall (LBBBsmall), dyssynchronous due to left branch bundle block with a longer delay in activation of the septum and lateral wall (LBBB), and a normal case (normal).
Anatomical affine optical flow
Anatomically constrained affine optical flow (AAOF) is the stateoftheart medical image tracking algorithm^{[1]} that can be used to track 3D speckle patterns in echocardiography images. It is the 3D extension of the work proposed by Queiros et al.^{[13]} for fast left ventricle tracking in MRI cardiac imaging. It estimates the motion between consecutive frames using optical flow. To make tracking computationally more efficient, motion estimation is calculated in a constraint region around the tracked surface. This also avoids the effect of nearby cardiac tissues and makes tracking procedure more robust. Therefore, a local motion estimation is performed by an affine motion model in which every single point is only using the motion of neighbor points and their relative distances. Depending on the affine motion to use a local or global model, AAOF can perform tracking locally or globally. Affine motion parameters are estimated by solving an optimization equation. To be concise, interested readers are referred to as the original articles above.
The reason for choosing AAOF among all the other medical image tracking methods is the superior performance reported in Alessandrini et al.'s study^{[1]} in which this method performs more precisely compared to four other leading medical image tracking methodologies. These five methods are compared in the sense of 3D speckle tracking and strain accuracy. According to the reference,^{[1]} AAOF outperforms compared to the rest and we have selected the best performing algorithm to demonstrate accuracy enhancement using temporal super resolution.
Tracking scenario
To do the tracking, the endocardial surface from the first frame of each sequence has been used as the beginning point of the tracking, and then tracked borders have been reused to be tracked to the next frame. This procedure continues to the last frame. In lowresolution frames, every frame has ground truth surfaces delineated, and as a result, tracked border can be compared with that of the ground truth in each frame, as shown in [Figure 3]. However, in highresolution frames, one frame in every four consecutive frames has ground truth border since three frames are added in the interpolation step, and therefore, no border exists as the ground truth. As a result, only borders from frames with ground truth are stored and other frames (i.e., interpolated frames) are ignored after tracking and they only act as the catalyst, as shown in [Figure 4].  Figure 3: Lowresolution frame tracking. The first row (blue arrows) shows the frames with ground truth borders and the second row (orange arrows) shows frames with tracked borders that are compared (vertical green arrows) with the first row for evaluation of tracking
Click here to view 
 Figure 4: Highresolution frame tracking. The first row (blue arrow) shows the frames with ground truth borders and the second row (orange arrows) shows the frames with tracked borders. Three first tracked borders (shown in red border) are ignored and the fourth is compared (green vertical arrow) with the corresponding ground truth
Click here to view 
Evaluation metrics
Comparing tracked surfaces with the ground truth to evaluate the quality of tracking is an essential part of measuring tracking improvement. Accordingly, three metrics are used in this work that is described precisely here.
Hausdorff distance
The Hausdorff distance is a measure of local maximum distance between two surfaces. Consider S_{gt} to be the ground truth surface and S_{tr} the tracked surface. Then, the Hausdorff distance between two surfaces is defined as:
Where h(.,.) is called the directed Hausdorff distance given by:
And . is a typical norm like Euclidean distance.
Dice similarity coefficient
The Dice coefficient is a measure of overlap between the ground truth volume (V_{tr}) and the tracked volume (V_{tr}) giving a value between 0 and 1. Here, volume means the space surrounded by the surface. Fully overlapping volumes have 1 and nonoverlapping volumes have 0 value of the Dice coefficient. This coefficient is calculated as:
In the above equation, the numerator calculates the overlapping volume between the ground truth and tracked volumes, and the denominator is the sum of these volumes.
Mean absolute distance
The mean value of the absolute distance between each pair of points, one in the ground truth surface and the other in the tracked surface, is called mean absolute distance calculated as:
In this equation, E [.] is the expected value operator and X_{gt and }X_{tr} are the coordinate vector of points on the ground truth and tracked surface, respectively.
Results   
Here, in [Figure 5], we have demonstrated tracking error for different interpolation factors during the time. As expected, more frames added to the original sequence, better tracking accuracy is reached. We should note that calculation for large interpolation factors (e.g., five, six, or bigger) is a burden since, for every single interpolation factor increase, data of the same size as the original sequence have to be allocated in memory and considered in the calculation procedures. The interpolation factor used in this work is 4 for all 3D data used. Due to large memory requirements and calculation time, selecting bigger factors is not feasible. Interpolation by factor 4 means we have added 3 points between every two consecutive points in IVTC signals (i.e., we have added 3 volumes between every two consecutive 3D echocardiography volumes in the original sequence).  Figure 5: Comparison between different interpolation factors according to tracking accuracy
Click here to view 
Tracking has been performed on lowresolution (original) and highresolution (temporally interpolated) sequences. Evaluation metrics have been calculated for each frame for both lowresolution and highresolution images (regardless of interpolated frames) afterward. [Figure 6], [Figure 7], [Figure 8] demonstrate three metrics for each sequence of the dataset in both low and high resolutions for every single volume of the sequence indicated sequentially.  Figure 6: Mean absolute distance for threedimensional echocardiography for lowresolution (shown in red) and highresolution (shown in green) sequences of cases: (a) left anterior descending coronary artery, (b) left anterior descending coronary artery, (c) left circumflex coronary artery, (d) right coronary artery, (e) Sync, (f) LBBBsmall, (g) LBBB, (h) normal
Click here to view 
 Figure 7: Dice similarity coefficient for threedimensional echocardiography for lowresolution (shown in red) and highresolution (shown in green) sequences of cases: (a) left anterior descending coronary artery, (b) left anterior descending coronary artery, (c) left circumflex coronary artery, (d) right coronary artery, (e) Sync, (f) LBBBsmall, (g) LBBB, (h) normal
Click here to view 
 Figure 8: Hausdorff distance for threedimensional echocardiography for lowresolution (shown in red) and highresolution (shown in green) sequences of cases: (a) left anterior descending coronary artery, (b) left anterior descending coronary artery, (c) left circumflex coronary artery, (d) right coronary artery, (e) Sync, (f) LBBBsmall, (g) LBBB, (h) normal
Click here to view 
Notice that for each sequence, the ground truth surface has been considered as the beginning surface in the tracking procedure. Thus, for both low and highresolution sequences, the Dice similarity measure value is one, and Hausdorff and mean absolute distance values are zero.
These results were represented to a cardiology expert to evaluate the proposed work in clinical practice. Both lowresolution and highresolution videos were played without any label and it was asked to mention both pros and cons of these two. From a cardiologist's point of view, left ventricle volume change, which is an important factor for ejection fraction determination, is clearer in the highresolution video and more details are tractable during a cardiac cycle. The subtle movements and contractions of the heart walls are seen more clearly when interpolated frames are added to the original sequence, and it is easier to see the slight movement disorders of the ventricular wall which were not tractable in the lowresolution video.
Conclusions   
In the present study, we proposed to increase the number of frames in 3D echocardiography sequences to enhance the accuracy of speckle tracking of the endocardial surface. To this end, we used anatomical affine optical flow tracking as the stateoftheart speckle tracking method. We demonstrated that using cubic Bspline interpolation for frame rate enhancement helps to improve tracking precision compared to the original sequences.
Three metrics were used to compare the tracked surface with ground truth. Results show that using our proposed method can make tracking algorithm to perform better and follow spatial patterns more accurately. The reason is that in fastmoving organs such as cardiac muscles and valves, speckle pattern changes constantly and tracking algorithm has to adapt itself to new patterns in almost every new frame. Frames added to the original sequence can cause this pattern changes to happen more smoothly in a more continuous manner. Therefore, speckle patterns become more tractable.
It is inferred from the results that for the first few frames of each sequence, surfaces look closer compared to subsequent frames. This can happen due to the cumulative error that tracking algorithms have in nature. Since tracking each frame is initialized with the surface from the previously tracked frame, faulty tracking of previous frames causes a bigger error in the next frame tracking. Therefore, at first, metrics for low and highresolution tracking are closer and then separate while lowresolution tracking accumulates more errors.
The other thing that infers from the result section figures is that wherever the image movements are slower, low and highresolution tracking metrics are closer. The reason is that adding more frames to slowly moving images does not make a great difference. If two frames are close enough for tracking algorithm to succeed, providing more frames does not change the result greatly. Thus, in final frames, where cardiac cycle is closing to end diastole, metrics become closer.
As a future study, we plan to use frame rate enhancement by cubic Bspline interpolation as a preprocessing step in deep learning neural network algorithms that take advantage of temporal information of medical videos, like long shortterm memory networks, as an augmentation to training data. We hope this will enhance the network learning procedure and result in overall performance improvement specifically for 3D ultrasound imaging sequences since they lack low temporal resolution. As another suggestion, it would be beneficial to use Bspline frame rate increase for MRI images to make tracking more precisely, since low temporal resolution is a burden in MRI images.
Acknowledgment
I would like to thank Dr. Sandro Queirós from the University of Minho for sharing his knowledge and guidance to use the MITT in this study. In addition, I thank Dr. Maryam Shojaeifard for her kind comments on resulting videos that helped us better evaluate our results.
Financial support and sponsorship
None.
Conflicts of interest
There are no conflicts of interest.
References   
1.  Alessandrini M, Heyde B, Queiros S, Cygan S, Zontak M, Somphone O, et al. Detailed evaluation of five 3D speckle tracking algorithms using synthetic echocardiographic recordings. IEEE Trans Med Imaging 2016;35:191526. 
2.  Crosby J, Amundsen BH, Hergum T, Remme EW, Langeland S, Torp H. 3D speckle tracking for assessment of regional left ventricular function. Ultrasound Med Biol 2009;35:45871. 
3.  Duan Q, Angelini ED, Herz SL. Ingrassia C, Costa KD, Holmes JW, et al. Regionbased endocardium tracking on realtime threedimensional ultrasound. Ultrasound Med Biol 2009;35:25665. 
4.  Alessandrini M, Liebgott H, Barbosa D, Bernard O. Monogenic Phase Based Optical Flow Computation for Myocardial Motion Analysis in 3D Echocardiography. In: International Workshop on Statistical Atlases and Computational Models of the Heart. Berlin, Heidelberg: Springer; 2012. p. 15968. 
5.  Horn BK, Schunck BG. Determining Optical Flow. In Techniques and Applications of Image Understanding. International Society for Optics and Photonics; 1981:281. p. 31931. 
6.  Lucas BD, Kanade T. An Iterative Image Registration Technique with an Application to Stereo Vision; 1981. 
7.  Mukherjee R, Sprouse C, Pinheiro A, Abraham T, Burlina P. Computing myocardial motion in 4dimensional echocardiography. Ultrasound Med Biol 2012;38:128497. 
8.  Heyde B, Alessandrini M, Hermans J, Barbosa D, Claus P, D'hooge J. Anatomical image registration using volume conservation to assess cardiac deformation from 3D ultrasound recordings. IEEE Trans Med Imaging 2016;35:50111. 
9.  Heyde B, Bouchez S, Thieren S, Vandenheuvel M, Jasaityte R, Barbosa D, et al. Elastic image registration to quantify 3D regional myocardial deformation from volumetric ultrasound: Experimental validation in an animal model. Ultrasound Med Biol 2013;39:168897. 
10.  Zhu Y, Papademetris X, Sinusas AJ, Duncan JS. A coupled deformable model for tracking myocardial borders from realtime echocardiography using an incompressibility constraint. Med Image Anal 2010;14:42948. 
11.  Elen A, Choi HF, Loeckx D, Gao H, Claus P, Suetens P, et al. Threedimensional cardiac strain estimation using spatiotemporal elastic registration of ultrasound images: A feasibility study. IEEE Trans Med Imaging 2008;27:158091. 
12.  Queiros S, Morais P, Barbosa D, Fonseca JC, Vilaca JL, D'Hooge J. MITT: Medical Image Tracking Toolbox. IEEE Trans Med Imaging 2018;37:254757. 
13.  Queirós S, Vilaça JL, Morais P, Fonseca JC, D'hooge J, Barbosa D. Fast left ventricle tracking using localized anatomical affine optical flow. Int J Numer Method Biomed Eng 2017;33:e2871. 
14.  Jalali M, Behnam H, Davoodi F, Shojaeifard M. Temporal superresolution of 2D/3D echocardiography using cubic Bspline interpolation . Biomed Signal Process Control 2020;58:101868. 
15.  Briand T, Monasse P. Theory and practice of image Bspline interpolation. Image Process Line 2018;8:99141. 
[Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5], [Figure 6], [Figure 7], [Figure 8]
