Advertisement

CT ​Evaluation ​by ​Artificial ​Intelligence ​for ​Atherosclerosis, Stenosis and Vascular ​Morphology ​(CLARIFY): ​A ​Multi-center, international study

Open AccessPublished:June 03, 2021DOI:https://doi.org/10.1016/j.jcct.2021.05.004

      Abstract

      Background

      Atherosclerosis evaluation by coronary computed tomography angiography (CCTA) is promising for coronary artery disease (CAD) risk stratification, but time consuming and requires high expertise. Artificial Intelligence (AI) applied to CCTA for comprehensive CAD assessment may overcome these limitations. We hypothesized AI aided analysis allows for rapid, accurate evaluation of vessel morphology and stenosis.

      Methods

      This was a multi-site study of 232 patients undergoing CCTA. Studies were analyzed by FDA-cleared software service that performs AI-driven coronary artery segmentation and labeling, lumen and vessel wall determination, plaque quantification and characterization with comparison to ground truth of consensus by three L3 readers. CCTAs were analyzed for: % maximal diameter stenosis, plaque volume and composition, presence of high-risk plaque and Coronary Artery Disease Reporting & Data System (CAD-RADS) category.

      Results

      AI performance was excellent for accuracy, sensitivity, specificity, positive predictive value and negative predictive value as follows: >70% stenosis: 99.7%, 90.9%, 99.8%, 93.3%, 99.9%, respectively; >50% stenosis: 94.8%, 80.0%, 97.0, 80.0%, 97.0%, respectively. Bland-Altman plots depict agreement between expert reader and AI determined maximal diameter stenosis for per-vessel (mean difference −0.8%; 95% CI 13.8% to −15.3%) and per-patient (mean difference −2.3%; 95% CI 15.8% to −20.4%). L3 and AI agreed within one CAD-RADS category in 228/232 (98.3%) exams per-patient and 923/924 (99.9%) vessels on a per-vessel basis. There was a wide range of atherosclerosis in the coronary artery territories assessed by AI when stratified by CAD-RADS distribution.

      Conclusions

      AI-aided approach to CCTA interpretation determines coronary stenosis and CAD-RADS category in close agreement with consensus of L3 expert readers. There was a wide range of atherosclerosis identified through AI.

      Graphical abstract

      Keywords

      List of abbreviations

      CCTA
      Coronary computed tomography angiography
      FDA
      Food and Drug Administration
      AI
      Artificial Intelligence
      CAD
      Coronary Artery Disease
      CVMG
      Cardiovascular Medical Group
      MDCT
      Multidetector Computed Tomography
      LM
      Left Main
      LAD
      Left Anterior Descending
      LCx
      Left Circumflex
      RCA
      Right Coronary Artery
      L3
      Level 3
      CAD-RADS
      Coronary Artery Disease-Reporting and Data System
      CP
      Calcified Plaque
      NCP
      Noncalcified Plaque
      IVUS
      Intravascular Ultrasound

      1. Introduction

      Cardiovascular disease remains the leading cause of morbidity and mortality globally, accounting for an estimated 18 million deaths annually.
      • Virani S.S.
      • Alonso A.
      • Benjamin E.J.
      • et al.
      Heart disease and stroke statistics-2020 update: a report from the American heart association.
      Coronary computed tomography angiography (CCTA) has emerged as a front line approach for assessment of coronary artery stenosis severity, with its application encouraged for use in professional societal guidance documents and appropriate use criteria. Further, CCTA enables early identification of atherosclerosis and high risk plaque features, as well as non-invasive guidance for use of preventive therapies in a manner that improves patient outcomes.
      • Choi A.D.
      • Geske J.B.
      • Lopez-Mattei J.C.
      • et al.
      Cardiovascular imaging through the prism of modern metrics.
      • Choi A.D.
      • Feuchtner G.M.
      • Weir-McCall J.
      • Shaw L.J.
      • Min J.K.
      • Villines T.C.
      Accelerating the future of cardiac CT: social media as sine qua non?.
      • Investigators S.-H.
      • Newby D.E.
      • Adamson P.D.
      • et al.
      Coronary CT angiography and 5-year risk of myocardial infarction.
      • Michos E.D.
      • Choi A.D.
      Coronary artery disease in young adults: a hard lesson but a good teacher.
      • Knuuti J.
      • Wijns W.
      • Saraste A.
      • et al.
      2019 ESC Guidelines for the diagnosis and management of chronic coronary syndromes.
      • Choi A.D.
      • Abbara S.
      • Branch K.R.
      • et al.
      Society of Cardiovascular Computed Tomography guidance for use of cardiac computed tomography admist the COVID-19 pandemic. Endorsed by the American College of Cardiology.
      While atherosclerosis quantification and characterization by CCTA appears to be a highly effective non-invasive approach for direct visualization for assessing disease progression, stabilization and future cardiovascular events
      • Kelion A.D.
      • Nicol E.D.
      The rationale for the primacy of coronary CT angiography in the National Institute for Health and Care Excellence (NICE) guideline (CG95) for the investigation of chest pain of recent onset.
      , whole heart quantitative CCTA analysis of atherosclerosis requires high expertise and is time-intensive for manual or semi-automated evaluation.
      • Nakanishi R.
      • Motoyama S.
      • Leipsic J.
      • Budoff M.J.
      How accurate is atherosclerosis imaging by coronary computed tomography angiography?.
      Recently, advancements in data science and computational processing power now enable application of deep learning frameworks applied to CCTA images for enhanced automation.
      • Singh G.
      • Al'Aref S.J.
      • Van Assen M.
      • et al.
      Machine learning in cardiac CT: basic concepts and contemporary data.
      • van Rosendael A.R.
      • Maliakal G.
      • Kolli K.K.
      • et al.
      Maximization of the usage of coronary CTA derived plaque information using a machine learning based algorithm to improve risk stratification; insights from the CONFIRM registry.
      • Lindsell C.J.
      • Stead W.W.
      • Johnson K.B.
      Action-informed artificial intelligence-matching the algorithm to the problem.
      • Sengupta P.P.
      • Shrestha S.
      • Berthon B.
      • et al.
      Proposed requirements for cardiovascular imaging-related machine learning evaluation (PRIME): a checklist: reviewed by the American college of cardiology Healthcare innovation council.
      • Litjens G.
      • Ciompi F.
      • Wolterink J.M.
      • et al.
      State-of-the-Art deep learning in cardiovascular image analysis.
      • Dey D.
      • Slomka P.J.
      • Leeson P.
      • et al.
      Artificial intelligence in cardiovascular imaging: JACC state-of-the-art review.
      We hypothesized that AI-aided analysis would allow for rapid, accurate evaluation of vessel morphology and stenosis when compared to consensus of Level 3 (L3) expert readers. The study, ​CT EvaLuation by ARtificial Intelligence ForAtherosclerosis, Stenosis and Vascular MorphologY (CLARIFY), presents the assessment of a novel end-to-end AI aided evaluation for CCTA analysis.

      2. Methods

      Study Design. We identified patients undergoing CCTA for acute and stable chest pain at high-volume expert centers of excellence in cardiac CT to include The George Washington University Hospital Washington D.C., Centro Hospitalar Universitário de Lisboa Central, Lisbon, Portugal, and the Cardiovascular Medical Group (CVMG) and Cardiovascular Research Foundation of Southern California (CVRF), Beverly Hills, California. We excluded exams (J.P.E.) with incomplete data, significant artifacts, poor enhancement, stents or bypass grafts (Appendix A, Supplemental Figure 1). We recorded age, gender, body mass index (BMI), hypertension, hyperlipidemia, diabetes, smoking history, family history of coronary artery disease (CAD), statin, antiplatelet therapy, and use of beta-blockers. The study was approved by the GWU Institutional Review Board with a waiver of individual consent.
      CCTA Scan Acquisition. CCTA scans were performed on a 64-MDCT General Electric VCT (General Electric Healthcare, Milwaukee, Wisconsin; CVMG), and a 128-DSCT Siemens FLASH (Siemens Healthcare, Erlangen, Germany; GWU and Hospital de Santa Marta). Acquisition techniques included prospective and retrospective gating based upon institutional protocols. Iterative reconstruction was used on the DSCT scanners but not on the CVMG VCT. Patients received beta blockade, nitroglycerin and iodinated contrast in accordance with institutional and Society of Cardiovascular Computed Tomography guidelines.
      • Abbara S.
      • Blanke P.
      • Maroules C.D.
      • et al.
      SCCT guidelines for the performance and acquisition of coronary computed tomographic angiography: a report of the society of cardiovascular computed tomography guidelines committee: endorsed by the north American society for cardiovascular imaging (NASCI).
      Exams were reconstructed in 5–10% increments.
      Artificial Intelligence Segmentation and Plaque Quantification. CCTA studies were uploaded to and analyzed by FDA-cleared software Cleerly LABS (Cleerly, New York, New York).
      • Hwang T.J.
      • Kesselheim A.S.
      • Vokinger K.N.
      Lifecycle regulation of artificial intelligence- and machine learning-based software devices in medicine.
      ,
      • United States Food and Drug Administration Cleerly Labs
      510 (k) premarket notification.
      The three sites contributing cases were not used for software development or validation. This study is an investigator initiated study and Cleerly had no role in the study design or performance. Cleerly performed AI-aided CCTA analyses for the study in a blinded manner, and provided statistical services as determined and requested by study investigators.
      This is an AI-aided approach (Central Illustration) that performs automated analysis of CCTA using a series of validated convolutional neural network models (including VGG 19 network, 3D U-Net and VGG Network Variant) for image quality assessment, coronary segmentation and labeling, lumen wall evaluation and vessel contour determination and plaque characterization.
      • Singh G.
      • Al'Aref S.J.
      • Van Assen M.
      • et al.
      Machine learning in cardiac CT: basic concepts and contemporary data.
      ,
      • Carin L.
      • Pencina M.J.
      On deep learning for medical image analysis.
      A full graphical representation of the algorithm with validation details is presented in Appendix B. First, the AI-aided approach leverages 2 deep convolutional neural networks to produce a centerline along the length of the vessel, and then for lumen and outer vessel wall contouring. This approach is applied to multiple phases/series of the CCTA examination, if present, and enables phase-specific evaluation at the coronary segment vessel. The algorithm reviewed all series and determined the top 2 optimal series for further analysis including vessel and lumen segmentation, plaque, and stenosis quantification. The algorithm rank-orders all available phases for the segmentation of the arteries. It then uses the top two phases interactively on a per vessel basis, e.g., the right coronary artery (RCA) will be reconstructed from the phase which yields the highest RCA image quality, while the posterior descending artery (PDA) may come from the second phase if the PDA has a higher image quality on that phase. Once coronary artery segmentation is performed, an automated labeling is done to classify arteries by their location as well the proximal, mid and distal portions within a single vessel. The AI further allows for defining of coronary artery lesions (i.e., those areas where plaque is present). Utilizing a normal proximal reference vessel cross-sectional slide, the start and the end of the lesion, and the cross-sectional slice that demonstrates the greatest absolute narrowing, % diameter stenosis severity is automatically calculated. The software determines the start and end of lesions and drops stenosis markers at the region of the highest stenosis. Within coronary artery lesions, plaque is quantified in a similar fashion, and further characterized as low-attenuation non-calcified plaque, non-calcified plaque and calcified plaque based upon Hounsfield unit (HU) densities of <30, −189 to 350, >350, respectively. Positive arterial remodeling was identified as a remodeling index ≥1.10 by diameter when compared to a proximal vessel reference. We used a coronary artery territory-based analysis which included the left main (LM), left anterior descending (LAD) including diagonals and ramus intermedius, left circumflex (LCx) including obtuse marginals and left-posterior descending and posterolateral branches, and RCA including right posterolateral and PDA. For each territory we recorded vessel length, vessel volume, lumen volume, total plaque volume, calcified plaque volume, noncalcified plaque volume, low density noncalcified plaque volume, maximum diameter and area stenosis, and maximum remodeling index. After the AI algorithm has finished all operations, as mandated by the FDA, a quality control cardiac CT trained technician reviews the results of the AI analysis in all cases with manual adjustment if necessary. The QA process included visual inspection of the lumen and vessel boundaries on the straightened multiplanar reformat views of all vessels 1.5 ​mm and larger, as well as every cross-section of each of these vessels, placed at contiguous 0.25 ​mm increments.
      The time from CT data upload until AI processing completion was recorded. We also recorded time for technician or physician quality assurance review and generation of a final report. Finally, CAD-RADS category was scored based upon maximum percent stenosis in accordance with the guidance document.
      • Cury R.C.
      • Abbara S.
      • Achenbach S.
      • et al.
      CAD-RADSTM coronary artery disease - reporting and data System. An expert consensus document of the society of cardiovascular computed tomography (SCCT), the American college of radiology (ACR) and the north American society for cardiovascular imaging (NASCI). Endorsed by the American college of cardiology.
      When there was 0% diameter stenosis but plaque was present, CAD-RADS category was recorded as 1 per CAD-RADS guidelines.
      Level 3 Expert Reads. Three advanced imaging attending physicians who were Level 3 (L3) readers, ranging from 7 to 17 years’ experience, performed blinded assessment of CCTA. A consensus of their individual reads represented ground truth for the study. Each reader read each case independently and in distinct reading sessions. The readers interpreted the original dataset and chose phases independent of the AI image segmentation. After independent reading sessions, the disagreements were analyzed. The consensus percent diameter stenosis was the mathematical average of the three independent reader determined stenoses. For discrete variables, such as plaque volume which was scored from 0 to 5, if two of the three readers agreed then that value was used. If all three readers disagreed, then the readers came together to reanalyze and adjudicate the differences.
      Readers recorded the following on a territory basis: CAD-RADS category and maximum diameter stenosis. In accordance with SCCT CAD-RADS and SCCT reporting guidelines, L3 readers stenosis was assessed by qualitative visual assessment and recorded in 10% increments.
      • Leipsic J.
      • Abbara S.
      • Achenbach S.
      • et al.
      SCCT guidelines for the interpretation and reporting of coronary CT angiography: a report of the Society of Cardiovascular Computed Tomography Guidelines Committee.
      Readers recorded presence or absence of high risk plaque features including positive remodeling and low attenuation plaque.
      Statistical Analysis. The consistency of the L3 categorization of CAD-RADS and AI-enabled CAD-RADS was assessed by evaluating correlation and numeric agreement. This was done for readers individually versus AI; as well as L3 consensus read versus AI, for scores generated per patient and per vessel. Spearman's correlation coefficient was used to assess correlation, weighted kappa statistic measured agreement.
      L3 readers determined maximum diameter stenosis was compared with AI stenosis on a per-patient and per-vessel basis. Correlation and numeric agreement were assessed. The Pearson correlation coefficient was used to evaluate correlation, linear regression plots were generated for visualization of the relationship. Bland-Altman plots with limits of agreement was performed. Diagnostic performance of AI vs L3 was assessed through diagnostic accuracy, sensitivity, specificity, positive and negative predictive values at both >50% and >70% stenosis thresholds on per vessel and per patient basis.
      Readers determined presence of two high risk plaque features—low attenuation plaque <30 Hounsfield units (HU) and positive arterial remodeling with a remodeling index ≥1.10 by diameter—with this analysis compared with AI on per vessel and per patient basis. This binary outcome was compared by calculating the percent agreement and kappa statistic.

      3. Results

      Demographics and Analysis Time. Demographic and AI analysis time data is depicted in Table 1. The study population consisted of n ​= ​232 patients, mean age 60 ​± ​12 years, 37% female, 61% with hypertension, 69% with hyperlipidemia and 38% smokers. The AI analysis time was rapid at 9.7 ​± ​3.2 ​min. AI analysis plus quality assurance analysis and report generation was 23.7 ​± ​6.4 ​min (Table 1).
      Table 1Demographics and AI analysis data.
      Demographics and AI Analysis Data (n ​= ​232)
      VariableN (%)
      Age ​± ​SD60 ​± ​12 years
      Female Sex, Mean (%)86 (37)
      Body Mass Index (BMI) ​± ​SD27.5 ​± ​6 ​kg/m2
      Hypertension, n (%)142 (61)
      Hyperlipidemia, n (%)161 (69)
      Diabetes Mellitus, n (%)67 (29)
      Smoking, n (%)88 (38)
      Family History of Coronary Artery Disease, n (%)116 (35)
      Statin therapy, n (%)159 (68)
      Antiplatelet therapy, n (%)84 (36)
      Beta-Blocker therapy, n (%)58 (25)
      Coronary artery calcium score, mean ​± ​SD
      Coronary artery calcium score by the Agatston method was available for 147 of the 232 (63%) of patients as one of the sites does not routinely perform non-contrast calcium scoring prior to CCTA.
      150 ​± ​495

      (Range 0–3607)
      AI Analysis Data
      AI Analysis Series available, mean ​± ​SD minutes3.6 ​± ​1.6

      (Range 1–10)
      AI Analysis Time, mean ​± ​SD minutes9.7 ​± ​3.2
      AI Analysis ​+ ​QA analysis and report generation, mean ​± ​SD minutes23.7 ​± ​6.4
      a Coronary artery calcium score by the Agatston method was available for 147 of the 232 (63%) of patients as one of the sites does not routinely perform non-contrast calcium scoring prior to CCTA.
      CAD-RADS Categorization. Fig. 1 depicts consensus reads versus AI results. Overall, 182/232 (78.0%) had CAD-RADS categorical agreement, 228/232(98.3%) agreed within one category. The most frequent disagreement occurred with expert consensus CAD-RADS 0 and AI CAD-RADS 1 (n ​= ​29 12.5% per patient, n ​= ​161 17.4% per vessel). To further evaluate L3 consensus vs AI for a collated mild-moderate versus severe stenosis categories, at a threshold for potential interventional treatment (>70% stenosis), we evaluated CAD-RADS 0–3 and CAD-RADS 4–5 to assess accuracy and found only 1 case of discrepancy on either a per-vessel or per-patient basis respectively with >99% category agreement for these thresholds (Appendix A, Supplement Figure 2). Example case of discordance is shown in Appendix A, Supplement Figure 3. There were 33 discordant cases with L3 CAD-RADS 0, AI detected non-negligible plaque without stenosis in 26/33 (78.7%) and nonsignificant stenosis in 7/33 (21.2%) (mean 2.6% ​± ​8%, range 0–31%).
      Fig. 1
      Fig. 1Contingency Tables of Level 3 Reader versus AI CAD-RADS scores. On a per vessel and per patient basis, L3 and AI depicted 98.3% agreement within 1 CAD-RADS category per patient and 99.9% per vessel. The most common misclassification was expert consensus CAD-RADS 0 and AI CAD-RADS 1 (n ​= ​31 10.0% per patient, n ​= ​161 17.4% per vessel). The weighted kappa is 0.72 per vessel and 0.82 per patient.
      The weighted kappa coefficient between readers and AI was higher for consensus (0.812) than individual readers (0.781, 0.606, and 0.729) per patient and per vessel (consensus 0.723; readers 0.689, 0.566 and 0.670). There was high degree of agreement between readers and AI, weighted kappa value was greater for consensus reads (0.812) than individual L3 readers (0.781, 0.606, and 0.728) per patient and per vessel (consensus 0.723; readers 0.689, 0.566 and 0.670). Discordance was primarily seen for minimal and mild (<25%) diameter stenoses.
      Maximal Diameter Stenosis. Diagnostic performance of AI compared to L3 consensus was excellent (Table 2). Example case is shown in Fig. 2. A non-zero stenosis was depicted in 182/232 (78.8%) studies by readers and 154/232 (66.7%) by AI. In 32 cases of AI “no stenosis,” L3 consensus reads depicted stenoses (mean 5.0 ​± ​4.4%, range 2–18%); in 4 cases of consensus reader “zero stenosis,” AI depicted stenoses (mean 9.8 ​± ​6.7%, range 1–20%). In 86% of cases, AI quantified stenosis severity was within 10% of L3 consensus.
      Table 2Diagnostic performance of Artificial Intelligence vs Level 3 Expert Consensus. The diagnostic performance of the artificial intelligence reads was calculated using Level 3 expert consensus reads as a gold standard, generating high diagnostic accuracy, sensitivity, specificity, positive and negative predictive values at both >50% and >70% stenosis thresholds on both a per vessel and per patient basis.
      Diagnostic performance
      ThresholdBasisReader% AgreeSensitivitySpecificityPPVNPV
      >50%per vessel197.6%83.3%98.1%59.5%99.4%
      296.2%61.1%97.8%52.4%98.4%
      395.8%55.8%98.5%69.0%97.4%
      Consensus97.5%77.1%98.3%64.3%99.1%
      per patient194.4%84.0%95.7%70.0%98.0%
      291.4%67.9%94.6%63.3%95.5%
      391.4%61.9%97.9%86.7%92.1%
      Consensus94.8%80.0%97.0%80.0%97.0%
      >70%per vessel199.7%90.9%99.8%83.3%99.9%
      299.2%71.4%99.8%83.3%99.6%
      398.8%62.5%99.8%83.3%99.3%
      Consensus99.7%90.9%99.8%83.3%99.9%
      per patient199.1%88.9%99.6%88.9%99.6%
      297.8%66.7%99.5%88.9%98.2%
      397.4%61.5%99.5%88.9%97.8%
      Consensus99.1%88.9%99.6%88.9%99.6%
      Fig. 2
      Fig. 2Example Case of Consensus Between Artificial Intelligence and Level 3. Example of a study depicting excellent agreement between maximal percent diameter stenosis in a 53-year-old make with exertional chest pain. On the left readers using a curved multiplanar reformat (cMPR) of the RCA determined a consensus stenosis of 26%, the straightened MPR to its right with colored plaque overlay (in blue and yellow) generated by AI found 35% maximal stenosis. In the middle the cMPR used by readers determined 55% stenosis of the left main and 53% of the LAD, AI to its right with colored plaque overlay (in blue and yellow) by AI found 53 and 47% respectively. On the right the cMPR used by readers determined 24% stenosis of the left circumflex, AI with colored plaque overlay (in blue and yellow) by AI determined 34%. (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)
      There was high agreement between L3 consensus and AI percent maximal stenosis (Fig. 3). The Intraclass correlation coefficient between readers and AI was higher for consensus L3 reads (0.91) than individual L3 readers (0.90, 0.80, and 0.85) per vessel and per patient (consensus 0.93, readers 0.92, 0.83 and 0.87). Linear regression depicted a close relationship between reader and AI stenosis (Fig. 3). Bland-Altman plots depicted close mean differences between L3 consensus and AI with mean difference of (−) 0.8% per-vessel and (−) 2.3% per-patient (Fig. 3).
      Fig. 3
      Fig. 3Linear regression and Bland-Altman of Level 3 Consensus vs Artificial Intelligence. Linear regression plots depicted a close relationship between reader and AI diameter stenosis on both a per patient and per vessel basis. Bland-Altman plots depict very good numeric agreement between expert reader and AI determined maximal diameter stenosis for per-vessel (mean difference −0.8%; 95% CI 13.8% to −15.3%) and per-patient (mean difference −2.3%; 95% CI 15.8% to −20.4%). The intraclass correlation is 0.91 for per-vessel and 0.93 for per-patient evaluation.
      Atherosclerotic Plaque Volume and Characterization. Non-negligible plaque (>3 ​mm3) was detected by AI in 170/232 (73.2%) RCA, 154/228 (67.5%) LM, 196/232 (84.5%) LAD, and 150/232 (64.7%) LCx. There was a wide range of plaque when stratified by CAD-RADS distribution, including 16% that were CAD-RADS 3 or higher (Appendix A, Supplemental Figure 4). Full evaluation of atherosclerotic plaque volume and plaque composition are provided in Appendix A, Supplemental Table 1.
      Presence of High Risk Plaque Features. High risk plaque features were found in 49/232 (21.1%) patients using AI and in 31/232 (13.4%) by consensus expert reads which had an 82% agreement. The kappa statistic was highest for consensus reads (0.372) than it was for individual readers (0.363, 0.364, and 0.313), although the correlation remained fair.

      4. Discussion

      This multicenter study represents the first external validation of a novel, FDA-cleared AI-aided approach for evaluation of CCTA vessel morphology, stenosis in comparison to ground truth of consensus of L3 expert readers. AI demonstrated high correlation to L3 consensus for determination of CCTA maximum % stenosis severity and CADRADS score. The AI determined percent diameter stenosis had stronger correlation to L3 consensus than to any individual reader. AI rapidly performed whole heart atherosclerosis plaque volume quantification to include full automated vessel segmentation and labeling, with identification of plaque more often than L3 readers. Further, AI detected and quantified high risk plaque characteristics including low attenuation plaque and high risk plaque characteristics more often than L3 readers. The AI analysis was performed rapidly, with a mean time of under 10 ​min that totaled under 24 ​min after including FDA mandated quality analysis and reporting time.
      AI-enabled CCTA to improve interpretation and reduce downstream testing. Multiple studies—including the recent SCOT-Heart Trial 5-year analysis—have demonstrated that a CCTA guided approach to stable chest pain reduces downstream testing and significantly lowers cardiovascular events.
      • Investigators S.-H.
      • Newby D.E.
      • Adamson P.D.
      • et al.
      Coronary CT angiography and 5-year risk of myocardial infarction.
      ,
      • McKavanagh P.
      • Lusk L.
      • Ball P.A.
      • et al.
      A comparison of cardiac computerized tomography and exercise stress electrocardiogram test for the investigation of stable chest pain: the clinical results of the CAPP randomized prospective trial.
      ,
      • Min J.K.
      • Newby D.E.
      Coronary computed tomography angiography as the investigation of choice for stable chest pain.
      Yet, heterogeneity in real-world interpretation of CCTA may lead to overestimation of coronary stenosis and increased unnecessary testing, thus mitigating the potential benefits of a CCTA-guided approach. Analysis of the PROMISE trial revealed that core laboratory interpretation classified 41% fewer patients with significant CAD when compared to site interpretation – an important issue when considering that 42% of CCTA at local sites were read by less experienced level II readers.
      • Lu M.T.
      • Meyersohn N.M.
      • Mayrhofer T.
      • et al.
      Central core laboratory versus site interpretation of coronary CT angiography: agreement and association with cardiovascular events in the PROMISE trial.
      Use of this novel rapid AI-enabled approach may aid in significantly improving the use and performance of CCTA by local clinical readers to better implement international guidelines.
      • Knuuti J.
      • Wijns W.
      • Saraste A.
      • et al.
      2019 ESC Guidelines for the diagnosis and management of chronic coronary syndromes.
      ,
      • Truong Q.A.
      • Rinehart S.
      • Abbara S.
      • et al.
      Coronary computed tomographic imaging in women: an expert consensus statement from the Society of Cardiovascular Computed Tomography.
      CAD-RADS Assessment by AI. Using CAD-RADS as a framework for AI to compare to current clinical standard of care. The 98% agreement within 1 CAD-RADS category on a per-patient basis and 99.9% on a per-vessel basis with a mean difference of about 0.6–2.2% offers confidence that this approach provides a reasonable estimation of stenosis severity through the current clinical CAD-RADS standard. Our further analysis on a maximum stenosis basis allows for a threshold utilized in clinical practice that belies decision making for invasive angiography and potential revascularization decisions. A negative predictive value for >70% stenosis threshold of 99% offers confidence that the AI approach does not miss any major obstructive disease.
      Role of atherosclerosis plaque quantification by CCTA. Beyond stenosis, an advanced understanding of atherosclerosis has evolved from predicting myocardial infarction via individual high-risk plaques to a contemporary understanding of the importance of quantitative plaque burden and type. The Providing Regional Observations to Study Predictors of Events in the Coronary Tree (PROSPECT) trial employed intravascular ultrasound (IVUS) to identify plaque components including dense calcium, necrotic core, fibrofatty tissue or fibrous tissue; and found that these contributed to the 20% of patients with recurrent major adverse cardiovascular events.
      • Stone G.W.
      • Maehara A.
      • Lansky A.J.
      • et al.
      A prospective natural-history study of coronary atherosclerosis.
      However, use of IVUS was associated with serious adverse events including coronary dissection and perforation.
      In contrast, noninvasive imaging through CCTA allows for comprehensive evaluation of plaque components and composition, but evaluation is time-consuming and requires high expertise limiting its application in clinical practice.
      • Fischer C.
      • Hulten E.
      • Belur P.
      • Smith R.
      • Voros S.
      • Villines T.C.
      Coronary CT angiography versus intravascular ultrasound for estimation of coronary stenosis and atherosclerotic plaque burden: a meta-analysis.
      • van Rosendael A.R.
      • Al'Aref S.J.
      • Dwivedi A.
      • et al.
      Quantitative evaluation of high-risk coronary plaque by coronary CTA and subsequent acute coronary events.
      • Ferencik M.
      • Mayrhofer T.
      • Puchner S.B.
      • et al.
      Computed tomography-based high-risk coronary plaque score to predict acute coronary syndrome among patients with acute chest pain--Results from the ROMICAT II trial.
      • Williams M.C.
      • Moss A.J.
      • Dweck M.
      • et al.
      Coronary artery plaque characteristics associated with adverse outcomes in the SCOT-heart study.
      • Lee S.E.
      • Villines T.C.
      • Chang H.J.
      Should CT replace IVUS for evaluation of CAD in large-scale clinical trials: effects of medical therapy on atherosclerotic plaque.
      • Choi A.D.
      • Parwani P.
      • Michos E.D.
      • et al.
      The global social media response to the 14th annual Society of Cardiovascular Computed Tomography scientific sessions.
      In the case:control Incident Coronary Syndromes Identified by Computed Tomography (ICONIC) study, semi-automated plaque quantitative analysis (Medis Medical Imaging, Leiden, Netherlands) to identify constituents of atherosclerosis was performed to identify those that are associated with future acute coronary syndrome. In this study, future ACS was associated with specific plaque characteristics, most significantly the burden of low attenuation plaque.
      • Chang H.J.
      • Lin F.Y.
      • Lee S.E.
      • et al.
      Coronary atherosclerotic precursors of acute coronary syndromes.
      Progression of atherosclerosis plaque determined by CTA (PARADIGM) evaluated the long-term effects of statin therapy on atherosclerosis, and employed CCTA for quantifying disease progression rates of fibrous and fibro-fatty percent atheroma volume. Importantly, rate of progression of non-calcified plaque (fibrous, fibro-fatty and necrotic care) was found to be slower in statin taking than statin-naïve patients.
      • Lee S.E.
      • Chang H.J.
      • Sung J.M.
      • et al.
      Effects of statins on coronary atherosclerotic plaques: the PARADIGM study.
      • Lee S.E.
      • Sung J.M.
      • Andreini D.
      • et al.
      Differences in progression to obstructive lesions per high-risk plaque features and plaque volumes with CCTA.
      • Won K.B.
      • Lee S.E.
      • Lee B.K.
      • et al.
      Longitudinal assessment of coronary plaque volume change related to glycemic status using serial coronary computed tomography angiography: a PARADIGM (Progression of AtheRosclerotic PlAque DetermIned by Computed TomoGraphic Angiography Imaging) substudy.
      Analysis of the SCOT-Heart trial used semiautomatic software to quantify stenosis and plaque and found that low attenuation plaque burden was the strongest predictor of fatal or nonfatal myocardial infarction irrespective of risk score, Agatston score or obstructive coronary disease.
      • Investigators S.-H.
      • Newby D.E.
      • Adamson P.D.
      • et al.
      Coronary CT angiography and 5-year risk of myocardial infarction.
      ,
      • Williams M.C.
      • Kwiecinski J.
      • Doris M.
      • et al.
      Low-attenuation noncalcified plaque on coronary computed tomography angiography predicts myocardial infarction: results from the multicenter SCOT-heart trial (scottish computed tomography of the HEART).
      Machine Learning Applied to CCTA. Recent studies have begun to evaluate individual aspects of AI-enabled plaque quantification, though few of these solutions are currently available clinically. Zreik et al. used a multitask recurrent convolutional neural network in n ​= ​166 for automatic characterization of plaque.
      • Zreik M.
      • Lessmann N.
      • van Hamersvelt R.W.
      • et al.
      Deep learning analysis of the myocardium in coronary CT angiography for identification of patients with functionally significant coronary artery stenosis.
      They achieved a linearly weighted kappa of 0.68, 0.66 and 0.67 ​at the segment, artery and patient-level for the binary presence or absence of non-calcified, mixed and calcified plaque. Kang et al. used a support vector machine (SVM) learning algorithms in a small number of datasets (n ​= ​42) for detection of plaque for lesions a simplistic severity of ≥25% in comparison to experienced expert consensus readers and reported accuracy of 94%.
      • Kang D.
      • Dey D.
      • Slomka P.J.
      • et al.
      Structured learning algorithm for detection of nonobstructive and obstructive coronary plaque lesions from computed tomography angiography.
      This present study takes this previous work and extends it through a novel end-to-end solution that automates plaque analysis and provides rapid graphical output of plaque components. Plaque quantification is challenging owing to the need for selection of the best imaging phase, determining the appropriate center line segmentation, which may take thousands of data points, understanding vessel morphology and contouring, setting thresholds for appropriate plaque components, differentiating coronary from non-coronary structures and providing direct quantification of both stenosis and volume of plaque components. Standards for plaque quantification are currently under development by medical societies.
      Atherosclerosis Assessment by AI. Beyond stenosis evaluation, the most common areas of disagreement between L3 and AI was for atherosclerosis identified in the CAD-RADS 1 range. As shown by an example case of discordance in Supplemental Figure 3, AI depicted 82 ​mm3 of atherosclerotic plaque that was not identified by the L3 expert consensus. While large studies of prognosis have demonstrated the important of non-obstructive atherosclerosis, the prognostic significance of this AI quantified plaque remains an area of further study.
      • van Rosendael A.R.
      • Bax A.M.
      • Smit J.M.
      • et al.
      Clinical risk factors and atherosclerotic plaque extent to define risk for major events in patients without obstructive coronary artery disease: the long-term coronary computed tomography angiography CONFIRM registry.
      The degree of atherosclerotic plaque quantified by AI in the present cohort is consistent with prior analysis. An AI approach may provide a means of providing a reliable means of standardization for plaque quantification while providing superior risk stratification when compared to current models, however the studied AI-aided approach in the current study requires further validation to an invasive ultrasound gold standard.
      • Betancur J.
      • Otaki Y.
      • Motwani M.
      • et al.
      Prognostic value of combined clinical and myocardial perfusion imaging data using machine learning.
      ,
      • Motwani M.
      • Dey D.
      • Berman D.S.
      • et al.
      Machine learning for prediction of all-cause mortality in patients with suspected coronary artery disease: a 5-year multicentre prospective registry analysis.
      Limitations. Ground truth in this present study was the consensus of 3 L3 readers without validation to invasive approaches (eg IVUS or optical coherence tomography). Only 15% of studied population of consecutive chest pain patients had anatomically obstructive stenosis. Ongoing study will evaluate AI stenosis to patients referred to invasive angiography. A guideline based reference standard for CCTA atherosclerosis quantification was not available to include in this study and is currently under development by medical societies. In addition, an ongoing multicenter study is examining the diagnostic performance of this AI-enabled approach to quantitative coronary angiography, IVUS and optical coherence tomography; examination of those results will further establish the role of AI in CCTA imaging. AI was not performed in CCTA studies of poor image quality deemed uninterpretable by L3 readers. The prognostic significance of atherosclerotic plaque quantified by AI is unknown. Established HU thresholds for plaque characterization were utilized without adjustment, in the absence of a standardized methodology, for high luminal contrast enhancement.

      5. Conclusions

      This study presents a novel, AI-enabled, end-to-end analysis tool for CCTA interpretation (Central Illustration). This approach accurately and rapidly quantifies stenosis and CAD-RADS category when compared to L3 expert consensus. The AI approach identified a wide range of atherosclerosis plaque volume and plaque composition in all coronary arteries and their branches. Use of this FDA-cleared device as a clinical decision support tool in combination with enhanced CCTA education may improve the reproducibility of CCTA interpretation in various clinical and investigational settings. The results of this study provide an important foundational platform for future research in AI-guided atherosclerosis evaluation across a wide spectrum of disease and patients.

      Funding

      This work was supported by The GW Heart and Vascular Institute (ADC).

      Declaration of competing interest

      Equity, Cleerly, Inc (ADC, HM, JPE). All other authors report no disclosures.

      Acknowledgements

      None.

      Appendix A. Supplementary data

      The following are the supplementary data to this article:

      References

        • Virani S.S.
        • Alonso A.
        • Benjamin E.J.
        • et al.
        Heart disease and stroke statistics-2020 update: a report from the American heart association.
        Circulation. 2020; 141: e139-e596
        • Choi A.D.
        • Geske J.B.
        • Lopez-Mattei J.C.
        • et al.
        Cardiovascular imaging through the prism of modern metrics.
        JACC Cardiovasc Imag. 2020; 13: 1256-1269
        • Choi A.D.
        • Feuchtner G.M.
        • Weir-McCall J.
        • Shaw L.J.
        • Min J.K.
        • Villines T.C.
        Accelerating the future of cardiac CT: social media as sine qua non?.
        J Cardiovasc Comput Tomogr. 2020; 14: 382-385
        • Investigators S.-H.
        • Newby D.E.
        • Adamson P.D.
        • et al.
        Coronary CT angiography and 5-year risk of myocardial infarction.
        N Engl J Med. 2018; 379: 924-933
        • Michos E.D.
        • Choi A.D.
        Coronary artery disease in young adults: a hard lesson but a good teacher.
        J Am Coll Cardiol. 2019; 74: 1879-1882
        • Knuuti J.
        • Wijns W.
        • Saraste A.
        • et al.
        2019 ESC Guidelines for the diagnosis and management of chronic coronary syndromes.
        Eur Heart J. 2020; 41: 407-477
        • Choi A.D.
        • Abbara S.
        • Branch K.R.
        • et al.
        Society of Cardiovascular Computed Tomography guidance for use of cardiac computed tomography admist the COVID-19 pandemic. Endorsed by the American College of Cardiology.
        Journal of Cardiovascular Computed Tomography. 2020; 14: 101-104
        • Kelion A.D.
        • Nicol E.D.
        The rationale for the primacy of coronary CT angiography in the National Institute for Health and Care Excellence (NICE) guideline (CG95) for the investigation of chest pain of recent onset.
        J Cardiovasc Comput Tomogr. 2018; 12: 516-522
        • Nakanishi R.
        • Motoyama S.
        • Leipsic J.
        • Budoff M.J.
        How accurate is atherosclerosis imaging by coronary computed tomography angiography?.
        J Cardiovasc Comput Tomogr. 2019; 13: 254-260
        • Singh G.
        • Al'Aref S.J.
        • Van Assen M.
        • et al.
        Machine learning in cardiac CT: basic concepts and contemporary data.
        J Cardiovasc Comput Tomogr. 2018; 12: 192-201
        • van Rosendael A.R.
        • Maliakal G.
        • Kolli K.K.
        • et al.
        Maximization of the usage of coronary CTA derived plaque information using a machine learning based algorithm to improve risk stratification; insights from the CONFIRM registry.
        J Cardiovasc Comput Tomogr. 2018; 12: 204-209
        • Lindsell C.J.
        • Stead W.W.
        • Johnson K.B.
        Action-informed artificial intelligence-matching the algorithm to the problem.
        J Am Med Assoc. 2020; 323: 2141-2142https://doi.org/10.1001/jama.2020.5035
        • Sengupta P.P.
        • Shrestha S.
        • Berthon B.
        • et al.
        Proposed requirements for cardiovascular imaging-related machine learning evaluation (PRIME): a checklist: reviewed by the American college of cardiology Healthcare innovation council.
        JACC Cardiovasc Imag. 2020; 13: 2017-2035
        • Litjens G.
        • Ciompi F.
        • Wolterink J.M.
        • et al.
        State-of-the-Art deep learning in cardiovascular image analysis.
        JACC Cardiovasc Imag. 2019; 12: 1549-1565
        • Dey D.
        • Slomka P.J.
        • Leeson P.
        • et al.
        Artificial intelligence in cardiovascular imaging: JACC state-of-the-art review.
        J Am Coll Cardiol. 2019; 73: 1317-1335
        • Abbara S.
        • Blanke P.
        • Maroules C.D.
        • et al.
        SCCT guidelines for the performance and acquisition of coronary computed tomographic angiography: a report of the society of cardiovascular computed tomography guidelines committee: endorsed by the north American society for cardiovascular imaging (NASCI).
        J Cardiovasc Comput Tomogr. 2016; 10: 435-449
        • Hwang T.J.
        • Kesselheim A.S.
        • Vokinger K.N.
        Lifecycle regulation of artificial intelligence- and machine learning-based software devices in medicine.
        J Am Med Assoc. 2019; 322: 2285-2286https://doi.org/10.1001/jama.2019.16842
        • United States Food and Drug Administration Cleerly Labs
        510 (k) premarket notification.
        • Carin L.
        • Pencina M.J.
        On deep learning for medical image analysis.
        J Am Med Assoc. 2018; 320: 1192-1193
        • Cury R.C.
        • Abbara S.
        • Achenbach S.
        • et al.
        CAD-RADSTM coronary artery disease - reporting and data System. An expert consensus document of the society of cardiovascular computed tomography (SCCT), the American college of radiology (ACR) and the north American society for cardiovascular imaging (NASCI). Endorsed by the American college of cardiology.
        J Cardiovasc Comput Tomogr. 2016; 10 (Epub 2016 Jun 15): 269-281https://doi.org/10.1016/j.jcct.2016.04.005
        • Leipsic J.
        • Abbara S.
        • Achenbach S.
        • et al.
        SCCT guidelines for the interpretation and reporting of coronary CT angiography: a report of the Society of Cardiovascular Computed Tomography Guidelines Committee.
        J Cardiovasc Comput Tomogr. 2014; 8: 342-358
        • McKavanagh P.
        • Lusk L.
        • Ball P.A.
        • et al.
        A comparison of cardiac computerized tomography and exercise stress electrocardiogram test for the investigation of stable chest pain: the clinical results of the CAPP randomized prospective trial.
        Eur Heart J Cardiovasc Imag. 2015; 16: 441-448
        • Min J.K.
        • Newby D.E.
        Coronary computed tomography angiography as the investigation of choice for stable chest pain.
        JAMA Cardiol. 2019; 4: 948
        • Lu M.T.
        • Meyersohn N.M.
        • Mayrhofer T.
        • et al.
        Central core laboratory versus site interpretation of coronary CT angiography: agreement and association with cardiovascular events in the PROMISE trial.
        Radiology. 2018; 287: 87-95
        • Truong Q.A.
        • Rinehart S.
        • Abbara S.
        • et al.
        Coronary computed tomographic imaging in women: an expert consensus statement from the Society of Cardiovascular Computed Tomography.
        J Cardiovasc Comput Tomogr. 2018; 12: 451-466
        • Stone G.W.
        • Maehara A.
        • Lansky A.J.
        • et al.
        A prospective natural-history study of coronary atherosclerosis.
        N Engl J Med. 2011; 364: 226-235
        • Fischer C.
        • Hulten E.
        • Belur P.
        • Smith R.
        • Voros S.
        • Villines T.C.
        Coronary CT angiography versus intravascular ultrasound for estimation of coronary stenosis and atherosclerotic plaque burden: a meta-analysis.
        J Cardiovasc Comput Tomogr. 2013; 7: 256-266
        • van Rosendael A.R.
        • Al'Aref S.J.
        • Dwivedi A.
        • et al.
        Quantitative evaluation of high-risk coronary plaque by coronary CTA and subsequent acute coronary events.
        JACC Cardiovasc Imag. 2019; 12: 1568-1571
        • Ferencik M.
        • Mayrhofer T.
        • Puchner S.B.
        • et al.
        Computed tomography-based high-risk coronary plaque score to predict acute coronary syndrome among patients with acute chest pain--Results from the ROMICAT II trial.
        J Cardiovasc Comput Tomogr. 2015; 9: 538-545
        • Williams M.C.
        • Moss A.J.
        • Dweck M.
        • et al.
        Coronary artery plaque characteristics associated with adverse outcomes in the SCOT-heart study.
        J Am Coll Cardiol. 2019; 73: 291-301
        • Lee S.E.
        • Villines T.C.
        • Chang H.J.
        Should CT replace IVUS for evaluation of CAD in large-scale clinical trials: effects of medical therapy on atherosclerotic plaque.
        J Cardiovasc Comput Tomogr. 2019; 13: 248-253
        • Choi A.D.
        • Parwani P.
        • Michos E.D.
        • et al.
        The global social media response to the 14th annual Society of Cardiovascular Computed Tomography scientific sessions.
        J Cardiovasc Comput Tomogr. 2020; 14: 124-130
        • Chang H.J.
        • Lin F.Y.
        • Lee S.E.
        • et al.
        Coronary atherosclerotic precursors of acute coronary syndromes.
        J Am Coll Cardiol. 2018; 71: 2511-2522
        • Lee S.E.
        • Chang H.J.
        • Sung J.M.
        • et al.
        Effects of statins on coronary atherosclerotic plaques: the PARADIGM study.
        JACC Cardiovasc Imag. 2018; 11: 1475-1484
        • Lee S.E.
        • Sung J.M.
        • Andreini D.
        • et al.
        Differences in progression to obstructive lesions per high-risk plaque features and plaque volumes with CCTA.
        JACC Cardiovasc Imag. 2020; 13: 1409-1417
        • Won K.B.
        • Lee S.E.
        • Lee B.K.
        • et al.
        Longitudinal assessment of coronary plaque volume change related to glycemic status using serial coronary computed tomography angiography: a PARADIGM (Progression of AtheRosclerotic PlAque DetermIned by Computed TomoGraphic Angiography Imaging) substudy.
        J Cardiovasc Comput Tomogr. 2019; 13: 142-147
        • Williams M.C.
        • Kwiecinski J.
        • Doris M.
        • et al.
        Low-attenuation noncalcified plaque on coronary computed tomography angiography predicts myocardial infarction: results from the multicenter SCOT-heart trial (scottish computed tomography of the HEART).
        Circulation. 2020; 141: 1452-1462
        • Zreik M.
        • Lessmann N.
        • van Hamersvelt R.W.
        • et al.
        Deep learning analysis of the myocardium in coronary CT angiography for identification of patients with functionally significant coronary artery stenosis.
        Med Image Anal. 2018; 44: 72-85
        • Kang D.
        • Dey D.
        • Slomka P.J.
        • et al.
        Structured learning algorithm for detection of nonobstructive and obstructive coronary plaque lesions from computed tomography angiography.
        J Med Imaging. 2015; 2014003
        • van Rosendael A.R.
        • Bax A.M.
        • Smit J.M.
        • et al.
        Clinical risk factors and atherosclerotic plaque extent to define risk for major events in patients without obstructive coronary artery disease: the long-term coronary computed tomography angiography CONFIRM registry.
        Eur Heart J Cardiovasc Imag. 2020; 21: 479-488
        • Betancur J.
        • Otaki Y.
        • Motwani M.
        • et al.
        Prognostic value of combined clinical and myocardial perfusion imaging data using machine learning.
        JACC Cardiovasc Imag. 2018; 11: 1000-1009
        • Motwani M.
        • Dey D.
        • Berman D.S.
        • et al.
        Machine learning for prediction of all-cause mortality in patients with suspected coronary artery disease: a 5-year multicentre prospective registry analysis.
        Eur Heart J. 2017; 38: 500-507