Set-theory based benchmarking of three different variant callers for targeted sequencing

Abstract

Background: Next generation sequencing (NGS) technologies have improved the study of hereditary diseases. Since the evaluation of bioinformatics pipelines is not straightforward, NGS demands efective strategies to analyze data that is of paramount relevance for decision making under a clinical scenario. According to the benchmark‑ ing framework of the Global Alliance for Genomics and Health (GA4GH), we imple‑ mented a new simple and user-friendly set-theory based method to assess variant call‑ ers using a gold standard variant set and high confdence regions. As model, we used TruSight Cardio kit sequencing data of the reference genome NA12878. This targeted sequencing kit is used to identify variants in key genes related to Inherited Cardiac Conditions (ICCs), a group of cardiovascular diseases with high rates of morbidity and mortality. Results: We implemented and compared three variant calling pipelines (Isaac, Freebayes, and VarScan). Performance metrics using our set-theory approach showed high-resolution pipelines and revealed: (1) a perfect recall of 1.000 for all three pipe‑ lines, (2) very high precision values, i.e. 0.987 for Freebayes, 0.928 for VarScan, and 1.000 for Isaac, when compared with the reference material, and (3) a ROC curve analysis with AUC>0.94 for all cases. Moreover, signifcant diferences were obtained between the three pipelines. In general, results indicate that the three pipelines were able to recog‑ nize the expected variants in the gold standard data set. Conclusions: Our set-theory approach to calculate metrics was able to identify the expected ICCs related variants by the three selected pipelines, but results were completely dependent on the algorithms. We emphasize the importance to assess pipelines using