Summary
A bioRxiv preprint benchmarked 16 tools using 30× Illumina whole-genome sequencing and found that only five reliably produced exact deletion, insertion and duplication calls against truth sets. It also reported accuracy losses in ensemble pipelines compared with their strongest component tools.
A bioRxiv preprint from researchers at the University of Tasmania reports that structural-variant calls from short-read genome sequencing can vary substantially in accuracy between analysis tools. Testing 16 tools on 30× Illumina whole-genome sequencing data, the researchers found that only five reliably produced deletion, insertion and duplication calls that exactly matched reference truth sets at base-pair resolution. They also found that some widely used pipelines combining multiple tools performed less accurately than their strongest individual components.
Structural variants are changes to the genome, including deletions, insertions and duplications. Sequencing tools identify these events from DNA data and estimate their locations and sizes. A breakpoint is the position where an event begins or ends; even a small difference in that position can prevent two calls from counting as an exact match. Comparing calls against carefully assembled truth sets helps measure how closely a tool identifies those events.
Why exact calls matter
The researchers used the HG002 T2T-Q100 and Platinum Pedigree truth sets, alongside long-read sequencing databases including 1KGP-ONT and CoLoRS, to assess breakpoints and sizes. Short-read sequencing produces many relatively short DNA fragments, so identifying and comparing some genomic changes can be challenging. Consistent analysis matters when researchers combine or compare results from different studies and population databases.
The preprint reports that ensemble pipelines—including GATK-SV, MOPline and Parliament2—reduced perfect-match accuracy by as much as 98% compared with their highest-performing constituent tools. An ensemble pipeline combines calls from multiple tools. The authors say this accuracy problem carries into gnomAD-SV, which was built using GATK-SV, and may also affect other databases, including TOPMed, All of Us and Biobank Japan.
A proposed merging approach
As a proof of concept, the researchers tested a strategy for merging calls within a sample. On HG002, it produced the highest F1-scores for deletions and insertions. F1-score combines precision and recall: how many reported calls are correct and how many relevant events are detected. The authors also report that their approach recovered more exact matches than the total number of matches from the best-performing ensemble pipeline they tested.
For common HG002 truth-set variants, defined in the abstract as having an allele frequency above 1%, the researchers report very high exact-match concordance between their ensemble callset and the orthogonal long-read databases CoLoRS and 1KGP-ONT.
The findings support closer attention to how short-read tools and their outputs are combined, especially when building datasets intended for comparisons across studies. The proposed merging strategy is a proof of concept evaluated on HG002, and the reported benchmark reflects the specified 30× Illumina data and truth sets. The work is a bioRxiv preprint and describes a step towards standards, rather than an established consensus standard.