DECA

BACKGROUND: XHMM is a widely used tool for copy-number variant (CNV) discovery
from whole exome sequencing data but can require hours to days to run for large
cohorts. A more scalable implementation would reduce the need for specialized
computational resources and enable increased exploration of the configuration
parameter space to obtain the best possible results. RESULTS: DECA is a
horizontally scalable implementation of the XHMM algorithm using the ADAM
framework and Apache Spark that incorporates novel algorithmic optimizations to
eliminate unneeded computation. DECA parallelizes XHMM on both multi-core shared
memory computers and large shared-nothing Spark clusters. We performed CNV
discovery from the read-depth matrix in 2535 exomes in 9.3 min on a 16-core
workstation (35.3x speedup vs. XHMM), 12.7 min using 10 executor cores on a Spark
cluster (18.8x speedup vs. XHMM), and 9.8 min using 32 executor cores on Amazon
AWS' Elastic MapReduce. We performed CNV discovery from the original BAM files in
292 min using 640 executor cores on a Spark cluster. CONCLUSIONS: We describe
DECA's performance, our algorithmic and implementation enhancements to XHMM to
obtain that performance, and our lessons learned porting a complex genome
analysis application to ADAM and Spark. ADAM and Apache Spark are a performant
and productive platform for implementing large-scale genome analyses, but
efficiently utilizing large clusters can require algorithmic optimizations and
careful attention to Spark's configuration parameters.

DECA

Topic

Detail

Publications

Download and documentation