bigmemoryExtras

bigmemoryExtras provides enhanced management of large file-backed matrices in R, enabling safe and efficient access to BigMatrix objects for high-throughput genomics and molecular biology data analysis.

Key Features:

• Safety Enhancements: The BigMatrix ReferenceClass monitors and restores connections to on-disk data to protect against segmentation faults (segfaults) when working with large datasets.
• Data Integrity Protection: A filesystem-based permissions system prevents accidental modification of critical file-backed matrix data.
• Optimized Data Access: Optimizations improve attaching to and indexing into file-backed matrices with dimnames to accelerate retrieval and manipulation of large matrices.

Scientific Applications:

• High-throughput genomics: Managing and accessing large genomic matrices for downstream analyses in sequencing and array-based studies.
• Molecular biology: Handling extensive molecular datasets that require stable, file-backed storage and indexed access.
• Bioconductor workflows: Integration with the Bioconductor ecosystem to support interoperable analyses of large-scale biological data.

Methodology:

Implemented in R using the BigMatrix ReferenceClass to operate on file-backed matrices with dimnames, employing filesystem-based permissions and connection monitoring/restoration, and integrating with the Bioconductor ecosystem with continuous automated testing.