Streamer

Streamer implements streaming data processing in R to incrementally read, transform, and reduce large high-throughput genomics and molecular biology datasets for memory-efficient analysis.

Key Features:

• Producer-Consumer Architecture: Producers stream data from disk and consumers sequentially perform operations to transform, reduce, or refine data in a modular pipeline.
• Chunked Streaming and Memory Optimization: Processes data in chunks rather than loading entire datasets into memory to optimize memory usage.
• Data Operations: Supports operations such as sampling, indexing, transformation, and reduction on streamed data.
• Bioconductor Interoperability: Integrates with the Bioconductor ecosystem to interoperate with Bioconductor packages within the R environment.
• Formal Review and Automated Testing: Compatible packages undergo Bioconductor's initial review and continuous automated testing to promote reliability and accuracy in computations.

Scientific Applications:

• High-throughput Genomics and Molecular Biology: Enables handling and processing of large-scale high-throughput sequencing and molecular datasets.
• Genomic Sequence Analysis: Facilitates streaming of sequence data for downstream sequence analysis workflows.
• Gene Expression Profiling: Supports processing of large expression matrices or count data for gene expression analyses.
• Complex Bioinformatic Workflows: Applicable to workflows where data volume and memory constraints require incremental processing.

Methodology:

Data are incrementally read from storage by producers into R, and consumers process streamed chunks performing sampling, indexing, transformation, and reduction to manage memory.