CIDACS-RL

CIDACS-RL performs iterative deterministic record linkage by integrating indexing, search, and scoring methods with Apache Lucene to link records belonging to the same individual across disparate datasets.

Key Features:

• Iterative deterministic algorithm: Employs an iterative deterministic algorithm that refines linkage through successive approximations.
• Indexing, search and scoring: Integrates indexing, search, and scoring methodologies for candidate retrieval and match evaluation.
• Apache Lucene integration: Leverages Apache Lucene for indexing and search operations.
• Validation on gold standard datasets: Validated against gold standard datasets with reported positive predictive value of 99.93% and sensitivity of 99.87%.
• ROC-based threshold optimization: Uses Receiver Operating Characteristic (ROC) analysis to determine an optimal cut-off of 0.896, yielding sensitivity 92.5%, specificity 93.5%, and AUC 97% in a case study.
• Scalability and performance: Demonstrated scalability on a 20 million record simulation with multi-core computation reducing execution time from 550 seconds (single-core) to 150 seconds (multi-core).
• Standard hardware operation: Operates effectively on standard computing hardware without requiring distributed infrastructures.
• Comparative performance: Reported to outperform open-source linkage tools AtyImo, Febrl, FRIL, and RecLink in positive predictive value and sensitivity.

Scientific Applications:

• Record linkage across disparate data sources: Identifying and combining records that pertain to the same individual across disparate datasets.
• Large-scale health data integration: Linking and integrating large-scale health-related datasets for epidemiological and public health research.
• Benchmarking linkage methods: Benchmarking record linkage algorithms using gold standard datasets and ROC analysis.

Methodology:

Uses an iterative deterministic process with indexing, search, and scoring methods implemented via Apache Lucene; employs ROC analysis to set a 0.896 cutoff and uses multi-core computation for performance scaling, as validated on gold standard datasets and a 20 million record simulation.