ChemPix

ChemPix recognizes hand-drawn hydrocarbon structures from photographs and converts them into machine-readable SMILES representations for computational chemistry.

Key Features:

• Offline Functionality: Operates without continuous internet connectivity.
• Hand-Drawn Recognition: Specialized to recognize hand-drawn hydrocarbon structures from photographs.
• Deep Learning Approach: Uses a neural image captioning framework with a convolutional neural network (CNN) encoder and a long short-term memory (LSTM) decoder to map images to SMILES.
• Training Datasets: Trained on a large auxiliary dataset of RDKit-generated molecular images augmented with image augmentation, degradation, and background addition, plus a crowd-sourced dataset of approximately 600 hand-drawn hydrocarbons collected via a mobile web application.
• Ensemble Model: Employs an ensemble of trained neural networks that vote on predicted molecules and assign confidence values based on consensus.
• Recognition Accuracy: Achieved 76% top-1 accuracy for hand-drawn hydrocarbons and 86% accuracy when considering the top three predictions.

Scientific Applications:

• Research and Education: Conversion of hand-drawn molecular structures into digital SMILES for integration into computational chemistry workflows in research and educational settings.

Methodology:

Maps photographs of hand-drawn hydrocarbons to SMILES using a neural image captioning framework with a CNN encoder and LSTM decoder; training used RDKit-generated images with augmentation, degradation, and background addition plus a crowd-sourced set of ~600 hand-drawn structures; final predictions are produced by an ensemble voting scheme that provides confidence values.