Transformer-CNN

Transformer-CNN combines transformer-based SMILES embeddings with convolutional neural network architectures to generate QSAR and QSPR models predicting molecular activities and properties.

Key Features:

• SMILES Canonicalization: A Transformer model is trained as a sequence-to-sequence task to convert arbitrary SMILES strings into their canonical forms.
• SMILES Embeddings: The internal encoder output of the Transformer is used to produce SMILES embeddings that capture molecular structural information.
• CharNN Architecture: SMILES embeddings are input to a CharNN model that incorporates convolutional and highway layers similar to those used in DeepChem for downstream modeling.
• SMILES Augmentation: SMILES augmentation is applied during training and inference to improve robustness via a consensus-based prognosis, particularly for small datasets.
• Transfer Learning: Transfer learning is applied on the generated embeddings to enhance performance with limited data.
• Versatility in Tasks: The approach supports both regression and classification tasks across diverse QSAR/QSPR benchmark datasets.

Scientific Applications:

• QSAR/QSPR Modeling: Construction of predictive models for molecular activities and physicochemical properties using learned SMILES representations.
• Small-Dataset Prediction: Improved prediction reliability on small datasets through SMILES augmentation and transfer learning strategies.
• Representation Learning: Generation of transformer-derived SMILES embeddings for use in downstream cheminformatics and machine learning workflows.
• Classification and Regression: Application to both classification and regression problems within cheminformatics benchmarking.

Methodology:

Train a Transformer as a seq2seq model for SMILES canonicalization, extract encoder outputs as SMILES embeddings, apply SMILES augmentation during training and inference, and feed embeddings into a CharNN with convolutional and highway layers (DeepChem-like) with transfer learning on embeddings for regression and classification tasks.