subs2vec

subs2vec generates and evaluates multilingual word embeddings using the fastText skipgram algorithm trained on the OpenSubtitles corpus to capture lexical semantics from pseudo-conversational speech transcriptions across 55 languages.

Key Features:

• Multilingual Coverage: Provides pretrained word embeddings in 55 languages that reflect conversational language use.
• Training Corpus: Trained on the OpenSubtitles corpus, comprising large-scale pseudo-conversational transcriptions from television shows and movies.
• Performance Benchmarking: Demonstrates performance comparable to, or exceeding, embeddings trained on non-conversational text across standard benchmark datasets.
• Lexical Norm Prediction Evaluation: Introduces a novel evaluation method that predicts experimental lexical norms in multiple languages.
• Evaluation Tasks and Tools: Includes scripts and command-line tools for evaluating word vectors on semantic similarity, semantic and syntactic analogy, and lexical norm prediction.
• Implementation Package: Provides a Python 3.7 package and a Subs2Vec.py script for training fastText models from an OpenSubtitles archive or Wikipedia and producing word vectors.

Scientific Applications:

• Cross-Linguistic Studies: Enables exploration of linguistic phenomena across languages using a consistent embedding framework.
• Psycholinguistic Research: Supports prediction of lexical norms and analysis of how embeddings relate to human language processing.
• Machine Translation and NLP Tasks: Supplies contextually rich word representations that can enhance machine translation and other NLP applications.

Methodology:

Training uses the fastText implementation of the skipgram algorithm to predict surrounding words and learn semantic and syntactic word representations from the OpenSubtitles corpus (and optionally Wikipedia), with evaluation on standard benchmark datasets and by predicting experimental lexical norms in multiple languages.