We recommend Anaconda as a Python package management system. Please refer to pytorch.org for the details of PyTorch installation. LTS versions are distributed through a different channel than the other versioned releases. Alternatively, you might want to use the Moses tokenizer port in SacreMoses (split from NLTK). You have to install SacreMoses. To build torchtext from source, you need git, CMake and C++11 compiler such as g++. When building from source, make sure that you have the same C++ compiler as the one used to build PyTorch. A simple way is to build PyTorch from source and use the same environment to build torchtext. If you are using the nightly build of PyTorch, check out the environment it was built with conda (here) and pip (here). Text classification: SST2, AG_NEWS, SogouNews, DBpedia, YelpReviewPolarity, YelpReviewFull, YahooAnswers, AmazonReviewPolarity, AmazonReviewFull, IMDB, etc.
Features
- To build torchtext from source, you need git, CMake and C++11 compiler such as g++
- Language modeling: WikiText2, WikiText103, PennTreebank, EnWik9
- Machine translation: IWSLT2016, IWSLT2017, Multi30k
- Sequence tagging (e.g. POS/NER): UDPOS, CoNLL2000Chunking
- Question answering: SQuAD1, SQuAD2
- Model pre-training: CC-100
Project Samples
