A version of the Tashkeela Arabic diacritized text dataset cleaned from the non-Arabic content and the undiacritized text, then divided into training, development, and testing sets.
The cleaning process includes removing the XML tags and strange symbols, as well as fixing diacritics errors. After that, the tokenization is performed while focusing on the extraction of the Arabic words. The result is a space-separated tokens file, where the words and the numbers are separated, but not the sequences of punctuation (ie, an ending parenthesis followed by a dot). The sentence segmentation is done at usual punctuations such as dots, commas, interrogation/exclamation marks, and line end as well.
The partition process is done by shuffling groups of sentences then dividing each group into three parts (Train/Val/Test) and storing them in individual files.
The original Tashkeela dataset is available at https://sourceforge.net/projects/tashkeela/
Features
- Raw fully-diacritized Arabic texts.
- Over 3 million sentences with different number of words.
- Mostly Classical Arabic.
- Space separated tokens.
- 90% training , 5% validation and 5% testing data.