Funani Ndou KUDANDOU

Witty-Kitty / alp_data_prep.py

Created February 18, 2019 11:54

Text pre-processing

	import nltk
	from nltk.tokenize import word_tokenize
	from nltk.text import Text

	# read in text data
	file = open("crawl-for-parallel-corpora/DataSet/luganda.txt", "r")
	raw = file.read()

	# tokenize
	tokens = word_tokenize(raw)