Skip to content

Instantly share code, notes, and snippets.

@taoyds

taoyds/preprocess_dict.py

Last active March 20, 2020 12:24
Show Gist options
  • Save taoyds/07889e38dbcb0aa2b73fcf0cff66a33b to your computer and use it in GitHub Desktop.
Save taoyds/07889e38dbcb0aa2b73fcf0cff66a33b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
preprocess_dict.py
if args.joined_dictionary:
assert not args.srcdict or not args.tgtdict, \
"cannot use both --srcdict and --tgtdict with --joined-dictionary"
if args.srcdict:
src_dict = task.load_dictionary(args.srcdict)
elif args.tgtdict:
src_dict = task.load_dictionary(args.tgtdict)
else:
assert args.trainpref, "--trainpref must be set if --srcdict is not specified"
src_dict = build_dictionary(
{train_path(lang) for lang in [args.source_lang, args.target_lang]}, src=True
)
tgt_dict = src_dict
else:
if args.srcdict:
src_dict = task.load_dictionary(args.srcdict)
else:
assert args.trainpref, "--trainpref must be set if --srcdict is not specified"
src_dict = build_dictionary([train_path(args.source_lang)], src=True)
if target:
if args.tgtdict:
tgt_dict = task.load_dictionary(args.tgtdict)
else:
assert args.trainpref, "--trainpref must be set if --tgtdict is not specified"
tgt_dict = build_dictionary([train_path(args.target_lang)], tgt=True)
else:
tgt_dict = None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment