Setup for measuring perplexity and latency of rwkv.cpp
implementation of RWKV:
- Script: measure_pexplexity.py
- Model: RWKV-4-Pile-169M-20220807-8023.pth from BlinkDL's Hugging Face.
- Text: Markdown file with 4757 tokens.
- First 1024 tokens were not accounted for in the loss, to let the model take some context in before starting to measure it.
Caveat: "perplexity" here is defined simply as "exp of average per-token cross-entropy loss". This may or may not be correct.