Build a persistent, high availability and low latency key-value storage engine for derived data. One of such use cases is feature store. These use cases share some common characteristics, that we could utililze to build a better performing database.
- The data is derived, so no need to handle replication or data recovery.
- All the writes are through bulk loading, so no concurrency control needed.
In summary, it's like a scalable cache engine.
- High throughput bulk load
- Low latency (especially on read), target at < 10ms 99.99%
- Handle big data (larger than RAM)
- Horizontal scaling
- Stable performance on heavy access load
- Transactions