Skip to content

Creating test SSTable files to test end-to-end compaction process #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
VMois opened this issue Sep 29, 2023 · 1 comment · Fixed by #44
Closed

Creating test SSTable files to test end-to-end compaction process #22

VMois opened this issue Sep 29, 2023 · 1 comment · Fixed by #44
Assignees

Comments

@VMois
Copy link
Owner

VMois commented Sep 29, 2023

In a Chisel book, to test a custom CPU IP the author used a text file to provide assembly instructions. It would be great before starting developing on real hardware to make sure that our compaction process work correctly. For this, we will need to create a few SSTable files (in a basic text format) with various KV pairs that cover all possible scenarious that can occur during the compaction. This issue covers both creation of test files and creating a full Chisel tests to read files, feed data to circuit, read output and validate it.

It would be really nice if we can run those tests using Verilator backend that is being developed in #21.

The SSTable files created in this issue can be also used for testing after integration with hardware.

@VMois VMois changed the title Creating test SSTable files to test core modules and compare it to the output Creating test SSTable files to test end-to-end compaction process Sep 29, 2023
@VMois
Copy link
Owner Author

VMois commented Oct 18, 2023

It might be beneficial to create a simple binary format for SSTable files as it will make ingestion to custom IP faster. No need to write C/C++ code to convert a text file to binary. This is also the usual way files are stored. I can still create text files but write a short Python script to convert them to binary form.

Binary format should include length of key and value parts, and, of course, key and value data itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant