Skip to content

Add ClpIrStreamWriter for writing IR streams. #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
junhaoliao opened this issue Apr 21, 2025 · 3 comments
Open

Add ClpIrStreamWriter for writing IR streams. #75

junhaoliao opened this issue Apr 21, 2025 · 3 comments
Assignees
Labels
enhancement New feature or request

Comments

@junhaoliao
Copy link
Member

Request

We currently have a ClpIrStreamReader class, which allows reading of CLP IR streams. However, it is difficult to write meaningful tests for this reader within the project itself because there is no built-in mechanism to generate valid IR streams directly in JavaScript/TypeScript.

Currently, we rely on external libraries in another language like clp-ffi-py or clp-loglib-py to create IR files, and use a temporary script to exercise the ClpIrStreamReader functionality. Since these IR test files are not checked into Git, we are unable to check in this script or any related unit tests. This severely limits our ability to verify and maintain the reader's behaviour in a sustainable and automated way.

Possible implementation

Introducing a ClpIrStreamWriter class to the clp-ffi-js project would solve this problem. It would allow us to generate valid IR streams directly within the project, making it possible to write self-contained unit tests for ClpIrStreamReader.

Additionally, having a writer would open up the possibility of implementing a CLP IR logger in JavaScript that logs directly in the CLP IR format. This would expand the usability of clp-ffi-js and bring it more in line with the capabilities provided in the Python bindings.

The writer should support output to a WritableStream, allowing it to integrate seamlessly with browser and Node.js Web Stream APIs. Supporting WritableStream will likely require implementing a custom sink interface that handles write(), close(), and abort() methods as per the WritableStream spec.

@junhaoliao junhaoliao added the enhancement New feature or request label Apr 21, 2025
@junhaoliao junhaoliao self-assigned this Apr 21, 2025
@junhaoliao
Copy link
Member Author

a prototype has been shared at #74 , but read this https://streams.spec.whatwg.org/ before formalizing the design.

@zzxthehappiest
Copy link

Hi @junhaoliao , I have created a draft PR #76 (I am not sure do I need to create another PR with the same name since I don't know how to push my changes into your PR), because I didn't realize that last Sunday is not a day off in China (for May First festival reschedule) and I was planning to finish the rest of work last weekend (will plan in advance more carefully next time!). I think I may spent too much time on learning the background and the emsk framework.

But I still finished the desiredSize (and ready to support backpressure) as well as the abort. Though I am not sure they are implemented in correct way but I add the unit test for them and they seem correct. I have listed what I have done and where I have questions in the draft PR description!

@junhaoliao
Copy link
Member Author

I am not sure do I need to create another PR with the same name since I don't know how to push my changes into your PR

Right, i think you may need permission to push to my repo for updating the existing PR. Anyways, #74 was meant to be a reference PR and that you created #76 is absolutely appropriate.

Thanks for the efforts to contribute even on a make-up workday! Getting to know about the emsdk framework in less than a week is definitely not trivial.

I'm on a business trip this week and may not be getting to the review right away. I'll try my best to go through your questions first, hopefully by tonight or tomorrow. As always, if you need anything urgent, don't hesitate to let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants