Skip to content

Conversation

Rutik7066
Copy link
Contributor

@Rutik7066 Rutik7066 commented May 8, 2025

Adds Ollama Support

  • We can use Ollama both locally and remotely by setting the environment variable GOLEAM_OLLAMA_BASE_URL with the -e flag.
  • Users can specify a local path or a URL for the image.
  • We send and stream the full received response by relaying the remaining data in provider_metadata_json, which is highly beneficial for consumers.
  • We support all model options provided by Ollama, allowing consumers to have fine-grained control over their models.
  • We have introduced the common StreamError and a common Stream interface that is identical to the existing EventStream. This is extended by EventStream and NdJsonStream, and we can easily add more streaming formats while ensuring minimal code changes, better code maintainability, and adherence to best practices.

Test Video Link

Closes #6
Fixes #6
/claim #6

@@ -1,6 +1,6 @@
[package]
name = "golem-llm-anthropic"
version = "0.0.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not adjust the other creates in the project please

Copy link
Contributor

@mschuwalow mschuwalow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add automated tests or at least a video showcasing the examples passing against a local ollama instance

@jdegoes
Copy link
Contributor

jdegoes commented May 22, 2025

@Rutik7066 Others are ahead of you by a significant amount, I will close this, but feel free to re-open if you address all of these issues and follow instructions precisely. Thanks!

@jdegoes jdegoes closed this May 22, 2025
Rutik Thakre added 2 commits May 22, 2025 17:27
@Rutik7066
Copy link
Contributor Author

Rutik7066 commented May 23, 2025

@jdegoes could you please open this PR I am working on this. I will update the lastest change I have implemented ndjson alongside eventsource.

@jdegoes jdegoes reopened this May 23, 2025
Rutik Thakre added 2 commits May 23, 2025 15:33
@Rutik7066 Rutik7066 force-pushed the main branch 2 times, most recently from 0081403 to 7113ac6 Compare May 23, 2025 16:08
@Rutik7066
Copy link
Contributor Author

All tests are successful. I will add CI test and record video ASAP

Screenshot 2025-05-24 200853

@Rutik7066
Copy link
Contributor Author

@mschuwalow PR is ready for 🚀

Copy link
Contributor

@mschuwalow mschuwalow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, aside the few minor comments. I like how you handled the ndjson and sse stream differences.

NdJsonStream(NdJsonStream),
}

pub trait Stream {
Copy link
Contributor

@mschuwalow mschuwalow May 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As Stream is already a commonly used trait, I would prefer a name that causes less collision

@Rutik7066
Copy link
Contributor Author

Rutik7066 commented May 26, 2025

@mschuwalow added the changes and ci fixes as well. CI run link https://github.com/Rutik7066/golem-llm/actions/runs/15250412169/job/42886202542

@Rutik7066
Copy link
Contributor Author

Looks good to me, aside the few minor comments. I like how you handled the ndjson and sse stream differences.

Yes, with this implementation it would be easier to add different types of stream and will be more maintainable.

@vigoo vigoo merged commit 603526a into golemcloud:main May 26, 2025
5 checks passed
This was referenced May 26, 2025
@Rutik7066
Copy link
Contributor Author

Rutik7066 commented May 26, 2025

Thank you guys @vigoo @mschuwalow 🎉🍾🥂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add local model support
4 participants