Strathweb Phi Engine

A cross-platform library for running Microsoft's Phi-3 locally using candle in GGUF and safe tensors format.

Supported platforms

	Swift	.NET	Kotlin	Python
Package	Swift Package	Nuget	N/A	N/A
Manual Integration	Bindings + XCFramework Bindings + native library	Bindings + native library	Bindings + native library	Bindings + native library
Platforms	macOS arm64 iOS	Windows x64 Windows arm64 (not via Nuget) Linux x64 Linux arm64 (not via Nuget) macOS arm64	Windows x64 Linux x64 macOS arm64	Windows x64 Windows arm64 Linux x64 Linux arm64 macOS arm64

Building instructions

Swift

Build the Swift Package (arm64 Mac required).

./build-swift.sh

This builds:

• the Swift Package under packages/swift/Strathweb.Phi.Engine
• XCFramework under artifacts/swift/strathweb_phi_engine_framework.xcframework

Now open samples/io/phi.engine.sample/phi.engine.sample.xcodeproj and build the SwiftUI app (iOS), or go to samples/swift and run ./run.sh (macOS) to launch the Swift console app.

C#

Install UniFFI C# bindings generator

cargo install uniffi-bindgen-cs --git https://github.com/NordSecurity/uniffi-bindgen-cs --tag v0.8.0+v0.25.0

Build the Nuget package for your platform:

./build-dotnet.sh

or (on Windows)

build-dotnet.bat

or

cargo build --release --manifest-path strathweb-phi-engine/Cargo.toml
dotnet build packages/csharp -c Release
dotnet pack packages/csharp -c Release -o artifacts/csharp

Nuget package will be in artifacts/csharp/Strathweb.Phi.Engine.0.1.0.nupkg. (Optional) Run the sample console app:

cd samples/csharp/console
dotnet run -c Release

Kotlin

Run the sample console app:

cd samples/kotlin
./run.sh

Python

Run the sample console app:

cd samples/python/console
./run.sh # or run.bat on Windows

or use the Jupyter Notebooks

cd samples/python/jupyter
./init.sh # or init.bat on Windows

Now open the Notebook and run the cells.

Compatibility notes

.NET

✅ Tested on Windows arm64

✅ Tested on Windows x64

✅ Tested on Linux arm64

✅ Tested on Linux x64

✅ Tested on macOS arm64.

Swift

✅ Tested on macOS arm64.

✅ Tested on iPad Air M1 8GB RAM

✅ Should work on 6GB RAM iPhones too

❌ Will not work on 4GB RAM iPhones

However, for 4GB RAM iPhones, it's possible to use the (very) low fidelity Q2_K quantized model. Such model is not included in the official Phi-3 release, but I tested this one from HuggingFace on an iPhone 12 mini successfully.

Kotlin

✅ Tested on macOS arm64.

Python

✅ Tested on Windows arm64

✅ Tested on macOS arm64.

GPU Support

Currently the library supports Metal on MacOS. On other platforms only CPU is supported.

AutoGen

The repository also contains a C# integration library for AutoGen, called Strathweb.Phi.Engine.AutoGen. There is an example in the samples/csharp/autogen folder. It allows creating a local Phi-3 agent and integrating it into the other typical AutoGen workflows.

The NuGet package is built when running ./build-dotnet.sh (build-dotnet.bat on Windows).

Blog posts

📝 an initial announcement post

📝 safe tensors announcement post

📝 AutoGen library

📝 .NET examples and Microso.Extensions.AI support