Refactor the AI packages

[IMax153]

Type

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update

Description

This release includes a complete refactor of the internals of the base @effect/ai library, with a focus on flexibility for the end user and incorporation of more information from model providers.

Notable Changes

AiLanguageModel and AiEmbeddingModel

The Completions service from @effect/ai has been renamed to AiLanguageModel, and the Embeddings service has similarly been renamed to AiEmbeddingModel. In addition, Completions.create and Completions.toolkit have been unified into AiLanguageModel.generateText. Similarly, Completions.stream and Completions.toolkitStream have been unified into AiLanguageModel.streamText.

Structured Outputs

Completions.structured has been renamed to AiLanguageModel.generateObject, and this method now returns a specialized AiResponse.WithStructuredOutput type, which contains a value property with the result of the structured output call. This enhancement prevents the end user from having to unnecessarily unwrap an Option.

AiModel and AiPlan

The .provide method on a built AiModel / AiPlan has been renamed to .use to improve clarity given that a user is using the services provided by the model / plan to run a particular piece of code.

In addition, the AiPlan.fromModel constructor has been simplified into AiPlan.make, which allows you to create an initial AiPlan with multiple steps incorporated.

For example:

import { AiPlan } from "@effect/ai"
import { OpenAiLanguageModel } from "@effect/ai-openai"
import { AnthropicLanguageModel } from "@effect/ai-anthropic"
import { Effect } from "effect"

const main = Effect.gen(function*() {
  const plan = yield* AiPlan.make({
    model: OpenAiLanguageModel.model("gpt-4"),
    attempts: 1
  }, {
    model: AnthropicLanguageModel.model("claude-3-7-sonnet-latest"),
    attempts: 1
  }, {
    model: AnthropicLanguageModel.model("claude-3-5-sonnet-latest"),
    attempts: 1
  })

  yield* plan.use(program)
})

AiInput and AiResponse

The AiInput and AiResponse types have been refactored to allow inclusion of more information and metadata from model providers where possible, such as reasoning output and prompt cache token utilization.

In addition, for an AiResponse you can now access metadata that is specific to a given provider. For example, when using OpenAi to generate audio, you can check the input and output audio tokens used:

import { OpenAiLanguageModel } from "@effect/ai-openai"
import { Effect, Option } from "effect"

const getDadJoke = OpenAiLanguageModel.generateText({
  prompt: "Generate a hilarious dad joke"
})

Effect.gen(function*() {
  const model = yield* OpenAiLanguageModel.model("gpt-4o")
  const response = yield* model.use(getDadJoke)
  const metadata = response.getProviderMetadata(OpenAiLanguageModel.ProviderMetadata)
  if (Option.isSome(metadata)) {
    console.log(metadata.value)
  }
})

AiTool and AiToolkit

The AiToolkit has been completely refactored to simplify creating a collection of tools and using those tools in requests to model providers. A new AiTool data type has also been introduced to simplify defining tools for a toolkit. AiToolkit.implement has been renamed to AiToolkit.toLayer for clarity, and defining handlers is now very similar to the way handlers are defined in the @effect/rpc library.

A complete example of an AiToolkit implementation and usage can be found below:

import { AiLanguageModel, AiTool, AiToolkit } from "@effect/ai"
import { OpenAiClient, OpenAiLanguageModel } from "@effect/ai-openai"
import { FetchHttpClient, HttpClient, HttpClientRequest, HttpClientResponse } from "@effect/platform"
import { NodeHttpClient, NodeRuntime } from "@effect/platform-node"
import { Array, Config, Console, Effect, Layer, Schema } from "effect"

// =============================================================================
// Domain Models 
// =============================================================================

const DadJoke = Schema.Struct({
  id: Schema.String,
  joke: Schema.String
})

const SearchResponse = Schema.Struct({
  current_page: Schema.Int,
  limit: Schema.Int,
  next_page: Schema.Int,
  previous_page: Schema.Int,
  search_term: Schema.String,
  results: Schema.Array(DadJoke),
  status: Schema.Int,
  total_jokes: Schema.Int,
  total_pages: Schema.Int
})

// =============================================================================
// Service Definitions 
// =============================================================================

export class ICanHazDadJoke extends Effect.Service<ICanHazDadJoke>()("ICanHazDadJoke", {
  dependencies: [FetchHttpClient.layer],
  effect: Effect.gen(function*() {
    const httpClient = (yield* HttpClient.HttpClient).pipe(
      HttpClient.mapRequest(HttpClientRequest.prependUrl("https://icanhazdadjoke.com"))
    )
    const httpClientOk = HttpClient.filterStatusOk(httpClient)

    const search = Effect.fn("ICanHazDadJoke.search")(
      function(term: string) {
        return httpClientOk.get("/search", {
          acceptJson: true,
          urlParams: { term }
        }).pipe(
          Effect.flatMap(HttpClientResponse.schemaBodyJson(SearchResponse)),
          Effect.orDie
        )
      }
    )

    return {
      search
    } as const
  })
}) {}

// =============================================================================
// Toolkit Definition
// =============================================================================

export class DadJokeTools extends AiToolkit.make(
  AiTool.make("GetDadJoke", {
    description: "Fetch a dad joke based on a search term from the ICanHazDadJoke API",
    success: DadJoke,
    parameters: Schema.Struct({
      searchTerm: Schema.String
    })
  })
) {}

// =============================================================================
// Toolkit Handlers
// =============================================================================

export const DadJokeToolHandlers = DadJokeTools.toLayer(
  Effect.gen(function*() {
    const icanhazdadjoke = yield* ICanHazDadJoke
    return {
      GetDadJoke: (params) =>
        icanhazdadjoke.search(params.searchTerm).pipe(
          Effect.flatMap((response) => Array.head(response.results)),
          Effect.orDie
        )
    }
  })
).pipe(Layer.provide(ICanHazDadJoke.Default))

// =============================================================================
// Toolkit Usage
// =============================================================================

const makeDadJoke = Effect.gen(function*() {
  const languageModel = yield* AiLanguageModel.AiLanguageModel
  const toolkit = yield* DadJokeTools

  const response = yield* languageModel.generateText({
    prompt: "Come up with a dad joke about pirates",
    toolkit
  })

  // Here we're only performing a single step with the LLM,
  // but you could loop to perform as many as you like
  if (response.finishReason === "tool-calls") {
      return yield* languageModel.generateText({
        prompt: response
      })
  }
  
  return response
})

const program = Effect.gen(function*() {
  const model = yield* OpenAiLanguageModel.model("gpt-4o")
  const result = yield* model.provide(makeDadJoke)
  yield* Console.log(result.text)
})

const OpenAi = OpenAiClient.layerConfig({
  apiKey: Config.redacted("OPENAI_API_KEY")
}).pipe(Layer.provide(NodeHttpClient.layerUndici))

program.pipe(
  Effect.provide([OpenAi, DadJokeToolHandlers]),
  Effect.tapErrorCause(Effect.logError),
  NodeRuntime.runMain
)

• Related Issue #
• Closes #

[changeset-bot]

🦋 Changeset detected

Latest commit: 68b6d0b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@effect/ai-openai	Major
@effect/ai	Minor
@effect/ai-anthropic	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[tim-smart]

I wonder if we should avoid using interfaces for options. It takes more effort to grok the available options.

vs