Skip to content

feat(ai): add support for grounding with google search #9068

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/five-kids-grow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'firebase': minor
'@firebase/ai': minor
---

Add support for Grounding with Google Search.
46 changes: 39 additions & 7 deletions common/api-review/ai.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -485,6 +485,15 @@ export interface GoogleAIGenerateContentResponse {
usageMetadata?: UsageMetadata;
}

// @public
export interface GoogleSearch {
}

// @public
export interface GoogleSearchTool {
googleSearch: GoogleSearch;
}

// @public @deprecated (undocumented)
export interface GroundingAttribution {
// (undocumented)
Expand All @@ -497,16 +506,29 @@ export interface GroundingAttribution {
web?: WebAttribution;
}

// @public
export interface GroundingChunk {
web?: WebGroundingChunk;
}

// @public
export interface GroundingMetadata {
// @deprecated (undocumented)
groundingAttributions: GroundingAttribution[];
// (undocumented)
groundingChunks?: GroundingChunk[];
groundingSupports?: GroundingSupport[];
// @deprecated (undocumented)
retrievalQueries?: string[];
// (undocumented)
searchEntryPoint?: SearchEntrypoint;
webSearchQueries?: string[];
}

// @public
export interface GroundingSupport {
groundingChunkIndices?: number[];
segment?: Segment;
}

// @public
export enum HarmBlockMethod {
PROBABILITY = "PROBABILITY",
Expand Down Expand Up @@ -857,14 +879,17 @@ export enum SchemaType {
STRING = "string"
}

// @public (undocumented)
// @public
export interface SearchEntrypoint {
renderedContent?: string;
}

// @public
export interface Segment {
// (undocumented)
endIndex: number;
// (undocumented)
partIndex: number;
// (undocumented)
startIndex: number;
text: string;
}

// @public
Expand Down Expand Up @@ -901,7 +926,7 @@ export interface TextPart {
}

// @public
export type Tool = FunctionDeclarationsTool;
export type Tool = FunctionDeclarationsTool | GoogleSearchTool;

// @public
export interface ToolConfig {
Expand Down Expand Up @@ -961,5 +986,12 @@ export interface WebAttribution {
uri: string;
}

// @public
export interface WebGroundingChunk {
domain?: string;
title?: string;
uri?: string;
}


```
12 changes: 12 additions & 0 deletions docs-devsite/_toc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -76,10 +76,18 @@ toc:
path: /docs/reference/js/ai.generativemodel.md
- title: GoogleAIBackend
path: /docs/reference/js/ai.googleaibackend.md
- title: GoogleSearch
path: /docs/reference/js/ai.googlesearch.md
- title: GoogleSearchTool
path: /docs/reference/js/ai.googlesearchtool.md
- title: GroundingAttribution
path: /docs/reference/js/ai.groundingattribution.md
- title: GroundingChunk
path: /docs/reference/js/ai.groundingchunk.md
- title: GroundingMetadata
path: /docs/reference/js/ai.groundingmetadata.md
- title: GroundingSupport
path: /docs/reference/js/ai.groundingsupport.md
- title: ImagenGCSImage
path: /docs/reference/js/ai.imagengcsimage.md
- title: ImagenGenerationConfig
Expand Down Expand Up @@ -130,6 +138,8 @@ toc:
path: /docs/reference/js/ai.schemarequest.md
- title: SchemaShared
path: /docs/reference/js/ai.schemashared.md
- title: SearchEntrypoint
path: /docs/reference/js/ai.searchentrypoint.md
- title: Segment
path: /docs/reference/js/ai.segment.md
- title: StartChatParams
Expand All @@ -150,6 +160,8 @@ toc:
path: /docs/reference/js/ai.videometadata.md
- title: WebAttribution
path: /docs/reference/js/ai.webattribution.md
- title: WebGroundingChunk
path: /docs/reference/js/ai.webgroundingchunk.md
- title: analytics
path: /docs/reference/js/analytics.md
section:
Expand Down
21 changes: 21 additions & 0 deletions docs-devsite/ai.googlesearch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
Project: /docs/reference/js/_project.yaml
Book: /docs/reference/_book.yaml
page_type: reference

{% comment %}
DO NOT EDIT THIS FILE!
This is generated by the JS SDK team, and any local changes will be
overwritten. Changes should be made in the source code at
https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# GoogleSearch interface
Configuration for the [GoogleSearchTool](./ai.googlesearchtool.md#googlesearchtool_interface)<!-- -->.

Currently, this interface is empty and serves as a placeholder for future configuration options.

<b>Signature:</b>

```typescript
export declare interface GoogleSearch
```
37 changes: 37 additions & 0 deletions docs-devsite/ai.googlesearchtool.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
Project: /docs/reference/js/_project.yaml
Book: /docs/reference/_book.yaml
page_type: reference

{% comment %}
DO NOT EDIT THIS FILE!
This is generated by the JS SDK team, and any local changes will be
overwritten. Changes should be made in the source code at
https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# GoogleSearchTool interface
A tool that allows the generative model to connect to Google Search to access and incorporate up-to-date information from the web into its responses.

When using Grounding with Google Search, you may receive a response with data generated from Google's search engine. This response is a "Grounded Result" subject to the Grounding with Google Search terms in the [Service Specific Terms](https://cloud.google.com/terms/service-terms)<!-- -->.

<b>Signature:</b>

```typescript
export declare interface GoogleSearchTool
```

## Properties

| Property | Type | Description |
| --- | --- | --- |
| [googleSearch](./ai.googlesearchtool.md#googlesearchtoolgooglesearch) | [GoogleSearch](./ai.googlesearch.md#googlesearch_interface) | Specifies the Google Search configuration. Currently, this is an empty object, but it's reserved for future configuration options. |

## GoogleSearchTool.googleSearch

Specifies the Google Search configuration. Currently, this is an empty object, but it's reserved for future configuration options.

<b>Signature:</b>

```typescript
googleSearch: GoogleSearch;
```
37 changes: 37 additions & 0 deletions docs-devsite/ai.groundingchunk.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
Project: /docs/reference/js/_project.yaml
Book: /docs/reference/_book.yaml
page_type: reference

{% comment %}
DO NOT EDIT THIS FILE!
This is generated by the JS SDK team, and any local changes will be
overwritten. Changes should be made in the source code at
https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# GroundingChunk interface
Represents a chunk of retrieved data that supports a claim in the model's response. This is part of the grounding information provided when grounding is enabled.

Important: If using Grounding with Google Search, you are required to comply with the [Service Specific Terms](https://cloud.google.com/terms/service-terms) for "Grounding with Google Search".

<b>Signature:</b>

```typescript
export interface GroundingChunk
```

## Properties

| Property | Type | Description |
| --- | --- | --- |
| [web](./ai.groundingchunk.md#groundingchunkweb) | [WebGroundingChunk](./ai.webgroundingchunk.md#webgroundingchunk_interface) | Contains details if the grounding chunk is from a web source. |

## GroundingChunk.web

Contains details if the grounding chunk is from a web source.

<b>Signature:</b>

```typescript
web?: WebGroundingChunk;
```
57 changes: 55 additions & 2 deletions docs-devsite/ai.groundingmetadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,11 @@ https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# GroundingMetadata interface
Metadata returned to client when grounding is enabled.
Metadata returned when grounding is enabled.

Currently, the only way to use grounding is to include a [GoogleSearchTool](./ai.googlesearchtool.md#googlesearchtool_interface) in your [GenerationConfig](./ai.generationconfig.md#generationconfig_interface)<!-- -->.

Important: If using Grounding with Google Search, you are required to comply with the [Service Specific Terms](https://cloud.google.com/terms/service-terms) for "Grounding with Google Search".

<b>Signature:</b>

Expand All @@ -23,13 +27,17 @@ export interface GroundingMetadata
| Property | Type | Description |
| --- | --- | --- |
| [groundingAttributions](./ai.groundingmetadata.md#groundingmetadatagroundingattributions) | [GroundingAttribution](./ai.groundingattribution.md#groundingattribution_interface)<!-- -->\[\] | |
| [groundingChunks](./ai.groundingmetadata.md#groundingmetadatagroundingchunks) | [GroundingChunk](./ai.groundingchunk.md#groundingchunk_interface)<!-- -->\[\] | A list of [GroundingChunk](./ai.groundingchunk.md#groundingchunk_interface) objects. Each chunk represents a piece of retrieved content (e.g. from a web page). that the model used to ground its response.<!-- -->Important: If using Grounding with Google Search, you are required to comply with the [Service Specific Terms](https://cloud.google.com/terms/service-terms) for "Grounding with Google Search". |
| [groundingSupports](./ai.groundingmetadata.md#groundingmetadatagroundingsupports) | [GroundingSupport](./ai.groundingsupport.md#groundingsupport_interface)<!-- -->\[\] | A list of [GroundingSupport](./ai.groundingsupport.md#groundingsupport_interface) objects. Each object details how specific segments of the model's response are supported by the <code>groundingChunks</code>.<!-- -->Important: If using Grounding with Google Search, you are required to comply with the [Service Specific Terms](https://cloud.google.com/terms/service-terms) for "Grounding with Google Search". |
| [retrievalQueries](./ai.groundingmetadata.md#groundingmetadataretrievalqueries) | string\[\] | |
| [webSearchQueries](./ai.groundingmetadata.md#groundingmetadatawebsearchqueries) | string\[\] | |
| [searchEntryPoint](./ai.groundingmetadata.md#groundingmetadatasearchentrypoint) | [SearchEntrypoint](./ai.searchentrypoint.md#searchentrypoint_interface) | Google search entry point for web searches. This contains An HTML/CSS snippet that \*must\* be embedded in an app to display a Google Search Entry point for follow-up web searches related to the model's "Grounded Response".<!-- -->Important: If using Grounding with Google Search, you are required to comply with the [Service Specific Terms](https://cloud.google.com/terms/service-terms) for "Grounding with Google Search". |
| [webSearchQueries](./ai.groundingmetadata.md#groundingmetadatawebsearchqueries) | string\[\] | A list of web search queries that the model performed to gather the grounding information. These can be used to allow users to explore the search results themselves.<!-- -->Important: If using Grounding with Google Search, you are required to comply with the [Service Specific Terms](https://cloud.google.com/terms/service-terms) for "Grounding with Google Search". |

## GroundingMetadata.groundingAttributions

> Warning: This API is now obsolete.
>
> Use [GroundingChunk](./ai.groundingchunk.md#groundingchunk_interface) instead.
>

<b>Signature:</b>
Expand All @@ -38,16 +46,61 @@ export interface GroundingMetadata
groundingAttributions: GroundingAttribution[];
```

## GroundingMetadata.groundingChunks

A list of [GroundingChunk](./ai.groundingchunk.md#groundingchunk_interface) objects. Each chunk represents a piece of retrieved content (e.g. from a web page). that the model used to ground its response.

Important: If using Grounding with Google Search, you are required to comply with the [Service Specific Terms](https://cloud.google.com/terms/service-terms) for "Grounding with Google Search".

<b>Signature:</b>

```typescript
groundingChunks?: GroundingChunk[];
```

## GroundingMetadata.groundingSupports

A list of [GroundingSupport](./ai.groundingsupport.md#groundingsupport_interface) objects. Each object details how specific segments of the model's response are supported by the `groundingChunks`<!-- -->.

Important: If using Grounding with Google Search, you are required to comply with the [Service Specific Terms](https://cloud.google.com/terms/service-terms) for "Grounding with Google Search".

<b>Signature:</b>

```typescript
groundingSupports?: GroundingSupport[];
```

## GroundingMetadata.retrievalQueries

> Warning: This API is now obsolete.
>
> Use [GroundingSupport](./ai.groundingsupport.md#groundingsupport_interface) instead.
>

<b>Signature:</b>

```typescript
retrievalQueries?: string[];
```

## GroundingMetadata.searchEntryPoint

Google search entry point for web searches. This contains An HTML/CSS snippet that \*must\* be embedded in an app to display a Google Search Entry point for follow-up web searches related to the model's "Grounded Response".

Important: If using Grounding with Google Search, you are required to comply with the [Service Specific Terms](https://cloud.google.com/terms/service-terms) for "Grounding with Google Search".

<b>Signature:</b>

```typescript
searchEntryPoint?: SearchEntrypoint;
```

## GroundingMetadata.webSearchQueries

A list of web search queries that the model performed to gather the grounding information. These can be used to allow users to explore the search results themselves.

Important: If using Grounding with Google Search, you are required to comply with the [Service Specific Terms](https://cloud.google.com/terms/service-terms) for "Grounding with Google Search".

<b>Signature:</b>

```typescript
Expand Down
48 changes: 48 additions & 0 deletions docs-devsite/ai.groundingsupport.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
Project: /docs/reference/js/_project.yaml
Book: /docs/reference/_book.yaml
page_type: reference

{% comment %}
DO NOT EDIT THIS FILE!
This is generated by the JS SDK team, and any local changes will be
overwritten. Changes should be made in the source code at
https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# GroundingSupport interface
Provides information about how a specific segment of the model's response is supported by the retrieved grounding chunks.

Important: If using Grounding with Google Search, you are required to comply with the [Service Specific Terms](https://cloud.google.com/terms/service-terms) for "Grounding with Google Search".

<b>Signature:</b>

```typescript
export interface GroundingSupport
```

## Properties

| Property | Type | Description |
| --- | --- | --- |
| [groundingChunkIndices](./ai.groundingsupport.md#groundingsupportgroundingchunkindices) | number\[\] | A list of indices that refer to specific [GroundingChunk](./ai.groundingchunk.md#groundingchunk_interface) objects within the [GroundingMetadata.groundingChunks](./ai.groundingmetadata.md#groundingmetadatagroundingchunks) array. These referenced chunks are the sources that support the claim made in the associated <code>segment</code> of the response. For example, an array <code>[1, 3, 4]</code> means that <code>groundingChunks[1]</code>, <code>groundingChunks[3]</code>, and <code>groundingChunks[4]</code> are the retrieved content supporting this part of the response. |
| [segment](./ai.groundingsupport.md#groundingsupportsegment) | [Segment](./ai.segment.md#segment_interface) | Specifies the segment of the model's response content that this grounding support pertains to. |

## GroundingSupport.groundingChunkIndices

A list of indices that refer to specific [GroundingChunk](./ai.groundingchunk.md#groundingchunk_interface) objects within the [GroundingMetadata.groundingChunks](./ai.groundingmetadata.md#groundingmetadatagroundingchunks) array. These referenced chunks are the sources that support the claim made in the associated `segment` of the response. For example, an array `[1, 3, 4]` means that `groundingChunks[1]`<!-- -->, `groundingChunks[3]`<!-- -->, and `groundingChunks[4]` are the retrieved content supporting this part of the response.

<b>Signature:</b>

```typescript
groundingChunkIndices?: number[];
```

## GroundingSupport.segment

Specifies the segment of the model's response content that this grounding support pertains to.

<b>Signature:</b>

```typescript
segment?: Segment;
```
Loading
Loading