Merge pull request #6855 from EnterpriseDB/release-2025-05-28a

Release 2025-05-28a

Author: djw-m

.github/workflows/tools-events.yml

@@ -0,0 +1,46 @@
+name: Forward key events to the docs-tools repository
+on:
+  pull_request:
+  push:
+  create:
+  delete:
+  release:
+jobs:
+  dispatch-events:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Create event name
+        run: |
+          EVENT_PREFIX="docs-${GITHUB_REPOSITORY//\//-}"
+          echo EVENT_PREFIX="${EVENT_PREFIX@L}" >> $GITHUB_ENV
+      - name: Repository Dispatch
+        uses: peter-evans/repository-dispatch@v3
+        with:
+          token: ${{ secrets.TOOLS_EVENT_SYNC_PAT }}
+          repository: enterprisedb/docs-tools
+          event-type: ${{ env.EVENT_PREFIX }}-${{ github.event_name }}
+          client-payload: |
+            {
+              "ref": "${{ github.head_ref || github.ref }}",
+              "ref_type": "${{ github.ref_type }}",
+              "ref_name": "${{ github.ref_name }}",
+              "repo": "${{ github.repository }}",
+              "sha": "${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}",
+              "type": "${{ github.event_name }}",
+              "push": {
+                "before": "${{ github.event.before || '' }}",
+                "after": "${{ github.event.after || '' }}",
+                "created": "${{ github.event.created || false }}",
+                "deleted": "${{ github.event.deleted || false }}"
+              },
+              "action": "${{ github.event.action }}",
+              "number": "${{ github.event.number }}",
+              "release": {
+                "tag_name": "${{ github.event.release && github.event.release.tag_name || '' }}",
+                "id": "${{ github.event.release && github.event.release.id || '' }}",
+                "tag": "${{ github.event.release && github.event.release.tag_name || '' }}",
+                "name": "${{ github.event.release && github.event.release.name || '' }}",
+                "body": "${{ github.event.release && github.event.release.body || '' }}",
+                "draft": "${{ github.event.release && github.event.release.draft || false }}"
+              }
+            }

advocacy_docs/edb-postgres-ai/ai-accelerator/installing/complete.mdx

@@ -11,9 +11,37 @@ The AI Database extension provides a set of functions to run AI/ML models in the
 
 ```sql
 ebd=# CREATE EXTENSION aidb CASCADE;
+__OUTPUT__
 NOTICE:  installing required extension "vector"
 CREATE EXTENSION
-edb=#
+```
+
+#### Additional steps for EDB Postgres Distributed (PGD)
+A manual step is required to set up replication within the PGD cluster for the AIDB extension catalog tables:
+
+```sql
+bdrdb=# SELECT aidb.bdr_setup();
+__OUTPUT__
+ bdr_setup
+-----------
+
+(1 row)
+```
+
+You can confirm that the AIDB tables are part of the desired replication set using this SQL command:
+```sql
+bdrdb=# select * from bdr.tables where nspname='aidb';
+__OUTPUT__
+ relid | nspname |                relname                |    set_name    |             set_ops             | rel_columns | row_filter | conflict_detection
+-------+---------+---------------------------------------+----------------+---------------------------------+-------------+------------+--------------------
+ 18281 | aidb    | pipeline_runtime_state                | p-2y7357q195-a | {INSERT,UPDATE,DELETE,TRUNCATE} |             |            | row_origin
+ 18354 | aidb    | preparer_registry                     | p-2y7357q195-a | {INSERT,UPDATE,DELETE,TRUNCATE} |             |            | row_origin
+ 18363 | aidb    | preparer_registry_source_table        | p-2y7357q195-a | {INSERT,UPDATE,DELETE,TRUNCATE} |             |            | row_origin
+ 18375 | aidb    | preparer_registry_source_volume       | p-2y7357q195-a | {INSERT,UPDATE,DELETE,TRUNCATE} |             |            | row_origin
+ 18430 | aidb    | knowledge_base_registry               | p-2y7357q195-a | {INSERT,UPDATE,DELETE,TRUNCATE} |             |            | row_origin
+ 18440 | aidb    | knowledge_base_registry_source_table  | p-2y7357q195-a | {INSERT,UPDATE,DELETE,TRUNCATE} |             |            | row_origin
+ 18452 | aidb    | knowledge_base_registry_source_volume | p-2y7357q195-a | {INSERT,UPDATE,DELETE,TRUNCATE} |             |            | row_origin
+(7 rows)
 ```
 
 #### Setting up background workers

advocacy_docs/edb-postgres-ai/ai-accelerator/knowledge_base/performance_tuning.mdx

@@ -36,7 +36,7 @@ INSERT INTO test_data_10k (id, msg) SELECT generate_series(1, 10000) AS id, 'hel
 The optimal batch size may be very different for different models. Measure and tune the batch size for each different model you want to use.
 ```sql
 SELECT aidb.create_table_knowledge_base(
-    name => 'perf_test_b',
+    name => 'perf_test',
     model_name => 'dummy',  -- use the model you want to optimize for
     source_table => 'test_data_10k',
     source_data_column => 'msg',

advocacy_docs/edb-postgres-ai/ai-accelerator/knowledge_base/usage.mdx

@@ -157,8 +157,8 @@ A view is available that lists all the knowledge_bases. [aidb.knowledge_bases](.
 ```sql
 SELECT * FROM aidb.knowledge_bases;
 __OUTPUT__
- id |        name         |     vector_table_name      | vector_table_key_column | vector_table_vector_column |  model_name  | topk | distance_operator | options | source_table_name | source_table_data_column | source_table_data_column_type | source_table_key_column | source_volume_name
-----+---------------------+----------------------------+-------------------------+----------------------------+--------------+------+-------------------+---------+-------------------+--------------------------+-------------------------------+-------------------------+--------------------
+ id |          name            |       vector_table_name         | vector_table_key_column | vector_table_vector_column |  model_name  | topk | distance_operator | options | source_table_name | source_table_data_column | source_table_data_column_type | source_table_key_column | source_volume_name
+----+--------------------------+---------------------------------+-------------------------+----------------------------+--------------+------+-------------------+---------+-------------------+--------------------------+-------------------------------+-------------------------+--------------------
   2 | test_knowledge_base      | test_knowledge_base_vector      | id                      | embeddings                 | simple_model |    5 | InnerProduct      | {}      | test_source_table | content                  | Text                          | id                      |
   5 | test_knowledge_base_cosa | test_knowledge_base_cosa_vector | id                      | embeddings                 | simple_model |    1 | L2                | {}      | test_source_table | content                  | Text                          | id                      |
   3 | test_knowledge_base_cos  | test_knowledge_base_cos_vector  | id                      | embeddings                 | simple_model |    5 | Cosine            | {}      | test_source_table | content                  | Text                          | id                      |
@@ -170,8 +170,8 @@ We recommend that you select only the columns you're interested in:
 ```sql
 SELECT name, source_table_name FROM aidb.knowledge_bases;
 __OUTPUT__
-        name         | source_table_name
----------------------+-------------------
+        name              | source_table_name
+--------------------------+-------------------
  test_knowledge_base      | test_source_table
  test_knowledge_base_cos  | test_source_table
  test_knowledge_base_cosa | test_source_table

advocacy_docs/edb-postgres-ai/ai-accelerator/preparers/examples/chained_preparers.mdx

@@ -25,7 +25,7 @@ SELECT aidb.create_table_preparer(
     destination_table => 'chunked_data__1321',
     destination_data_column => 'chunk',
     destination_key_column => 'id',
-    options => '{"desired_length": 1, "max_length": 1000}'::JSONB  -- Configuration for the ChunkText operation
+    options => '{"desired_length": 1000}'::JSONB  -- Configuration for the ChunkText operation
 );
 ```
 

advocacy_docs/edb-postgres-ai/ai-accelerator/preparers/examples/chunk_text.mdx

@@ -60,18 +60,6 @@ __OUTPUT__
 (9 rows)
 ```
 
-```sql
--- Semantic chunking to split into the largest continuous semantic chunk that fits in the max_length
-SELECT * FROM aidb.chunk_text('This sentence should be its own chunk. This too.', '{"desired_length": 1, "max_length": 1000}');
-
-__OUTPUT__
- part_id |                 chunk
----------+----------------------------------------
-       0 | This sentence should be its own chunk.
-       1 | This too.
-(2 rows)
-```
-
 ## Preparer with table data source
 
 ```sql
@@ -83,7 +71,7 @@ CREATE TABLE source_table__1628
 );
 INSERT INTO source_table__1628
 VALUES (1, 'This is a significantly longer text example that might require splitting into smaller chunks. The purpose of this function is to partition text data into segments of a specified maximum length, for example, this sentence 145 is characters. This enables processing or storage of data in manageable parts.'),
-       (2, 'This sentence should be its own chunk. This too.');
+       (2, 'This is sentence number one. This is sentence number one.');
 
 SELECT aidb.create_table_preparer(
     name => 'preparer__1628',
@@ -94,20 +82,19 @@ SELECT aidb.create_table_preparer(
     destination_data_column => 'chunks',
     source_key_column => 'id',
     destination_key_column => 'id',
-    options => '{"desired_length": 1, "max_length": 1000}'::JSONB  -- Configuration for the ChunkText operation
+    options => '{"desired_length": 120}'::JSONB  -- Configuration for the ChunkText operation
 );
 
 SELECT aidb.bulk_data_preparation('preparer__1628');
 
 SELECT * FROM chunked_data__1628;
 
 __OUTPUT__
- id | part_id | unique_id |                                                                      chunks
-----+---------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------
+ id | part_id | unique_id |                                                        chunks
+----+---------+-----------+-----------------------------------------------------------------------------------------------------------------------
  1  |       0 | 1.part.0  | This is a significantly longer text example that might require splitting into smaller chunks.
- 1  |       1 | 1.part.1  | The purpose of this function is to partition text data into segments of a specified maximum length, for example, this sentence 145 is characters.
- 1  |       2 | 1.part.2  | This enables processing or storage of data in manageable parts.
- 2  |       0 | 2.part.0  | This sentence should be its own chunk.
- 2  |       1 | 2.part.1  | This too.
-(5 rows)
+ 1  |       1 | 1.part.1  | The purpose of this function is to partition text data into segments of a specified maximum length, for example, this
+ 1  |       2 | 1.part.2  | sentence 145 is characters. This enables processing or storage of data in manageable parts.
+ 2  |       0 | 2.part.0  | This is sentence number one. This is sentence number one.
+(4 rows)
 ```

advocacy_docs/edb-postgres-ai/ai-accelerator/preparers/examples/chunk_text_auto_processing.mdx

@@ -29,7 +29,7 @@ SELECT aidb.create_table_preparer(
     destination_data_column => 'chunk',
     source_key_column => 'id',
     destination_key_column => 'id',
-    options => '{"desired_length": 1, "max_length": 1000}'::JSONB  -- Configuration for the ChunkText operation
+    options => '{"desired_length": 1000}'::JSONB  -- Configuration for the ChunkText operation
 );
 
 SELECT aidb.set_auto_preparer('preparer__1628', 'Live');

advocacy_docs/edb-postgres-ai/ai-accelerator/preparers/primitives.mdx

@@ -14,7 +14,7 @@ All data preparation operations can be customized with different options. The AP
 
 ## Chunk text
 
-Call `aidb.chunk_text()` to break text into smaller chunks.
+Call `aidb.chunk_text()` to intelligently split long text into smaller, semantically coherent chunks, ideal for processing or storage within LLM context limits.
 
 ```sql
 SELECT * FROM aidb.chunk_text(
@@ -31,8 +31,18 @@ __OUTPUT__
 (3 rows)
 ```
 
-- The `desired_length` size is the target size for the chunk. In most cases, this value also serves as the maximum size of the chunk. It's possible for a chunk to be returned that's less than the `desired` value, as adding the next piece of text may have made it larger than the `desired` capacity.
-- The `max_length` size is the maximum possible chunk size that can be generated. Setting this to a value larger than `desired` means that the chunk should be as close to `desired` as possible but can be larger if it means staying at a larger semantic level.
+### Options
+
+- `desired_length` (required): The target chunk size, in characters. This is the size the splitter will try to reach when forming each chunk, while preserving semantic structure. If `max_length` is not provided, `desired_length` also acts as the hard upper limit for the chunk size.
+- `max_length` (optional): The upper bound for chunk size, in characters. If specified, the function will try to generate chunks close to `desired_length` but may extend up to `max_length` to preserve larger semantic units (like full sentences or paragraphs). Chunks will only exceed `desired_length` when it's necessary to avoid cutting across meaningful boundaries.
+  - Specifying `desired_length = max_length` results in fixed-size chunks (e.g., when filling a context window exactly for embeddings).
+  - Use a larger `max_length` if you want to stay within a soft limit but allow some flexibility to preserve higher semantic units, common in RAG, summarization, or Q&A applications.
+
+### Algorithm Summary
+
+- Text is split using a hierarchy of semantic boundaries: characters, graphemes, words, sentences, and increasingly long newline sequences (e.g., paragraphs).
+- The splitter attempts to form the largest semantically valid chunk that fits within the specified size range.
+- Chunks may be returned that are shorter than `desired_length` if adding the next semantic unit would exceed max_length.
 
 !!! Tip
 This operation transforms the shape of the data, automatically unnesting collections by introducing a `part_id` column. See the [unnesting concept](./concepts#unnesting) for more detail.

advocacy_docs/edb-postgres-ai/ai-accelerator/preparers/volume_destination.mdx

@@ -68,7 +68,7 @@ SELECT aidb.create_volume_preparer(
                operation => 'ChunkText',
                source_volume_name => 'text_files_volume',
                destination_table => 'chunk_output_volume',
-               options => '{"desired_length": 1, "max_length": 100}'::JSONB
+               options => '{"desired_length": 1000}'::JSONB
        );
 
 SELECT aidb.bulk_data_preparation('chunking_example_volumes');

advocacy_docs/edb-postgres-ai/ai-accelerator/reference/index.mdx

@@ -21,8 +21,9 @@ navigation:
 * [aidb.create_model](models#aidbcreate_model)
 * [aidb.get_model](models#aidbget_model)
 * [aidb.delete_model](models#aidbdelete_model)
-* [aidb.get_hcp_models](models#aidbget_hcp_models)
+* [aidb.list_hcp_models](models#aidblist_hcp_models)
 * [aidb.create_hcp_model](models#aidbcreate_hcp_model)
+* [aidb.sync_hcp_models](models#aidbsync_hcp_models)
 * [aidb.encode_text](models#aidbencode_text)
 * [aidb.encode_text_batch](models#aidbencode_text_batch)
 * [aidb.decode_text](models#aidbdecode_text)

advocacy_docs/edb-postgres-ai/ai-accelerator/reference/models.mdx

@@ -161,7 +161,7 @@ __OUTPUT__
 | `delete_model` | jsonb | The name, provider, and options of the deleted model |
 
 
-### `aidb.get_hcp_models`
+### `aidb.list_hcp_models`
 
 Gets models running on the hybrid control plane.
 
@@ -194,6 +194,19 @@ Creates a new model in the system by referencing a running instance in the HCP
 | `name`           | text |         | User-defined name of the model            |
 | `hcp_model_name` | text |         | Name of the model instance running on HCP |
 
+### `aidb.sync_hcp_models`
+
+Creates models in the HCP and sets `is_hcp_model=true`; deletes models with that setting if not found in the HCP.
+
+#### Returns
+
+| Column   | Type | Description                                                                     |
+| -------- | ---- | ------------------------------------------------------------------------------- |
+| `status` | text | Synchronization status; either `deleted`, `unchanged`, `created`, or `skipped`. |
+| `model`  | text | The name the synchronized HCP model.                                            |
+
+
+
 ### `aidb.encode_text`
 
 Encodes text using a model, generating a vector representation of a given text input.

advocacy_docs/edb-postgres-ai/ai-accelerator/rel_notes/ai-accelerator_4.1.1_rel_notes.mdx

@@ -0,0 +1,38 @@
+---
+title: AI Accelerator - Pipelines 4.1.1 release notes
+navTitle: Version 4.1.1
+originalFilePath: advocacy_docs/edb-postgres-ai/ai-accelerator/rel_notes/src/rel_notes_4.1.1.yml
+editTarget: originalFilePath
+---
+
+Released: 28 May 2025
+
+This is a minor release that includes bug fixes.
+Note: this release depends on the new PGFS version 2.0.2. If you're using PGFS together with AIDB, please update PGFS as well.
+
+## Highlights
+
+- Bug fixes and UX enhancements.
+
+## Enhancements
+
+<table class="table w-100"><thead><tr><th>Description</th><th width="10%">Addresses</th></tr></thead><tbody>
+<tr><td><details><summary>Change the tracking of &quot;run IDs&quot; for volume knowledge bases from a PG sequence to a PG table column</summary><hr/><p>Knowledge bases with volume source need to assign an incrementing ID to each listing done on the source.
+This is necessary to detect when a listing is complete and recognize which previously seen objects were not encountered any more.
+In this release the method for tracking run IDs is changed to a table column since the Postgres sequence used before did not
+work on Postgres Distributed (PGD) clusters.</p>
+</details></td><td></td></tr>
+<tr><td><details><summary>Support AIDB volumes in arbitrary Postgres schemas</summary><hr/><p>Volumes for AIDB can be created in arbitrary schemas in Postgres either by exlicitly passing a schema to the create function, or by
+setting a &quot;current schema&quot; via the search path. When using volumes in AIDB pipelines, the volume reference needs to be stored.
+AIDB will now store and explicitly use a schema when later executing a pipeline. Previously, the search path was used to find volumes.</p>
+</details></td><td></td></tr>
+<tr><td><details><summary>Background worker dispatcher is no longer persistent</summary><hr/><p>When background workers are enabled for AIDB, a root-worker will start a database dispatcher for each database within the Postgres instance.
+This database dispatcher checks if AIDB is installed in the DB and if any pipelines need to be run in the background.
+The database dispatcher is now no longer a permanent process by default. When a database does not use AIDB, the database dispatcher
+exits and will be re-started to check for AIDB every 2 minutes.
+This avoids holding a permanent connection to the DB that would block e.g. &quot;drop table&quot; commands.
+If AIDB is installed in the DB, then the database dispatcher worker will keep running. Users can drop the extension in order to release the connection.</p>
+</details></td><td></td></tr>
+</tbody></table>
+
+

advocacy_docs/edb-postgres-ai/ai-accelerator/rel_notes/index.mdx

@@ -4,6 +4,7 @@ navTitle: Release notes
 description: Release notes for EDB Postgres AI - AI Accelerator
 indexCards: none
 navigation:
+  - ai-accelerator_4.1.1_rel_notes
   - ai-accelerator_4.1.0_rel_notes
   - ai-accelerator_4.0.1_rel_notes
   - ai-accelerator_4.0.0_rel_notes
@@ -23,6 +24,7 @@ The EDB Postgres AI - AI Accelerator describes the latest version of AI Accelera
 
 | AI Accelerator version | Release Date |
 |---|---|
+| [4.1.1](./ai-accelerator_4.1.1_rel_notes) | 28 May 2025 |
 | [4.1.0](./ai-accelerator_4.1.0_rel_notes) | 19 May 2025 |
 | [4.0.1](./ai-accelerator_4.0.1_rel_notes) | 09 May 2025 |
 | [4.0.0](./ai-accelerator_4.0.0_rel_notes) | 05 May 2025 |