From 096affb7d9ad375150c9d2a52c08a082d868ea21 Mon Sep 17 00:00:00 2001 From: David Weiss Date: Fri, 22 Dec 2023 02:09:55 +0200 Subject: [PATCH] feat: added a way to rename key name to support the use of a field named Id (by mapping it to book_id for example) and to support mapping sub collactions key to root collactions field names. --- README.md | 47 +++++++++---------- extension.yaml | 11 ++++- .../firestore-typesense-search.env.local | 1 + .../test-params-flatten-nested-true.local.env | 1 + extensions/test-params.example.env | 1 + functions/src/config.js | 9 ++++ functions/src/utils.js | 7 ++- test/backfillToTypesenseFromFirestore.spec.js | 5 ++ test/indexToTypesenseOnFirestoreWrite.spec.js | 35 ++++++++++++++ 9 files changed, 90 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index 4a6a0f2..6156f1a 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,15 @@ -# Firestore / Firebase Typesense Search Extension ⚡ 🔍 +# Firestore / Firebase Typesense Search Extension ⚡ 🔍 -A Firebase extension to sync data from your Firestore collection to [Typesense](https://typesense.org/), +A Firebase extension to sync data from your Firestore collection to [Typesense](https://typesense.org/), to be able to do full-text fuzzy search on your Firestore data, with typo tolerance, faceting, filtering, sorting, curation, synonyms, geosearch and more. -This extension listens to your specified Firestore collection and syncs Firestore documents to Typesense +This extension listens to your specified Firestore collection and syncs Firestore documents to Typesense on creation, updates and deletes. It also provides a function to help you backfill data. **What is Typesense?** If you're new to [Typesense](https://typesense.org), it is an open source search engine that is simple to use, run and scale, with clean APIs and documentation. Think of it as an open source alternative to Algolia and an easier-to-use, batteries-included alternative to ElasticSearch. Get a quick overview from [this guide](https://typesense.org/docs/guide). - ## ⚙ī¸ Usage ### Step 1ī¸âƒŖ : Setup Prerequisites @@ -19,12 +18,12 @@ Before installing this extension, make sure that you have: 1. [Set up a Cloud Firestore database](https://firebase.google.com/docs/firestore/quickstart) in your Firebase project. 2. [Set up](https://typesense.org/docs/guide/install-typesense.html) a Typesense cluster on [Typesense Cloud](https://cloud.typesense.org) or [Self-Hosted](https://typesense.org/docs/guide/install-typesense.html#option-2-local-machine-self-hosting) (free). -3. Set up a Typesense Collection either through the Typesense Cloud dashboard or - through the [API](https://typesense.org/docs/latest/api/collections.html#create-a-collection). - -⚠ī¸ ☝ī¸ #3 above is a commonly missed step. This extension **does not create the Typesense Collection for you**. Instead it syncs data to a Typesense collection you've already created. If you see an HTTP 404 in the extension logs, it's most likely because of missing this step. +3. Set up a Typesense Collection either through the Typesense Cloud dashboard or + through the [API](https://typesense.org/docs/latest/api/collections.html#create-a-collection). + +⚠ī¸ ☝ī¸ #3 above is a commonly missed step. This extension **does not create the Typesense Collection for you**. Instead it syncs data to a Typesense collection you've already created. If you see an HTTP 404 in the extension logs, it's most likely because of missing this step. -### Step 2ī¸âƒŖ : Install the Extension +### Step 2ī¸âƒŖ : Install the Extension You can install this extension either through the Firebase Web console or through the Firebase CLI. @@ -47,22 +46,23 @@ firebase ext:install typesense/firestore-typesense-search --project=[your-projec When you install this extension, you'll be able to configure the following parameters: | Parameter | Description | -|-----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Firestore Collection Path | The Firestore collection that needs to be indexed into Typesense. | | Firestore Collection Fields | A comma separated list of fields that need to be indexed from each Firestore document. Leave blank to index all fields. | +| Typesense Fields Renames | A comma separated list of field renames in the format `old_field_name:new_field_name`. This is useful when you want to rename fields from Firestore to a different name in Typesense. Leave blank to not rename any fields. | | Flatten Nested Documents | Should nested documents in Firestore be flattened before they are indexed in Typesense? Set to "Yes" for Typesense Server versions v0.23.1 and below, since indexing Nested objects is natively supported only in Typesense Server v0.24 and above. | -| Typesense Hosts | A comma-separated list of Typesense Hosts (only domain without https or port number). For single node clusters, a single hostname is sufficient. For multi-node Highly Available or (Search Delivery Network) SDN Clusters, please be sure to mention all hostnames in a comma-separated list. | +| Typesense Hosts | A comma-separated list of Typesense Hosts (only domain without https or port number). For single node clusters, a single hostname is sufficient. For multi-node Highly Available or (Search Delivery Network) SDN Clusters, please be sure to mention all hostnames in a comma-separated list. | | Typesense API Key | A Typesense API key with admin permissions. Click on "Generate API Key" in cluster dashboard in Typesense Cloud. | | Typesense Collection Name | Typesense collection name to index data into (you need to create this collection in Typesense yourself. This extension does not create the Typesense Collection for you). | | Cloud Functions location | Where do you want to deploy the functions created for this extension? You usually want a location close to your database. For help selecting a location, refer to the [location selection guide](https://firebase.google.com/docs/functions/locations). | > ⚠ī¸ You'll notice that there is no way to configure the port number or protocol. -This is because this extension only supports connecting to Typesense running HTTPS on Port 443, since your data goes from Firebase to Typesense over the public internet and we want your data to be encrypted in transit. -For Typesense Cloud, HTTPS is already configured for you. -> +> This is because this extension only supports connecting to Typesense running HTTPS on Port 443, since your data goes from Firebase to Typesense over the public internet and we want your data to be encrypted in transit. +> For Typesense Cloud, HTTPS is already configured for you. +> > When self-hosting Typesense, you want to make sure you set `--api-port=443` and also get an SSL certificate from say [LetsEncrypt](https://letsencrypt.org/) or any registrar -and configure Typesense to use it using the `--ssl-certificate` and `--ssl-certificate-key` [server parameters](https://typesense.org/docs/latest/api/server-configuration.html). -> Alternatively, if you're running Typesense on your local machine, you can also set up a local HTTPS tunnel using something like [ngrok](https://ngrok.com/) (`ngrok http 8108`) and use the ngrok hostname in the extension. +> and configure Typesense to use it using the `--ssl-certificate` and `--ssl-certificate-key` [server parameters](https://typesense.org/docs/latest/api/server-configuration.html). +> Alternatively, if you're running Typesense on your local machine, you can also set up a local HTTPS tunnel using something like [ngrok](https://ngrok.com/) (`ngrok http 8108`) and use the ngrok hostname in the extension. ##### Example @@ -90,16 +90,15 @@ This will trigger the backfill background Cloud function, which will read data f ## ☁ī¸ Cloud Functions -* **indexToTypesenseOnFirestoreWrite:** A function that indexes data into Typesense when it's triggered by Firestore changes. - -* **backfillToTypesenseFromFirestore:** A function that backfills data from a Firestore collection into Typesense, triggered when a Firestore document with the path `typesense_sync/backfill` has the contents of `trigger: true`. +- **indexToTypesenseOnFirestoreWrite:** A function that indexes data into Typesense when it's triggered by Firestore changes. +- **backfillToTypesenseFromFirestore:** A function that backfills data from a Firestore collection into Typesense, triggered when a Firestore document with the path `typesense_sync/backfill` has the contents of `trigger: true`. ## 🔑 Access Required This extension will operate with the following project IAM roles: -* datastore.user (Reason: Required to backfill data from your Firestore collection into Typesense) +- datastore.user (Reason: Required to backfill data from your Firestore collection into Typesense) ## 🧾 Billing @@ -112,7 +111,6 @@ To install an extension, your project must be on the [Blaze (pay as you go) plan - Usage of this extension also requires you to have a running Typesense cluster either on Typesense Cloud or some self-hosted server. You are responsible for any associated costs with these services. - ## Development Workflow #### Run Emulator @@ -147,10 +145,9 @@ firebase ext:info ./ --markdown > README.md - Update version number in extension.yaml - Add entry to CHANGELOG.md - Create release in GitHub -- - ```shell - firebase ext:dev:upload typesense/firestore-typesense-search - ``` +- ```shell + firebase ext:dev:upload typesense/firestore-typesense-search + ``` ## ℹī¸ Support diff --git a/extension.yaml b/extension.yaml index e6ff725..d71be2e 100644 --- a/extension.yaml +++ b/extension.yaml @@ -73,6 +73,15 @@ params: example: field1,field2,field3 default: "" required: false + - param: TYPESENSE_FIELDS_RENAMES + label: Typesense Fields Renames + description: >- + A comma separated list of field renames in the format `old_field_name:new_field_name`. + This is useful when you want to rename fields from Firestore to a different name in Typesense. + Leave blank to not rename any fields. + example: field1:field_1,field2:field_2,field3:field_3 + default: "" + required: false - param: TYPESENSE_HOSTS label: Typesense Hosts description: >- @@ -98,7 +107,7 @@ params: - param: FLATTEN_NESTED_DOCUMENTS label: Flatten Nested Documents description: >- - Should nested documents in Firestore be flattened by this extension before they are indexed in Typesense? + Should nested documents in Firestore be flattened by this extension before they are indexed in Typesense? Set to "Yes" for Typesense versions 0.23.1 and earlier. Set to "No" for Typesense versions 0.24.0 and later. type: select options: diff --git a/extensions/firestore-typesense-search.env.local b/extensions/firestore-typesense-search.env.local index f808804..a1bfd17 100644 --- a/extensions/firestore-typesense-search.env.local +++ b/extensions/firestore-typesense-search.env.local @@ -1,6 +1,7 @@ LOCATION=us-central1 FIRESTORE_COLLECTION_PATH=books FIRESTORE_COLLECTION_FIELDS=author,title,rating,isAvailable,location,createdAt,nested_field,tags,nullField,ref +TYPESENSE_FIELDS_RENAMES=id:book_id FLATTEN_NESTED_DOCUMENTS=false TYPESENSE_HOSTS=localhost TYPESENSE_PORT=8108 diff --git a/extensions/test-params-flatten-nested-true.local.env b/extensions/test-params-flatten-nested-true.local.env index 75e7f6a..0e88332 100644 --- a/extensions/test-params-flatten-nested-true.local.env +++ b/extensions/test-params-flatten-nested-true.local.env @@ -1,6 +1,7 @@ LOCATION=us-central1 FIRESTORE_COLLECTION_PATH=books FIRESTORE_COLLECTION_FIELDS=author,title,rating,isAvailable,location,createdAt,nested_field,tags,nullField,ref +TYPESENSE_FIELDS_RENAMES=id:book_id FLATTEN_NESTED_DOCUMENTS=true TYPESENSE_HOSTS=localhost TYPESENSE_PORT=8108 diff --git a/extensions/test-params.example.env b/extensions/test-params.example.env index 749bebc..af0a09c 100644 --- a/extensions/test-params.example.env +++ b/extensions/test-params.example.env @@ -1,6 +1,7 @@ LOCATION=us-central1 FIRESTORE_COLLECTION_PATH=books FIRESTORE_COLLECTION_FIELDS=author,title +TYPESENSE_FIELDS_RENAMES=id:book_id FLATTEN_NESTED_DOCUMENTS=true TYPESENSE_HOSTS=xxx-1.a1.typesense.net TYPESENSE_COLLECTION_NAME=books diff --git a/functions/src/config.js b/functions/src/config.js index efa84ed..37a2c69 100644 --- a/functions/src/config.js +++ b/functions/src/config.js @@ -5,6 +5,15 @@ module.exports = { .split(",") .map((f) => f.trim()) .filter((f) => f), + typesenseFieldsRenames: (process.env.TYPESENSE_FIELDS_RENAMES || "") + .split(",") + .map((f) => f.trim()) + .filter((f) => f) + .map((f) => { + const [from, to] = f.split(":").map((f) => f.trim()); + return {from, to}; + }) + .filter((f) => f.from && f.to), shouldFlattenNestedDocuments: process.env.FLATTEN_NESTED_DOCUMENTS === "true", typesenseHosts: (process.env.TYPESENSE_HOSTS || "").split(",").map((e) => e.trim()), diff --git a/functions/src/utils.js b/functions/src/utils.js index 498cbf6..d129890 100644 --- a/functions/src/utils.js +++ b/functions/src/utils.js @@ -18,6 +18,11 @@ const mapValue = (value) => { } }; +const mapKey = (key) => { + const newKey = config.typesenseFieldsRenames[key] || key; + return newKey; +}; + /** * @param {DocumentSnapshot} firestoreDocumentSnapshot * @param {Array} fieldsToExtract @@ -37,7 +42,7 @@ exports.typesenseDocumentFromSnapshot = async ( } // Build a document with just the fields requested by the user, and mapped from Firestore types to Typesense types - const mappedDocument = Object.fromEntries(entries.map(([key, value]) => [key, mapValue(value)])); + const mappedDocument = Object.fromEntries(entries.map(([key, value]) => [mapKey(key), mapValue(value)])); // using flat to flatten nested objects for older versions of Typesense that did not support nested fields // https://typesense.org/docs/0.22.2/api/collections.html#indexing-nested-fields diff --git a/test/backfillToTypesenseFromFirestore.spec.js b/test/backfillToTypesenseFromFirestore.spec.js index d92aa16..c5e50e2 100644 --- a/test/backfillToTypesenseFromFirestore.spec.js +++ b/test/backfillToTypesenseFromFirestore.spec.js @@ -43,6 +43,7 @@ describe("backfillToTypesenseFromFirestore", () => { author: "Author A", title: "Title X", country: "USA", + id: "123", }; const firestoreDoc = await firestore.collection(config.firestoreCollectionPath).add(book); // Wait for firestore cloud function to write to Typesense @@ -76,6 +77,7 @@ describe("backfillToTypesenseFromFirestore", () => { id: firestoreDoc.id, author: book.author, title: book.title, + book_id: book.id, }); }); }); @@ -89,6 +91,7 @@ describe("backfillToTypesenseFromFirestore", () => { author: "Author A", title: "Title X", country: "USA", + id: "123", }; const firestoreDoc = await firestore.collection(config.firestoreCollectionPath).add(book); // Wait for firestore cloud function to write to Typesense @@ -125,6 +128,7 @@ describe("backfillToTypesenseFromFirestore", () => { id: firestoreDoc.id, author: book.author, title: book.title, + book_id: book.id, }); }); }); @@ -136,6 +140,7 @@ describe("backfillToTypesenseFromFirestore", () => { author: "Author A", title: "Title X", country: "USA", + id: "123", }; await firestore.collection(config.firestoreCollectionPath).add(book); // Wait for firestore cloud function to write to Typesense diff --git a/test/indexToTypesenseOnFirestoreWrite.spec.js b/test/indexToTypesenseOnFirestoreWrite.spec.js index 8a0f4b1..1f6123d 100644 --- a/test/indexToTypesenseOnFirestoreWrite.spec.js +++ b/test/indexToTypesenseOnFirestoreWrite.spec.js @@ -517,4 +517,39 @@ describe("indexToTypesenseOnFirestoreWrite", () => { expect(typesenseDocsStr).toBe(""); }); + + it("indexed fields are renamed to match the configured field name mapping", async () => { + const docData = {id: "123"}; + + // create document in Firestore + const docRef = await firestore.collection(config.firestoreCollectionPath).add(docData); + + // wait for the Firestore cloud function to write to Typesense + await new Promise((r) => setTimeout(r, 2500)); + + // check that the document was indexed + let typesenseDocsStr = await typesense + .collections(encodeURIComponent(config.typesenseCollectionName)) + .documents() + .export(); + const typesenseDocs = typesenseDocsStr.split("\n").map((s) => JSON.parse(s)); + + expect(typesenseDocs.length).toBe(1); + expect(typesenseDocs[0]).toStrictEqual({"id": docRef.id}); + expect(typesenseDocs[0]).toStrictEqual({"book_id": docData.id}); + + // delete document in Firestore + await docRef.delete(); + + // wait for the Firestore cloud function to write to Typesense + await new Promise((r) => setTimeout(r, 2500)); + + // check that the document was deleted + typesenseDocsStr = await typesense + .collections(encodeURIComponent(config.typesenseCollectionName)) + .documents() + .export(); + + expect(typesenseDocsStr).toBe(""); + }); });