Add example env file for text2sql (#142)

Author: BenConstable9

text_2_sql/.env.example

@@ -0,0 +1,36 @@
+# Environment variables for Text2SQL
+IdentityType=<identityType> # system_assigned or user_assigned or key
+
+# Open AI Connection Details
+OpenAI__CompletionDeployment=<openAICompletionDeploymentId. Used for data dictionary creator>
+OpenAI__MiniCompletionDeployment=<OpenAI__MiniCompletionDeploymentId. Used for agentic text2sql>
+OpenAI__Endpoint=<openAIEndpoint>
+OpenAI__ApiKey=<openAIKey if using non identity based connection>
+OpenAI__ApiVersion=<openAIApiVersion>
+
+# Azure AI Search Connection Details
+AIService__AzureSearchOptions__Endpoint=<AI search endpoint>
+AIService__AzureSearchOptions__Key=<AI search key if using non identity based connection>
+AIService__AzureSearchOptions__Text2SqlSchemaStore__Index=<Schema store index name. Default is created as "text-2-sql-schema-store-index">
+AIService__AzureSearchOptions__Text2SqlSchemaStore__SemanticConfig=<Schema store semantic config. Default is created as "text-2-sql-schema-store-semantic-config">
+AIService__AzureSearchOptions__Text2SqlQueryCache__Index=<Query cache index name. Default is created as "text-2-sql-query-cache-index">
+AIService__AzureSearchOptions__Text2SqlQueryCache__SemanticConfig=<Query cache semantic config. Default is created as "text-2-sql-query-cache-semantic-config">
+AIService__AzureSearchOptions__Text2SqlColumnValueStore__Index=<Column value store index name. Default is created as "text-2-sql-column-value-store-index">
+
+# All SQL Engine specific connection details
+Text2Sql__DatabaseName=<databaseName>
+
+# TSQL or PostgreSQL Specific Connection Details
+Text2Sql__DatabaseConnectionString=<databaseConnectionString>
+
+# Snowflake Specific Connection Details
+Text2Sql__Snowflake__User=<snowflakeUser if using Snowflake Data Source>
+Text2Sql__Snowflake__Password=<snowflakePassword if using Snowflake Data Source>
+Text2Sql__Snowflake__Account=<snowflakeAccount if using Snowflake Data Source>
+Text2Sql__Snowflake__Warehouse=<snowflakeWarehouse if using Snowflake Data Source>
+
+# Databricks Specific Connection Details
+Text2Sql__Databricks__Catalog=<databricksCatalog if using Databricks Data Source with Unity Catalog>
+Text2Sql__Databricks__ServerHostname=<databricksServerHostname if using Databricks Data Source with Unity Catalog>
+Text2Sql__Databricks__HttpPath=<databricksHttpPath if using Databricks Data Source with Unity Catalog>
+Text2Sql__Databricks__AccessToken=<databricks AccessToken if using Databricks Data Source with Unity Catalog>

text_2_sql/GETTING_STARTED.md

@@ -5,7 +5,7 @@ To get started, perform the following steps:
 1. Setup Azure OpenAI in your subscription with **gpt-4o-mini** & an embedding model, alongside a SQL Server sample database, AI Search and a storage account.
 2. Clone this repository and deploy the AI Search text2sql indexes from `deploy_ai_search`.
 3. Run `uv sync` within the text_2_sql directory to install dependencies.
-4. Configure the .env file based on the provided sample
-5. Generate a data dictionary for your target server using the instructions in `data_dictionary`.
-6. Upload these data dictionaries to the relevant contains in your storage account. Wait for them to be automatically indexed.
+4. Create your `.env` file based on the provided sample `.env.example`. Place this file in the same place as the `.env.example`.
+5. Generate a data dictionary for your target server using the instructions in the **Running** section of the `data_dictionary/README.md`.
+6. Upload these data dictionaries to the relevant containers in your storage account. Wait for them to be automatically indexed with the included skillsets.
 7. Navigate to `autogen` directory to view the AutoGen implementation. Follow the steps in `Iteration 5 - Agentic Vector Based Text2SQL.ipynb` to get started.

text_2_sql/README.md

@@ -54,7 +54,20 @@ As the query cache is shared between users (no data is stored in the cache), a n
 
 ![Vector Based with Query Cache Logical Flow.](./images/Agentic%20Text2SQL%20Query%20Cache.png "Agentic Vector Based with Query Cache Logical Flow")
 
-#### Parallel execution
+## Agents
+
+This agentic system contains the following agents:
+
+- **Query Cache Agent:** Responsible for checking the cache for previously asked questions.
+- **Query Decomposition Agent:** Responsible for decomposing complex questions, into sub questions that can be answered with SQL.
+- **Schema Selection Agent:** Responsible for extracting key terms from the question and checking the index store for the queries.
+- **SQL Query Generation Agent:** Responsible for using the previously extracted schemas and generated SQL queries to answer the question. This agent can request more schemas if needed. This agent will run the query.
+- **SQL Query Verification Agent:** Responsible for verifying that the SQL query and results question will answer the question.
+- **Answer Generation Agent:** Responsible for taking the database results and generating the final answer for the user.
+
+The combination of this agent allows the system to answer complex questions, whilst staying under the token limits when including the database schemas. The query cache ensures that previously asked questions, can be answered quickly to avoid degrading user experience.
+
+### Parallel execution
 
 After the first agent has rewritten and decomposed the user input, we execute each of the individual questions in parallel for the quickest time to generate an answer.
 
@@ -189,22 +202,9 @@ Below is a sample entry for a view / table that we which to expose to the LLM. T
 }
 ```
 
-See `./data_dictionary` for more details on how the data dictionary is structured and ways to **automatically generate it**.
-
-## Agentic Vector Based Approach (Iteration 5)
-
-This approach builds on the the Vector Based SQL Plugin approach that was previously developed, but adds a agentic approach to the solution.
-
-This agentic system contains the following agents:
-
-- **Query Cache Agent:** Responsible for checking the cache for previously asked questions.
-- **Query Decomposition Agent:** Responsible for decomposing complex questions, into sub questions that can be answered with SQL.
-- **Schema Selection Agent:** Responsible for extracting key terms from the question and checking the index store for the queries.
-- **SQL Query Generation Agent:** Responsible for using the previously extracted schemas and generated SQL queries to answer the question. This agent can request more schemas if needed. This agent will run the query.
-- **SQL Query Verification Agent:** Responsible for verifying that the SQL query and results question will answer the question.
-- **Answer Generation Agent:** Responsible for taking the database results and generating the final answer for the user.
-
-The combination of this agent allows the system to answer complex questions, whilst staying under the token limits when including the database schemas. The query cache ensures that previously asked questions, can be answered quickly to avoid degrading user experience.
+> [!NOTE]
+>
+> - See `./data_dictionary` for more details on how the data dictionary is structured and ways to **automatically generate it**.
 
 ## Tips for good Text2SQL performance.
 

text_2_sql/autogen/pyproject.toml

@@ -41,3 +41,6 @@ databricks = [
 postgresql = [
     "text_2_sql_core[postgresql]",
 ]
+sqlite = [
+    "text_2_sql_core[sqlite]",
+]

text_2_sql/data_dictionary/.env

@@ -74,20 +74,20 @@ async def run_completion_request(
         else:
             return message.content
 
-    async def run_embedding_request(self, batch: list[str]):
-        token_provider, api_key = self.get_authentication_properties()
+    # async def run_embedding_request(self, batch: list[str]):
+    #     token_provider, api_key = self.get_authentication_properties()
 
-        model_deployment = os.environ["OpenAI__EmbeddingModel"]
-        async with AsyncAzureOpenAI(
-            azure_deployment=model_deployment,
-            api_version=os.environ["OpenAI__ApiVersion"],
-            azure_endpoint=os.environ["OpenAI__Endpoint"],
-            azure_ad_token_provider=token_provider,
-            api_key=api_key,
-        ) as open_ai_client:
-            embeddings = await open_ai_client.embeddings.create(
-                model=os.environ["OpenAI__EmbeddingModel"],
-                input=batch,
-            )
+    #     model_deployment = os.environ["OpenAI__EmbeddingModel"]
+    #     async with AsyncAzureOpenAI(
+    #         azure_deployment=model_deployment,
+    #         api_version=os.environ["OpenAI__ApiVersion"],
+    #         azure_endpoint=os.environ["OpenAI__Endpoint"],
+    #         azure_ad_token_provider=token_provider,
+    #         api_key=api_key,
+    #     ) as open_ai_client:
+    #         embeddings = await open_ai_client.embeddings.create(
+    #             model=os.environ["OpenAI__EmbeddingModel"],
+    #             input=batch,
+    #         )
 
-        return embeddings
+    #     return embeddings