Merge branch 'master' into mdrxy/anthropic-web-fetch

Author: mdrxy

cookbook/README.md

@@ -64,3 +64,4 @@ Notebook | Description
 [visual_RAG_vdms.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/visual_RAG_vdms.ipynb) | Performs Visual Retrieval-Augmented-Generation (RAG) using videos and scene descriptions generated by open source models.
 [contextual_rag.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/contextual_rag.ipynb) | Performs contextual retrieval-augmented generation (RAG) prepending chunk-specific explanatory context to each chunk before embedding.
 [rag-agents-locally-on-intel-cpu.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/local_rag_agents_intel_cpu.ipynb) | Build a RAG agent locally with open source models that routes questions through one of two paths to find answers. The agent generates answers based on documents retrieved from either the vector database or retrieved from web search. If the vector database lacks relevant information, the agent opts for web search. Open-source models for LLM and embeddings are used locally on an Intel Xeon CPU to execute this pipeline.
+[rag_mlflow_tracking_evaluation.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/rag_mlflow_tracking_evaluation.ipynb) | Guide on how to create a RAG pipeline and track + evaluate it with MLflow.

cookbook/rag_mlflow_tracking_evaluation.ipynb

@@ -61,7 +61,7 @@
     "   * document addition by id (`add_documents` method with `ids` argument)\n",
     "   * delete by id (`delete` method with `ids` argument)\n",
     "\n",
-    "Compatible Vectorstores: `Aerospike`, `AnalyticDB`, `AstraDB`, `AwaDB`, `AzureCosmosDBNoSqlVectorSearch`, `AzureCosmosDBVectorSearch`, `AzureSearch`, `Bagel`, `Cassandra`, `Chroma`, `CouchbaseVectorStore`, `DashVector`, `DatabricksVectorSearch`, `DeepLake`, `Dingo`, `ElasticVectorSearch`, `ElasticsearchStore`, `FAISS`, `HanaDB`, `Milvus`, `MongoDBAtlasVectorSearch`, `MyScale`, `OpenSearchVectorSearch`, `PGVector`, `Pinecone`, `Qdrant`, `Redis`, `Rockset`, `ScaNN`, `SingleStoreDB`, `SupabaseVectorStore`, `SurrealDBStore`, `TimescaleVector`, `Vald`, `VDMS`, `Vearch`, `VespaStore`, `Weaviate`, `Yellowbrick`, `ZepVectorStore`, `TencentVectorDB`, `OpenSearchVectorSearch`.\n",
+    "Compatible Vectorstores: `AnalyticDB`, `AstraDB`, `AwaDB`, `AzureCosmosDBNoSqlVectorSearch`, `AzureCosmosDBVectorSearch`, `AzureSearch`, `Bagel`, `Cassandra`, `Chroma`, `CouchbaseVectorStore`, `DashVector`, `DatabricksVectorSearch`, `DeepLake`, `Dingo`, `ElasticVectorSearch`, `ElasticsearchStore`, `FAISS`, `HanaDB`, `Milvus`, `MongoDBAtlasVectorSearch`, `MyScale`, `OpenSearchVectorSearch`, `PGVector`, `Pinecone`, `Qdrant`, `Redis`, `Rockset`, `ScaNN`, `SingleStoreDB`, `SupabaseVectorStore`, `SurrealDBStore`, `TimescaleVector`, `Vald`, `VDMS`, `Vearch`, `VespaStore`, `Weaviate`, `Yellowbrick`, `ZepVectorStore`, `TencentVectorDB`, `OpenSearchVectorSearch`.\n",
     "  \n",
     "## Caution\n",
     "\n",

docs/api_reference/guide_imports.json

@@ -7,10 +7,7 @@
    "source": [
     "# Confident\n",
     "\n",
-    ">[DeepEval](https://confident-ai.com) package for unit testing LLMs.\n",
-    "> Using Confident, everyone can build robust language models through faster iterations\n",
-    "> using both unit testing and integration testing. We provide support for each step in the iteration\n",
-    "> from synthetic data creation to testing.\n"
+    ">[DeepEval](https://confident-ai.com) package for unit testing LLMs."
    ]
   },
   {
@@ -42,7 +39,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "%pip install --upgrade --quiet  langchain langchain-openai langchain-community deepeval langchain-chroma"
+    "!pip install deepeval langchain langchain-openai"
    ]
   },
   {
@@ -62,149 +59,41 @@
     "When you log in, you will also be asked to set the `implementation` name. The implementation name is required to describe the type of implementation. (Think of what you want to call your project. We recommend making it descriptive.)"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "!deepeval login"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Setup DeepEval\n",
-    "\n",
-    "You can, by default, use the `DeepEvalCallbackHandler` to set up the metrics you want to track. However, this has limited support for metrics at the moment (more to be added soon). It currently supports:\n",
-    "- [Answer Relevancy](https://docs.confident-ai.com/docs/measuring_llm_performance/answer_relevancy)\n",
-    "- [Bias](https://docs.confident-ai.com/docs/measuring_llm_performance/debias)\n",
-    "- [Toxicness](https://docs.confident-ai.com/docs/measuring_llm_performance/non_toxic)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from deepeval.metrics.answer_relevancy import AnswerRelevancy\n",
-    "\n",
-    "# Here we want to make sure the answer is minimally relevant\n",
-    "answer_relevancy_metric = AnswerRelevancy(minimum_score=0.5)"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Get Started"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "To use the `DeepEvalCallbackHandler`, we need the `implementation_name`. "
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain_community.callbacks.confident_callback import DeepEvalCallbackHandler\n",
-    "\n",
-    "deepeval_callback = DeepEvalCallbackHandler(\n",
-    "    implementation_name=\"langchainQuickstart\", metrics=[answer_relevancy_metric]\n",
-    ")"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Scenario 1: Feeding into LLM\n",
-    "\n",
-    "You can then feed it into your LLM with OpenAI."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {},
    "outputs": [
     {
      "data": {
+      "text/html": [
+       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">🎉🥳 Congratulations! You've successfully logged in! 🙌 \n",
+       "</pre>\n"
+      ],
       "text/plain": [
-       "LLMResult(generations=[[Generation(text='\\n\\nQ: What did the fish say when he hit the wall? \\nA: Dam.', generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text='\\n\\nThe Moon \\n\\nThe moon is high in the midnight sky,\\nSparkling like a star above.\\nThe night so peaceful, so serene,\\nFilling up the air with love.\\n\\nEver changing and renewing,\\nA never-ending light of grace.\\nThe moon remains a constant view,\\nA reminder of life’s gentle pace.\\n\\nThrough time and space it guides us on,\\nA never-fading beacon of hope.\\nThe moon shines down on us all,\\nAs it continues to rise and elope.', generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text='\\n\\nQ. What did one magnet say to the other magnet?\\nA. \"I find you very attractive!\"', generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text=\"\\n\\nThe world is charged with the grandeur of God.\\nIt will flame out, like shining from shook foil;\\nIt gathers to a greatness, like the ooze of oil\\nCrushed. Why do men then now not reck his rod?\\n\\nGenerations have trod, have trod, have trod;\\nAnd all is seared with trade; bleared, smeared with toil;\\nAnd wears man's smudge and shares man's smell: the soil\\nIs bare now, nor can foot feel, being shod.\\n\\nAnd for all this, nature is never spent;\\nThere lives the dearest freshness deep down things;\\nAnd though the last lights off the black West went\\nOh, morning, at the brown brink eastward, springs —\\n\\nBecause the Holy Ghost over the bent\\nWorld broods with warm breast and with ah! bright wings.\\n\\n~Gerard Manley Hopkins\", generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text='\\n\\nQ: What did one ocean say to the other ocean?\\nA: Nothing, they just waved.', generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text=\"\\n\\nA poem for you\\n\\nOn a field of green\\n\\nThe sky so blue\\n\\nA gentle breeze, the sun above\\n\\nA beautiful world, for us to love\\n\\nLife is a journey, full of surprise\\n\\nFull of joy and full of surprise\\n\\nBe brave and take small steps\\n\\nThe future will be revealed with depth\\n\\nIn the morning, when dawn arrives\\n\\nA fresh start, no reason to hide\\n\\nSomewhere down the road, there's a heart that beats\\n\\nBelieve in yourself, you'll always succeed.\", generation_info={'finish_reason': 'stop', 'logprobs': None})]], llm_output={'token_usage': {'completion_tokens': 504, 'total_tokens': 528, 'prompt_tokens': 24}, 'model_name': 'text-davinci-003'})"
+       "🎉🥳 Congratulations! You've successfully logged in! 🙌 \n"
       ]
      },
-     "execution_count": 7,
      "metadata": {},
-     "output_type": "execute_result"
+     "output_type": "display_data"
     }
    ],
    "source": [
-    "from langchain_openai import OpenAI\n",
+    "import os\n",
+    "import deepeval\n",
     "\n",
-    "llm = OpenAI(\n",
-    "    temperature=0,\n",
-    "    callbacks=[deepeval_callback],\n",
-    "    verbose=True,\n",
-    "    openai_api_key=\"<YOUR_API_KEY>\",\n",
-    ")\n",
-    "output = llm.generate(\n",
-    "    [\n",
-    "        \"What is the best evaluation tool out there? (no bias at all)\",\n",
-    "    ]\n",
-    ")"
+    "api_key = os.getenv(\"DEEPEVAL_API_KEY\")\n",
+    "deepeval.login(api_key)"
    ]
   },
   {
    "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can then check the metric if it was successful by calling the `is_successful()` method."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "answer_relevancy_metric.is_successful()\n",
-    "# returns True/False"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Once you have ran that, you should be able to see our dashboard below. \n",
-    "\n",
-    "![Dashboard](https://docs.confident-ai.com/assets/images/dashboard-screenshot-b02db73008213a211b1158ff052d969e.png)"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Scenario 2: Tracking an LLM in a chain without callbacks\n",
-    "\n",
-    "To track an LLM in a chain without callbacks, you can plug into it at the end.\n",
+    "### Setup Confident AI Callback (Modern)\n",
     "\n",
-    "We can start by defining a simple chain as shown below."
+    "The previous DeepEvalCallbackHandler and metric tracking are deprecated. Please use the new integration below."
    ]
   },
   {
@@ -213,76 +102,29 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import requests\n",
-    "from langchain.chains import RetrievalQA\n",
-    "from langchain_chroma import Chroma\n",
-    "from langchain_community.document_loaders import TextLoader\n",
-    "from langchain_openai import OpenAI, OpenAIEmbeddings\n",
-    "from langchain_text_splitters import CharacterTextSplitter\n",
+    "from deepeval.integrations.langchain import CallbackHandler\n",
     "\n",
-    "text_file_url = \"https://raw.githubusercontent.com/hwchase17/chat-your-data/master/state_of_the_union.txt\"\n",
-    "\n",
-    "openai_api_key = \"sk-XXX\"\n",
-    "\n",
-    "with open(\"state_of_the_union.txt\", \"w\") as f:\n",
-    "    response = requests.get(text_file_url)\n",
-    "    f.write(response.text)\n",
-    "\n",
-    "loader = TextLoader(\"state_of_the_union.txt\")\n",
-    "documents = loader.load()\n",
-    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
-    "texts = text_splitter.split_documents(documents)\n",
-    "\n",
-    "embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key)\n",
-    "docsearch = Chroma.from_documents(texts, embeddings)\n",
-    "\n",
-    "qa = RetrievalQA.from_chain_type(\n",
-    "    llm=OpenAI(openai_api_key=openai_api_key),\n",
-    "    chain_type=\"stuff\",\n",
-    "    retriever=docsearch.as_retriever(),\n",
-    ")\n",
-    "\n",
-    "# Providing a new question-answering pipeline\n",
-    "query = \"Who is the president?\"\n",
-    "result = qa.run(query)"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "After defining a chain, you can then manually check for answer similarity."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "answer_relevancy_metric.measure(result, query)\n",
-    "answer_relevancy_metric.is_successful()"
+    "handler = CallbackHandler(\n",
+    "    name=\"My Trace\",\n",
+    "    tags=[\"production\", \"v1\"],\n",
+    "    metadata={\"experiment\": \"A/B\"},\n",
+    "    thread_id=\"thread-123\",\n",
+    "    user_id=\"user-456\",\n",
+    ")"
    ]
   },
   {
    "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### What's next?\n",
-    "\n",
-    "You can create your own custom metrics [here](https://docs.confident-ai.com/docs/quickstart/custom-metrics). \n",
-    "\n",
-    "DeepEval also offers other features such as being able to [automatically create unit tests](https://docs.confident-ai.com/docs/quickstart/synthetic-data-creation), [tests for hallucination](https://docs.confident-ai.com/docs/measuring_llm_performance/factual_consistency).\n",
-    "\n",
-    "If you are interested, check out our Github repository here [https://github.com/confident-ai/deepeval](https://github.com/confident-ai/deepeval). We welcome any PRs and discussions on how to improve LLM performance."
+    "## Get Started"
    ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "langchain",
    "language": "python",
    "name": "python3"
   },
@@ -296,12 +138,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
-  },
-  "vscode": {
-   "interpreter": {
-    "hash": "a53ebf4a859167383b364e7e7521d0add3c2dbbdecce4edf676e8c4634ff3fbb"
-   }
+   "version": "3.12.11"
   }
  },
  "nbformat": 4,

docs/docs/how_to/indexing.ipynb

@@ -21,6 +21,38 @@ pip install deepeval
 
 See an [example](/docs/integrations/callbacks/confident).
 
+
+## Modern Integration Example
+
+Install the required packages:
+
+```bash
+pip install deepeval langchain langchain-openai
+```
+
+Authenticate with your API key:
+
 ```python
-from langchain.callbacks.confident_callback import DeepEvalCallbackHandler
+import os
+import deepeval
+
+# Load API key from environment variable for security
+api_key = os.environ.get("DEEPEVAL_API_KEY")
+deepeval.login(api_key)
+```
+
+Use the new callback handler:
+
+```python
+from deepeval.integrations.langchain import CallbackHandler
+
+handler = CallbackHandler(
+    name="My Trace",
+    tags=["production", "v1"],
+    metadata={"experiment": "A/B"},
+    thread_id="thread-123",
+    user_id="user-456"
+)
 ```
+
+See the [full example](/docs/integrations/callbacks/confident).

docs/docs/integrations/callbacks/confident.ipynb

@@ -77,7 +77,7 @@ from langchain_ibm import WatsonxRerank
 See a [usage example](/docs/integrations/tools/ibm_watsonx).
 
 ```python
-from langchain_ibm import WatsonxToolkit
+from langchain_ibm.agent_toolkits.utility import WatsonxToolkit
 ```
 
 ## DB2

docs/docs/integrations/providers/aerospike.mdx

@@ -40,11 +40,11 @@ embeddings.embed_query("What is the meaning of life?")
 ```
 
 ## LLMs
-`ModelScopeLLM` class exposes LLMs from ModelScope.
+`ModelScopeEndpoint` class exposes LLMs from ModelScope.
 
 ```python
-from langchain_modelscope import ModelScopeLLM
+from langchain_modelscope import ModelScopeEndpoint
 
-llm = ModelScopeLLM(model="Qwen/Qwen2.5-Coder-32B-Instruct")
+llm = ModelScopeEndpoint(model="Qwen/Qwen2.5-Coder-32B-Instruct")
 llm.invoke("The meaning of life is")
 ```