Implement Tests for Hallucinations for Virtual Assistant Demo

[riacheruvu]

Description

Implement a suite of unit tests to check if the output of the Large Language Model (LLM) used in the Virtual Assistant Demo involves hallucinations. This is a great opportunity to explore techniques on ensuring the reliability and accuracy of the assistant's responses by identifying and mitigating instances where the LLM generates incorrect or misleading information.

This project will involve Python programming; basic experience with AI models from frameworks like PyTorch, TensorFlow, OpenVINO, or ONNX is beneficial. To learn more about the OpenVINO toolkit, visit the documentation here.

Examples of hallucinations:

• Factual errors
• Inaccurately summarizing information
• Creating nonsensical content, such as random sentences (which can also manifest in the LLM trailing off in the middle of a response to a tangential topic)

Steps:

1. Develop unit tests for Hallucination Detection:

   • Define criteria for what constitutes a hallucination and implement checks based on these criteria.
   • Create unit tests that analyze the output of the LLM used in the Virtual Assistant Demo to detect hallucinations.

2. Enhance Testing Framework:

   • Integrate the new unit tests into the existing testing framework.
   • Ensure the tests are comprehensive and cover various scenarios where hallucinations might occur.

3. Documentation:

   • Provide clear and detailed documentation for setting up and running the unit tests.
   • Include comments in the code to explain key sections and logic.

How to Get Started:

1. Fork the OpenVINO Build Deploy repository.
2. Follow these installation instructions to setup your environment and install the required dependencies for this project.
3. Read Demo Contribution Guide.
4. Build your feature and ensure it meets the requirements specified in the contribution guide.
5. Submit a pull request.

[jnzw]

Team 3 is working on this issue.

[github-actions]

This issue has been marked because it has been open for 30 days with no activity. It is scheduled to close in 14 days.

[github-actions]

This issue has been marked because it has been open for 30 days with no activity. It is scheduled to close in 14 days.

[github-actions]

This issue was closed automatically because it has been inactive for 14 days since being marked as stale. Please reopen if needed.