VES - Vector Enhanced Search (for logs)

⚠️ Status: Pre-MVP. Expect breaking changes, incomplete features, and rapid iteration.

Kubernetes-native semantic log search for modern DevOps teams. Plug into existing log collection setups or run standalone, powered by an embedding sidecar + ChromaDB.

---

Why VES?

Traditional log search (ELK, Loki, Splunk, etc.) is keyword-based - you only find exact matches. But logs often express the same issue in many different ways:

• "NullPointerException"
• "object reference not set"
• "NoneType error"

By embedding logs into vector space, VES enables semantic search: "Find logs like this one" instead of rigid keyword matching.

---

🚀 Features

• 🔍 Semantic log search across services
• 📋 “Find similar logs” by pasting a stack trace
• 🧩 Cluster logs by similarity → spot recurring issues faster
• ⚡ Enrich alerts with past related logs
• 🛠️ Kubernetes-native, integrates with existing Loki/ELK stacks
• 🧑‍💻 Optional standalone mode (no Loki required)

---

🏗️ System Architecture

┌──────────────┐
│ Applications │
└───────┬──────┘
        │
┌──────▼────────┐
│ Log Collector │
└──────┬────────┘
       │
┌───────▼────────┐
│ Log Storage    │
└───────┬────────┘
        │
┌───────▼────────┐
│ Embedding Svc  │
└───────┬────────┘
        │
┌───────▼────────┐
│ ChromaDB       │
└────────────────┘

---

🔧 Components

1. Log Collector

   • Two modes:
     • Bootstrapped (default) → assumes a log-collection tool are already running
     • Automated → ships with log-collector microservice DaemonSet YAML for direct embedding ingestion
   • Runs as DaemonSets in Kubernetes
   • Forwards logs to both Loki/ELK and the Embedding Sidecar

2. Embedding Sidecar

   • Converts raw logs into embeddings (e.g. sentence-transformers/all-MiniLM-L6-v2)
   • Packages: log + metadata + embedding → sends to ChromaDB

   Example metadata:

   {
       "id": "log123",
       "embedding": [...],
       "metadata": {
           "service": "payment-api",
           "timestamp": "2025-08-28T12:00:00Z",
           "log_level": "ERROR",
           "raw_log": "NullPointerException at line 42"
       }
   }

3. ChromaDB Service

   • Vector database for log embedding storage
   • Runs as a StatefulSet in Kubernetes with persistent storage

4. API Service

   • Accepts log queries, converts to embedding, searched ChromaDB
   • Returns: raw log, timestamp, service, severity
   • (Coming Soon): If log-collector running in bootstrapped mode, deep-links back into the bootstrapped log-collector

5. Web UI (Coming Soon)

   • Minimal React frontend
   • Paste logs / stack traces -> get "similar logs"
   • Click through to bootstrapped log-collector UI

---

📦 Project Modes

1. Bootstrapped -> for teams already using a log-collector
2. Automated -> for teams without any log infrastructure(RECOMMENDED)

---

🚀 Getting Started

# clone the repo
git clone https://github.com/H3IMD3LL-Labs-Inc/VES-Vector-Enhanced-Search-.git
cd VES-Vector-Enhanced-Search-.git

# (Coming Soon) Install via Helm or kubectl
kubectl apply -f deploy/ves.yaml

---

🤝 Contributing

Contributors are welcome! Please see our upcoming CONTRIBUTING and CODE_OF_CONDUCT.

This is currently a v0.1 hackable infra experiment — long-term goal(afew months): ➡️ production-ready semantic log search for DevOps teams.

---

📜 License

This project is licensed under the Business Source License 1.1.

• Licensor: H3IMD3LL Labs, Inc.
• Change Date: 5 years from publication
• Change License: MPL 2.0

---

📬 Community & Support

• Email
• Website
• Discord(Coming Soon)

---

📍 Roadmap

• v0.0.0 - Pre-MVP: dev logs, README, roadmap, discussions.

• v0.0.1 - MVP: log-collector, embedding-service, API-Service, Web UI

  • Log Collector
  • Embedding Service
  • API Service

• v0.1.0 - Improvements: Documentation and Stability

  • Documentation
  • Stability

• v1.0.0 - Production-ready release: Further Documentation, Stability improvements and User Support

  • Further Documentation
  • Stability improvements
  • User Support

---

Built with ❤️ for A1m4 N4d1n3 a.k.a miss ma'am :)