VisionFlow: AI Agent Visualization & Knowledge Graph Explorer

Real-time 3D visualization of AI agent swarms working alongside your Logseq knowledge graph.

Inspired by the innovative work of Prof. Rob Aspin: https://github.com/trebornipsa

A Note on AI-Driven Development

This project was built over several months by a non-programmer, using AI agents as the primary tool for coding, debugging, and architectural design. This AI-first approach has shaped not only how the project was built but also the docs and diagrams you see here.

AI in the Development Process

The entire codebase, from the Rust backend with Actix to the TypeScript frontend with React and Three.js, was generated through an iterative, conversational process with AI agents. This workflow allowed for rapid prototyping, complex feature implementation, and exploration of technologies that would typically be beyond the reach of a solo, non-developer founder.

Integrated AI Features

Beyond its creation, LogseqXR integrates several cutting-edge AI services to enhance the user's interaction with their knowledge graph. These services are orchestrated by the Rust backend and made accessible through a dedicated conversational UI on the client.

graph TD
    subgraph Client
        UI["ConversationPane (React)"]
    end

    subgraph "Backend (Rust)"
        APIServer["API Server (Actix)"]
        RAGFlowSvc["RAGFlowService.rs"]
        PerplexitySvc["PerplexityService.rs"]
        SpeechSvc["SpeechService.rs"]
    end

    subgraph "External AI APIs"
        RAGFlowAPI["RAGFlow API"]
        PerplexityAPI["Perplexity API"]
        OpenAI_API["OpenAI API (TTS/STT)"]
        KokoroAPI_Ext["Kokoro API (TTS)"]
    end

    UI -->|REST API Call| APIServer
    APIServer --> RAGFlowSvc
    APIServer --> PerplexitySvc
    APIServer --> SpeechSvc

    RAGFlowSvc --> RAGFlowAPI
    PerplexitySvc --> PerplexityAPI
    SpeechSvc --> OpenAI_API
    SpeechSvc --> KokoroAPI_Ext

    OpenAI_API -->|Audio/Text| SpeechSvc
    KokoroAPI_Ext -->|Audio| SpeechSvc
    RAGFlowAPI -->|Text| RAGFlowSvc
    PerplexityAPI -->|Text| PerplexitySvc

    RAGFlowSvc -->|Response| APIServer
    PerplexitySvc -->|Response| APIServer
    SpeechSvc -->|Response| APIServer

    APIServer -->|Response| UI

Conversational Q&A: The ConversationPane.tsx allows users to chat with their knowledge graph. Backend services like RAGFlowService.rs and PerplexityService.rs process these queries to provide context-aware answers.
Voice Interaction: The SpeechService.rs integrates with OpenAI and Kokoro for high-quality Text-to-Speech (TTS) and Speech-to-Text (STT), enabling users to speak to their graph and hear responses.

Key Features

🤖 AI Agent Swarm Visualization

Real-time Hive Mind Display: Watch AI agents collaborate in stunning 3D
Claude Flow Integration: Full MCP (Model Context Protocol) support for agent orchestration
GPU-Accelerated Physics: Smooth 60 FPS visualization of 200+ agents
Interactive Control: Spawn, configure, and command agent swarms in real-time

📊 Dual Graph Architecture

Parallel Visualization: View Logseq knowledge graphs and AI agent swarms simultaneously
Independent Physics: Each graph has its own optimized physics engine
Binary Protocol: 85% bandwidth reduction for massive scalability

🚀 Production-Ready Architecture

Rust Backend: High-performance Actix-based server with actor model
React/Three.js Frontend: Immersive 3D visualization with WebXR support
Docker Deployment: One-command deployment with full MCP integration
Enterprise Features: Authentication, monitoring, and horizontal scaling

Quick Start

# Clone and deploy in minutes
git clone <repository-url>
cd ext

# Configure environment
cp .env.example .env

# Deploy with Docker
docker-compose up -d

# Access the application
open http://localhost:3001

Full Quick Start Guide - Spawn your first AI swarm in 5 minutes!

Documentation Hub

📚 Complete Documentation - Comprehensive guide to all features
🏗️ Architecture Overview - System design and components
🚀 Docker Deployment - Production deployment guide
🔌 API Reference - REST and WebSocket APIs
🤝 Contributing Guide - Join the development

What's New in VisionFlow

AI Agent Swarm Features

Hive Mind Spawning: Create coordinated AI agent swarms with different topologies (mesh, hierarchical, ring, star)
Real-time Monitoring: Track agent performance, task completion, and communication patterns
MCP Tool Integration: 70+ tools including task orchestration, memory management, and neural training
Agent Types: 12 specialized agents (Queen, Coordinator, Researcher, Coder, Analyst, etc.)

Technical Achievements

Binary Protocol: 28-byte format reduces bandwidth by 85% compared to JSON
GPU Physics: Process 200+ agents at 60 FPS with GPU-accelerated spring physics
Actor Model: Rust backend using Actix actors for safe concurrent state management
TCP Bridge: Clean separation between UI server and agent execution environment

System Architecture Diagram

graph TD
    subgraph ClientApp ["Frontend"]
        direction LR
        AppInit["AppInitializer"]
        TwoPane["TwoPaneLayout"]
        GraphView["GraphViewport <br/> 3D Scene Container"]
        GraphCanvas["GraphCanvas <br/> Three.js Canvas"]
        RightCtlPanel["RightPaneControlPanel"]
        SettingsUI["SettingsPanelRedesignOptimized"]
        ConvoPane["ConversationPane"]
        NarrativePane["NarrativeGoldminePanel"]
        SettingsMgr["settingsStore"]
        GraphDataMgr["GraphDataManager"]
        RenderEngine["GraphCanvas & GraphManager"]
        WebSocketSvc["WebSocketService"]
        APISvc["api"]
        NostrAuthSvcClient["nostrAuthService"]
        XRController["XRController"]

        AppInit --> TwoPane
        AppInit --> SettingsMgr
        AppInit --> NostrAuthSvcClient
        AppInit --> WebSocketSvc
        AppInit --> GraphDataMgr

        TwoPane --> GraphView
        TwoPane --> RightCtlPanel
        TwoPane --> ConvoPane
        TwoPane --> NarrativePane
        RightCtlPanel --> SettingsUI

        SettingsUI --> SettingsMgr
        GraphView --> RenderEngine
        RenderEngine <--> GraphDataMgr
        GraphDataMgr <--> WebSocketSvc
        GraphDataMgr <--> APISvc
        NostrAuthSvcClient <--> APISvc
        XRController <--> RenderEngine
        XRController <--> SettingsMgr
    end

    subgraph ServerApp ["Backend"]
        direction LR
        Actix["ActixWebServer"]

        subgraph Handlers_Srv ["API & WebSocket Handlers"]
            direction TB
            SettingsH["SettingsHandler"]
            NostrAuthH["NostrAuthHandler"]
            GraphAPI_H["GraphAPIHandler"]
            FilesAPI_H["FilesAPIHandler"]
            RAGFlowH_Srv["RAGFlowHandler"]
            SocketFlowH["SocketFlowHandler"]
            SpeechSocketH["SpeechSocketHandler"]
            HealthH["HealthHandler"]
        end

        subgraph Services_Srv ["Core Services"]
            direction TB
            GraphSvc_Srv["GraphService"]
            FileSvc_Srv["FileService"]
            NostrSvc_Srv["NostrService"]
            SpeechSvc_Srv["SpeechService"]
            RAGFlowSvc_Srv["RAGFlowService"]
            PerplexitySvc_Srv["PerplexityService"]
        end

        subgraph Actors_Srv ["Actor System"]
            direction TB
            GraphServiceActor["GraphServiceActor"]
            SettingsActor["SettingsActor"]
            MetadataActor["MetadataActor"]
            ClientManagerActor["ClientManagerActor"]
            GPUComputeActor["GPUComputeActor"]
            ProtectedSettingsActor["ProtectedSettingsActor"]
        end
        AppState_Srv["AppState holds Addr<...>"]

        Actix --> Handlers_Srv

        Handlers_Srv --> AppState_Srv
        SocketFlowH --> ClientManagerActor
        GraphAPI_H --> GraphServiceActor
        SettingsH --> SettingsActor
        NostrAuthH --> ProtectedSettingsActor

        GraphServiceActor --> ClientManagerActor
        GraphServiceActor --> MetadataActor
        GraphServiceActor --> GPUComputeActor
        GraphServiceActor --> SettingsActor

        FileSvc_Srv --> MetadataActor
        NostrSvc_Srv --> ProtectedSettingsActor
        SpeechSvc_Srv --> SettingsActor
        RAGFlowSvc_Srv --> SettingsActor
        PerplexitySvc_Srv --> SettingsActor
    end

    subgraph External_Srv ["External Services"]
        direction LR
        GitHub["GitHub API"]
        NostrRelays_Ext["Nostr Relays"]
        OpenAI["OpenAI API"]
        PerplexityAI_Ext["PerplexityAI API"]
        RAGFlow_Ext["RAGFlow API"]
        Kokoro_Ext["Kokoro API"]
    end

    WebSocketSvc <--> SocketFlowH
    APISvc <--> Actix

    FileSvc_Srv --> GitHub
    NostrSvc_Srv --> NostrRelays_Ext
    SpeechSvc_Srv --> OpenAI
    SpeechSvc_Srv --> Kokoro_Ext
    PerplexitySvc_Srv --> PerplexityAI_Ext
    RAGFlowSvc_Srv --> RAGFlow_Ext

    style ClientApp fill:#282C34,stroke:#61DAFB,stroke-width:2px,color:#FFFFFF
    style ServerApp fill:#282C34,stroke:#A2AAAD,stroke-width:2px,color:#FFFFFF
    style External_Srv fill:#282C34,stroke:#F7DF1E,stroke-width:2px,color:#FFFFFF
    style AppInit fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style TwoPane fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style GraphView fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style GraphCanvas fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style RightCtlPanel fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style SettingsUI fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style ConvoPane fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style NarrativePane fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style SettingsMgr fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style GraphDataMgr fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style RenderEngine fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style WebSocketSvc fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style APISvc fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style NostrAuthSvcClient fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF
    style XRController fill:#3A3F47,stroke:#61DAFB,color:#FFFFFF

    style Actix fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style Handlers_Srv fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style SettingsH fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style NostrAuthH fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style GraphAPI_H fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style FilesAPI_H fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style RAGFlowH_Srv fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style SocketFlowH fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style SpeechSocketH fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style HealthH fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style Services_Srv fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style GraphSvc_Srv fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style FileSvc_Srv fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style NostrSvc_Srv fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style SpeechSvc_Srv fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style RAGFlowSvc_Srv fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style PerplexitySvc_Srv fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style Actors_Srv fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style GraphServiceActor fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style SettingsActor fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style MetadataActor fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style ClientManagerActor fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style GPUComputeActor fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style ProtectedSettingsActor fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF
    style AppState_Srv fill:#3A3F47,stroke:#A2AAAD,color:#FFFFFF

    style GitHub fill:#3A3F47,stroke:#F7DF1E,color:#FFFFFF
    style NostrRelays_Ext fill:#3A3F47,stroke:#F7DF1E,color:#FFFFFF
    style OpenAI fill:#3A3F47,stroke:#F7DF1E,color:#FFFFFF
    style PerplexityAI_Ext fill:#3A3F47,stroke:#F7DF1E,color:#FFFFFF
    style RAGFlow_Ext fill:#3A3F47,stroke:#F7DF1E,color:#FFFFFF
    style Kokoro_Ext fill:#3A3F47,stroke:#F7DF1E,color:#FFFFFF

Class Diagram

classDiagram
    direction LR

    %% Frontend Classes
    class AppInitializer {
        <<ReactComponent>>
        +initializeServices()
    }
    class GraphManager {
        <<ReactComponent>>
        +renderNodesAndEdges()
    }
    class WebSocketService {
        <<Service>>
        +connect()
        +sendMessage()
        +onBinaryMessage()
        +isReady()
    }
    class SettingsStore {
        <<ZustandStore>>
        +settings: Settings
        +updateSettings()
    }
    class GraphDataManager {
        <<Service>>
        +fetchInitialData()
        +updateNodePositions()
        +getGraphData()
        +setWebSocketService()
    }
    class NostrAuthService {
        <<Service>>
        +loginWithNostr()
        +verifySession()
        +logout()
    }
    AppInitializer --> SettingsStore
    AppInitializer --> NostrAuthService
    AppInitializer --> WebSocketService
    AppInitializer --> GraphDataManager
    GraphDataManager --> WebSocketService
    GraphDataManager --> GraphManager

    %% Backend Classes
    class AppState {
        <<Struct>>
        +graph_service_addr: Addr<GraphServiceActor>
        +gpu_compute_addr: Option<Addr<GPUComputeActor>>
        +settings_addr: Addr<SettingsActor>
        +protected_settings_addr: Addr<ProtectedSettingsActor>
        +metadata_addr: Addr<MetadataActor>
        +client_manager_addr: Addr<ClientManagerActor>
        +github_client: Arc<GitHubClient>
        +content_api: Arc<ContentAPI>
        +perplexity_service: Option<Arc<PerplexityService>>
        +ragflow_service: Option<Arc<RAGFlowService>>
        +speech_service: Option<Arc<SpeechService>>
        +nostr_service: Option<Arc<NostrService>>
    }
    class GraphServiceActor {
        <<Actor>>
        +handle_UpdateGraphData()
        +handle_InitializeGPU()
        +handle_SimulationStep()
        +handle_UpdateNodePosition()
    }
    class PerplexityService {
        <<Struct>>
        +query()
    }
    class RagFlowService {
        <<Struct>>
        +chat()
    }
    class SpeechService {
        <<Struct>>
        +process_stt_request()
        +process_tts_request()
    }
    class NostrService {
        <<Struct>>
        +verify_auth_event()
        +validate_session()
        +manage_user_api_keys()
    }
    class GPUComputeActor {
        <<Actor>>
        +handle_InitializeGPU()
        +handle_UpdateGPUGraphData()
        +handle_UpdateSimulationParams()
        +handle_ComputeForces()
        +handle_GetNodeData()
        +handle_GetGPUStatus()
    }
    class FileService {
        <<Struct>>
        +fetch_and_process_content()
        +update_metadata_store()
    }
    AppState --> GraphServiceActor : holds_Addr
    AppState --> NostrService : holds_Arc
    AppState --> PerplexityService : holds_Arc
    AppState --> RagFlowService : holds_Arc
    AppState --> SpeechService : holds_Arc
    AppState --> GPUComputeActor : holds_Option_Addr
    AppState --> FileService : holds_Arc

    WebSocketService ..> GraphServiceActor : sends_UpdateNodePositions
    GraphServiceActor ..> GPUComputeActor : uses_optional
    NostrService ..> ProtectedSettingsActor : uses

Sequence Diagrams

Server Initialization Sequence

sequenceDiagram
    participant Main as main.rs
    participant AppStateMod as app_state.rs
    participant ConfigMod as config/mod.rs
    participant Services as Various Services (Graph, File, Nostr, AI)
    participant ClientMgr as ClientManager (Static)
    participant GraphSvc as GraphService

    Main->>ConfigMod: AppFullSettings::load()
    ConfigMod-->>Main: loaded_settings
    Main->>AppStateMod: AppState::new(loaded_settings, /* other deps */)
    AppStateMod->>Services: Initialize FileService, NostrService, AI Services with configs
    AppStateMod->>GraphSvc: GraphService::new(settings, gpu_compute_opt, ClientMgr::instance())
    GraphSvc->>GraphSvc: Start physics_loop (async task)
    GraphSvc->>ClientMgr: (inside loop) Send updates
    AppStateMod-->>Main: app_state_instance
    Main->>ActixServer: .app_data(web::Data::new(app_state_instance))

Client Initialization Sequence

sequenceDiagram
    participant ClientApp as AppInitializer.tsx
    participant SettingsStoreSvc as settingsStore.ts
    participant NostrAuthSvcClient as nostrAuthService.ts
    participant WebSocketSvcClient as WebSocketService.ts
    participant ServerAPI as Backend REST API
    participant ServerWS as Backend WebSocket Handler

    ClientApp->>SettingsStoreSvc: Load settings (from localStorage & defaults)
    SettingsStoreSvc-->>ClientApp: Initial settings

    ClientApp->>NostrAuthSvcClient: Check current session (e.g., from localStorage)
    alt Session token exists
        NostrAuthSvcClient->>ServerAPI: POST /api/auth/nostr/verify (token)
        ServerAPI-->>NostrAuthSvcClient: Verification Result (user, features)
        NostrAuthSvcClient->>ClientApp: Auth status updated
    else No session token
        NostrAuthSvcClient->>ClientApp: Auth status (unauthenticated)
    end

    ClientApp->>WebSocketSvcClient: connect()
    WebSocketSvcClient->>ServerWS: WebSocket Handshake
    ServerWS-->>WebSocketSvcClient: Connection Established (e.g., `onopen`)
    WebSocketSvcClient->>WebSocketSvcClient: Set isConnected = true
    ServerWS-->>WebSocketSvcClient: Send {"type": "connection_established"} (or similar)
    WebSocketSvcClient->>WebSocketSvcClient: Set isServerReady = true

    alt WebSocket isReady()
        WebSocketSvcClient->>ServerWS: Send {"type": "requestInitialData"}
        ServerWS-->>WebSocketSvcClient: Initial Graph Data (e.g., large JSON or binary)
        WebSocketSvcClient->>GraphDataManager: Process initial data
    end

Real-time Graph Updates Sequence

sequenceDiagram
    participant ClientApp
    participant WebSocketSvcClient as WebSocketService.ts
    participant GraphDataMgrClient as GraphDataManager.ts
    participant ServerGraphSvc as GraphService (Backend)
    participant ServerGpuUtil as GPUCompute (Backend, Optional)
    participant ServerClientMgr as ClientManager (Backend, Static)
    participant ServerSocketFlowH as SocketFlowHandler (Backend)

    %% Continuous Server-Side Loop
    ServerGraphSvc->>ServerGraphSvc: physics_loop() iteration
    alt GPU Enabled
        ServerGraphSvc->>ServerGpuUtil: run_simulation_step()
        ServerGpuUtil-->>ServerGraphSvc: updated_node_data_from_gpu
    else CPU Fallback
        ServerGraphSvc->>ServerGraphSvc: calculate_layout_cpu()
    end
    ServerGraphSvc->>ServerClientMgr: BroadcastBinaryPositions(updated_node_data)

    ServerClientMgr->>ServerSocketFlowH: Distribute to connected clients
    ServerSocketFlowH-->>WebSocketSvcClient: Binary Position Update (Chunk)

    WebSocketSvcClient->>GraphDataMgrClient: onBinaryMessage(chunk)
    GraphDataMgrClient->>GraphDataMgrClient: Decompress & Parse chunk
    GraphDataMgrClient->>ClientApp: Notify UI/Renderer of position changes

    %% Optional: Client sends an update (e.g., user drags a node)
    opt User Interaction
        ClientApp->>GraphDataMgrClient: User moves node X to new_pos
        GraphDataMgrClient->>WebSocketSvcClient: sendRawBinaryData(node_X_new_pos_update) %% Or JSON message
        WebSocketSvcClient->>ServerSocketFlowH: Forward client update
        ServerSocketFlowH->>ServerGraphSvc: Apply client update to physics model (if supported)
    end

Authentication Flow Sequence

sequenceDiagram
    participant User
    participant ClientUI
    participant NostrAuthSvcClient as nostrAuthService.ts
    participant WindowNostr as "window.nostr (Extension)"
    participant APISvcClient as api.ts
    participant ServerNostrAuthH as NostrAuthHandler (Backend)
    participant ServerNostrSvc as NostrService (Backend)

    User->>ClientUI: Clicks Login Button
    ClientUI->>NostrAuthSvcClient: initiateLogin()
    NostrAuthSvcClient->>ServerNostrAuthH: GET /api/auth/nostr/challenge (via APISvcClient)
    ServerNostrAuthH-->>NostrAuthSvcClient: challenge_string

    NostrAuthSvcClient->>WindowNostr: signEvent(kind: 22242, content: "auth", tags:[["challenge", challenge_string], ["relay", ...]])
    WindowNostr-->>NostrAuthSvcClient: signed_auth_event

    NostrAuthSvcClient->>APISvcClient: POST /api/auth/nostr (signed_auth_event)
    APISvcClient->>ServerNostrAuthH: Forward request
    ServerNostrAuthH->>ServerNostrSvc: verify_auth_event(signed_auth_event)
    alt Event Valid
        ServerNostrSvc->>ServerNostrSvc: Generate session_token, store user session
        ServerNostrSvc-->>ServerNostrAuthH: AuthResponse (user, token, expiresAt, features)
        ServerNostrAuthH-->>APISvcClient: AuthResponse
        APISvcClient-->>NostrAuthSvcClient: AuthResponse
        NostrAuthSvcClient->>NostrAuthSvcClient: Store token, user data
        NostrAuthSvcClient->>ClientUI: Update auth state (Authenticated)
    else Event Invalid
        ServerNostrSvc-->>ServerNostrAuthH: Error
        ServerNostrAuthH-->>APISvcClient: Error Response
        APISvcClient-->>NostrAuthSvcClient: Error
        NostrAuthSvcClient->>ClientUI: Show Login Error
    end

Settings Synchronization Sequence

sequenceDiagram
    participant User
    participant ClientUI
    participant SettingsStoreClient as settingsStore.ts
    participant SettingsSvcClient as settingsService.ts (part of api.ts or separate)
    participant ServerSettingsH as SettingsHandler (Backend)
    participant ServerAppState as AppState (Backend)
    participant ServerUserSettings as UserSettings Model (Backend)
    participant ServerClientMgr as ClientManager (Backend, Static, for broadcast if applicable)

    User->>ClientUI: Modifies a setting (e.g., node size)
    ClientUI->>SettingsStoreClient: updateSettings({ visualisation: { nodes: { nodeSize: newValue }}})
    SettingsStoreClient->>SettingsStoreClient: Update local state (Zustand) & persist to localStorage

    alt User is Authenticated
        SettingsStoreClient->>SettingsSvcClient: POST /api/user-settings/sync (ClientSettingsPayload)
        SettingsSvcClient->>ServerSettingsH: Forward request
        ServerSettingsH->>ServerAppState: Get current AppFullSettings / UserSettings
        alt User is PowerUser
            ServerSettingsH->>ServerAppState: Update AppFullSettings in memory
            ServerAppState->>ServerAppState: AppFullSettings.save() to settings.yaml
            ServerSettingsH-->>SettingsSvcClient: Updated UISettings (reflecting global)
            %% Optional: Server broadcasts global settings change if implemented
            %% ServerAppState->>ServerClientMgr: BroadcastGlobalSettingsUpdate(updated_AppFullSettings)
            %% ServerClientMgr-->>OtherClients: Global settings update message
        else Regular User
            ServerSettingsH->>ServerUserSettings: Load or create user's UserSettings file
            ServerUserSettings->>ServerUserSettings: Update UISettings part of UserSettings
            ServerUserSettings->>ServerUserSettings: Save UserSettings to user-specific YAML
            ServerSettingsH-->>SettingsSvcClient: Updated UISettings (user-specific)
        end
        SettingsSvcClient-->>SettingsStoreClient: Confirmation / Updated settings (if different)
        %% Client store might re-sync if server response indicates changes
    end

AR Features Implementation Status

Hand Tracking (Meta Quest 3)

XR Interaction is primarily managed by client/src/features/xr/systems/HandInteractionSystem.tsx and related hooks/providers like useSafeXRHooks.tsx.
Session management is in client/src/features/xr/managers/xrSessionManager.ts.
Initialisation logic is in client/src/features/xr/managers/xrInitializer.ts.
The main XR entry point and controller is client/src/features/xr/components/XRController.tsx.
Type definitions for WebXR, including hand tracking, are in client/src/features/xr/types/xr.ts and potentially augmented by client/src/features/xr/types/webxr-extensions.d.ts (though this file is noted as mostly commented out).

Current Challenges

The webxr-extensions.d.ts file is largely commented out, indicating potential gaps or reliance on default browser types for hand tracking APIs, which might vary.
Robust gesture recognition based on joint positions requires significant implementation in HandInteractionSystem.tsx.

Next Steps

Review and complete necessary type definitions in webxr-extensions.d.ts if standard types are insufficient.
Implement gesture recognition logic.
Optimize performance for AR/passthrough modes.

Authentication and Settings Inheritance

Unauthenticated Users

Use browser's localStorage for settings persistence (via Zustand persist middleware in client/src/store/settingsStore.ts).
Settings are stored locally and not synced to a user-specific backend store.
Default to basic settings visibility.
Limited to local visualisation features; AI and GitHub features requiring API keys will not be available unless default API keys are configured in the server's ProtectedSettings.

Authenticated Users (Nostr)

Regular Users:
- Settings are loaded from and saved to user-specific files on the server (e.g., /app/user_settings/<pubkey>.yaml), managed by src/handlers/settings_handler.rs using the UserSettings model (which contains UISettings).
- These user-specific settings are primarily UI/visualisation preferences defined in UISettings.
- Can access features based on their configuration in src/config/feature_access.rs.
- Can manage their own API keys for AI services via the /api/auth/nostr/api-keys endpoint. These keys are stored in their NostrUser profile within the server's ProtectedSettings.
Power Users:
- Directly load and modify the global server settings from settings.yaml (represented by AppFullSettings in Rust, which is then used to derive UISettings).
- Have full access to all settings and advanced API features. API keys for these might come from AppFullSettings (if globally configured for all power users) or their own NostrUser profile in ProtectedSettings.
- Settings modifications made by power users to AppFullSettings are persisted to the main settings.yaml and potentially broadcast to other clients (if implemented).

Settings Inheritance Flow

graph TD
    A[Start] --> B{"Authenticated?"}
    B -->|No| C["Load LocalSettings (localStorage via Zustand)"]
    B -->|Yes| D{"Is Power User? (feature_access.rs)"}
    D -->|No| E["Load UserSpecificSettings (user_settings/pubkey.yaml via API)"]
    D -->|Yes| F["Load GlobalServerSettings (settings.yaml via API)"]
    C --> X["Apply Settings to UI"]
    E --> X
    F --> X

Settings Sync Flow

graph TD
    A["Setting Changed in UI"] --> B{"Authenticated?"}
    B -->|No| C["Save Locally (localStorage via Zustand)"]
    B -->|Yes| D{"Is Power User?"}
    D -->|No| E["Save to UserSpecificSettings (user_settings/pubkey.yaml via API)"]
    D -->|Yes| F["Save to GlobalServerSettings (settings.yaml via API)"]
    F --> G["Server Broadcasts GlobalSettingsUpdate to All Clients"]
    G --> H["Other Clients Update Local Store"]
    E --> I["User's Local Store Updated"]
    C --> I

Modular Control Panel Architecture

The client's user interface for settings and controls is structured as follows:

Main Layout: client/src/app/TwoPaneLayout.tsx divides the screen.
Right Pane Host: client/src/app/components/RightPaneControlPanel.tsx hosts various panels within the right-hand side.
Settings UI Core: client/src/features/settings/components/panels/SettingsPanelRedesignOptimized.tsx provides the tabbed interface for different setting categories (Visualisation, System, AI, XR).
- Tabs Component: Uses a generic client/src/ui/Tabs.tsx component for tab navigation.
- Settings Sections: Each tab within SettingsPanelRedesign.tsx renders one or more SettingsSection.tsx components to group related settings. These sections can be collapsible.
- Individual Controls: Each SettingsSection.tsx uses multiple SettingControlComponent.tsx instances to render the actual UI controls (sliders, toggles, inputs, etc.) for each setting.
State Management:
- Settings values are primarily managed by the Zustand store defined in client/src/store/settingsStore.ts.
- Context for control panel specific state (like detached panel states or advanced view toggles) is managed by client/src/features/settings/components/control-panel-context.tsx.

The client/src/components/layout/ControlPanel.tsx mentioned in the original README seems to be superseded or refactored into the SettingsPanelRedesign.tsx and its constituent parts. Detachable sections are a feature of SettingsSection.tsx.

Performance Optimisations

Debounced Updates: SettingControlComponent.tsx uses onBlur or Enter key for text/number inputs, which acts as a form of debouncing for settings changes that might trigger expensive re-renders or API calls.
CSS Transforms: Used by react-draggable for smooth movement of detached panels.
Memoisation: useMemo is used in components like GraphManager.tsx to stabilise expensive calculations or object references.
Targeted Re-renders: Zustand store selectors for primitive values are used in some places (e.g., App.tsx) to avoid unnecessary re-renders.

The goal is to maintain responsiveness, especially during interactions with the 3D visualisation and real-time updates.

System Requirements

Docker: 20.10+ with Docker Compose
Node.js: 20+ (for local development)
Rust: 1.75+ (for backend development)
GPU: Optional but recommended for large agent swarms
Browser: Chrome/Firefox/Safari with WebGL support

Performance Benchmarks

Metric	Performance
Max Agents (60 FPS)	200+
Position Update Latency	< 5ms
Binary Protocol Efficiency	28 bytes/agent
Network Bandwidth (100 agents)	168 KB/s
Memory Usage	28 bytes/agent

License

This project is licensed under the MIT License. See the LICENSE file for details.

Acknowledgements

Prof Rob Aspin: For inspiring the project's vision and providing valuable resources
Claude Flow Team: For the amazing MCP hive-mind agent framework
Anthropic: For the Model Context Protocol specification
OpenAI: For their advanced AI models powering the question-answering features
Perplexity AI and RAGFlow: For their AI services enhancing content processing
Three.js: For the robust 3D rendering capabilities
Actix: For the high-performance web framework powering the backend

Community

Discord: Join our community (placeholder)
GitHub Issues: Report bugs or request features
Twitter: @LogseqXR (placeholder)

Built with ❤️ by the VisionFlow community

Name		Name	Last commit message	Last commit date
Latest commit History 1,138 Commits
agent-control-interface		agent-control-interface
client		client
data		data
docs		docs
scripts		scripts
src		src
.DS_Store		.DS_Store
.dockerignore		.dockerignore
.env_template		.env_template
.eslintrc.json		.eslintrc.json
.gitignore		.gitignore
.prettierrc		.prettierrc
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
ChloeOctave.jpg		ChloeOctave.jpg
Dockerfile.dev		Dockerfile.dev
Dockerfile.production		Dockerfile.production
README.md		README.md
config.yml		config.yml
docker-compose.dev.yml		docker-compose.dev.yml
docker-compose.production.yml		docker-compose.production.yml
docker-compose.yml		docker-compose.yml
groupOctave.jpg		groupOctave.jpg
jarvisOld.png		jarvisOld.png
nginx.conf		nginx.conf
nginx.dev.conf		nginx.dev.conf
visionflow.gif		visionflow.gif

DreamLab-AI/VisionFlow

Folders and files

Latest commit

History

Repository files navigation