diff --git a/docs/cpu_inference/readme.md b/docs/cpu_inference/readme.md new file mode 100644 index 0000000..001770b --- /dev/null +++ b/docs/cpu_inference/readme.md @@ -0,0 +1,169 @@ +# CPU Inference with Ollama + +This blueprint explains how to use CPU inference for running large language models using Ollama. It includes two main deployment strategies: +- Serving pre-saved models directly from Object Storage +- Pulling models from Ollama and saving them to Object Storage + +--- + +## Why CPU Inference? + +CPU inference is ideal for: +- Low-throughput or cost-sensitive deployments +- Offline testing and validation +- Prototyping without GPU dependency + +--- + +## Supported Models + +Ollama supports several high-quality open-source LLMs. Below is a small set of commonly used models: + +| Model Name | Description | +|------------|--------------------------------| +| gemma | Lightweight open LLM by Google | +| llama2 | Meta’s large language model | +| mistral | Open-weight performant LLM | +| phi3 | Microsoft’s compact LLM | + +--- + +## Deploying with OCI AI Blueprint + +### Running Ollama Models from Object Storage + +If you've already pushed your model to **Object Storage**, use the following service-mode recipe to run it. Ensure your model files are in the **blob + manifest** format used by Ollama. + +#### Recipe Configuration + +| Field | Description | +|--------------------------------|------------------------------------------------| +| recipe_id | `cpu_inference` – Identifier for the recipe | +| recipe_mode | `service` – Used for long-running inference | +| deployment_name | Custom name for the deployment | +| recipe_image_uri | URI for the container image in OCIR | +| recipe_node_shape | OCI shape, e.g., `BM.Standard.E4.128` | +| input_object_storage | Object Storage bucket mounted as input | +| recipe_container_env | List of environment variables | +| recipe_replica_count | Number of replicas | +| recipe_container_port | Port to expose the container | +| recipe_node_pool_size | Number of nodes in the pool | +| recipe_node_boot_volume_size_in_gbs | Boot volume size in GB | +| recipe_container_command_args | Arguments for the container command | +| recipe_ephemeral_storage_size | Temporary scratch storage | + +#### Sample Recipe (Service Mode) +```json +{ + "recipe_id": "cpu_inference", + "recipe_mode": "service", + "deployment_name": "gemma and BME4 service", + "recipe_image_uri": "iad.ocir.io/iduyx1qnmway/corrino-devops-repository:cpu_inference_service_v0.2", + "recipe_node_shape": "BM.Standard.E4.128", + "input_object_storage": [ + { + "bucket_name": "ollama-models", + "mount_location": "/models", + "volume_size_in_gbs": 20 + } + ], + "recipe_container_env": [ + { "key": "MODEL_NAME", "value": "gemma" }, + { "key": "PROMPT", "value": "What is the capital of France?" } + ], + "recipe_replica_count": 1, + "recipe_container_port": "11434", + "recipe_node_pool_size": 1, + "recipe_node_boot_volume_size_in_gbs": 200, + "recipe_container_command_args": [ + "--input_directory", "/models", "--model_name", "gemma" + ], + "recipe_ephemeral_storage_size": 100 +} +``` + +--- + +### Accessing the API + +Once deployed, send inference requests to the model via the exposed port: + +```bash +curl http://:11434/api/generate -d '{ + "model": "gemma", + "prompt": "What is the capital of France?", + "stream": false +}' +``` + +### Example Public Inference Calls +```bash +curl -L POST https://cpu-inference-mismistral.130-162-199-33.nip.io/api/generate \ + -d '{ "model": "mistral", "prompt": "What is the capital of Germany?" }' \ + | jq -r 'select(.response) | .response' | paste -sd " " + +curl -L -k POST https://cpu-inference-mistral-flexe4.130-162-199-33.nip.io/api/generate \ + -d '{ "model": "mistral", "prompt": "What is the capital of Germany?" }' \ + | jq -r 'select(.response) | .response' | paste -sd " " +``` +--- + +### Pulling from Ollama and Saving to Object Storage + +To download a model from Ollama and store it in Object Storage, use the job-mode recipe below. + +#### Recipe Configuration + +| Field | Description | +|--------------------------------|------------------------------------------------| +| recipe_id | `cpu_inference` – Same recipe base | +| recipe_mode | `job` – One-time job to save a model | +| deployment_name | Custom name for the saving job | +| recipe_image_uri | OCIR URI of the saver image | +| recipe_node_shape | Compute shape used for the job | +| output_object_storage | Where to store pulled models | +| recipe_container_env | Environment variables including model name | +| recipe_replica_count | Set to 1 | +| recipe_container_port | Typically `11434` for Ollama | +| recipe_node_pool_size | Set to 1 | +| recipe_node_boot_volume_size_in_gbs | Size in GB | +| recipe_container_command_args | Set output directory and model name | +| recipe_ephemeral_storage_size | Temporary storage | + +#### Sample Recipe (Job Mode) +```json +{ + "recipe_id": "cpu_inference", + "recipe_mode": "job", + "deployment_name": "gemma and BME4 saver", + "recipe_image_uri": "iad.ocir.io/iduyx1qnmway/corrino-devops-repository:cpu_inference_saver_v0.2", + "recipe_node_shape": "BM.Standard.E4.128", + "output_object_storage": [ + { + "bucket_name": "ollama-models", + "mount_location": "/models", + "volume_size_in_gbs": 20 + } + ], + "recipe_container_env": [ + { "key": "MODEL_NAME", "value": "gemma" }, + { "key": "PROMPT", "value": "What is the capital of France?" } + ], + "recipe_replica_count": 1, + "recipe_container_port": "11434", + "recipe_node_pool_size": 1, + "recipe_node_boot_volume_size_in_gbs": 200, + "recipe_container_command_args": [ + "--output_directory", "/models", "--model_name", "gemma" + ], + "recipe_ephemeral_storage_size": 100 +} +``` + +--- + +## Final Notes + +- Ensure all OCI IAM permissions are set to allow Object Storage access. +- Confirm that bucket region and deployment region match. +- Use the job-mode recipe once to save a model, and the service-mode recipe repeatedly to serve it. diff --git a/docs/healthcheck/readme.md b/docs/healthcheck/readme.md new file mode 100644 index 0000000..1266f91 --- /dev/null +++ b/docs/healthcheck/readme.md @@ -0,0 +1,202 @@ +# GPU Health Check & Pre-Check Blueprint + +This blueprint provides a **pre-check blueprint** for GPU health validation before running production or research workloads. The focus is on delivering a **diagnostic tool** that can run on both single-node and multi-node environments, ensuring that your infrastructure is ready for demanding experiments. + +The workflow includes: +- **Data types** as input (`fp8`, `fp16`, `fp32`, `fp64`) +- **Custom Functions** for GPU diagnostics +- **GPU-Burn** for stress testing +- **Results** collected in JSON files (and optionally PDF reports) + +By following this blueprint, you can identify and localize issues such as thermal throttling, power irregularities, or GPU instability before they impact your main workloads. + +--- + +## 1. Architecture Overview + +Below is a simplified overview: + +image + + +### Key Points + +- Data Types: You can specify one of several floating-point precisions (`fp8`, fp16, fp32, `fp64`). +- Custom Functions: Diagnostic functions that measure performance metrics such as throughput, memory bandwidth, etc. +- Single-Node vs. Multi-Node: Tests can be run on a single machine or scaled to multiple machines. +- GPU-Burn: A specialized stress-testing tool for pushing GPUs to their maximum performance limits. +- Results: Output is aggregated into JSON files (and optionally PDFs) for analysis. + +--- + +## 2. Health Check Blueprint + +This blueprint aims to give you confidence that your GPUs are healthy. The key checks include: + +1. Compute Throughput + - Dense matrix multiplications or arithmetic operations stress the GPU cores. + - Ensures sustained performance without degradation. + +2. Memory Bandwidth + - Reading/writing large chunks of data (e.g., `torch.rand()`) tests memory throughput. + - Verifies the memory subsystem operates at expected speeds. + +3. Temperature & Thermal Stability + - Uses commands like nvidia-smi to monitor temperature. + - Checks for throttling under load. + +4. Power Consumption + - Monitors power draw (e.g., `nvidia-smi --query-gpu=power.draw --format=csv`). + - Identifies irregular or excessive power usage. + +5. GPU Utilization + - Ensures GPU cores (including Tensor Cores) are fully engaged during tests. + - Confirms no unexpected idle time. + +6. Error Detection + - Checks for hardware errors or CUDA-related issues. + - Asserts numerical correctness to ensure no silent failures. + +7. Multi-GPU Testing + - Validates multi-GPU or multi-node setups. + - Ensures the entire environment is consistent and stable. + +8. Mixed Precision Testing + - Uses AMP for fp8 or fp16 operations (e.g., `torch.cuda.amp.autocast()`). + - Confirms performance and compatibility with mixed-precision ops. + +--- + +## 3. Data Types and How They Work + +- fp8, fp16: Lower precision can offer speedups but requires checks for numerical stability. +- fp32 (single precision): Standard for most deep learning tasks; tests confirm typical GPU operations. +- fp64 (double precision): Used in HPC/scientific workloads; verifies performance and accuracy at high precision. + +Depending on the dtype you select, the script either runs a set of Custom Functions or launches GPU-Burn to push the hardware to its limits. The results are saved in JSON for analysis. + +--- + +## 4. Custom Functions + +These Python-based diagnostic functions systematically measure: + +- Throughput (matrix multiplies, convolution stubs, etc.) +- Memory bandwidth (large tensor reads/writes) +- Temperature (via nvidia-smi or other sensors) +- Power usage +- GPU utilization +- Error detection (assert checks, error logs) +- Multi-GPU orchestration (parallel usage correctness) +- Mixed precision compatibility (AMP in PyTorch) + +They can run on a single node or multiple nodes, with each run producing structured JSON output. + +--- + +## 5. GPU-Burn + +[GPU-Burn](https://github.com/wilicc/gpu-burn) is a stress-testing tool designed to push GPUs to their maximum performance limits. It is typically used to: + +- Validate hardware stability +- Identify potential overheating or faulty components +- Confirm GPUs can handle extreme workloads without errors or throttling + +When you run GPU-Burn in float32 or float64 mode, its output can be captured in a log file, then parsed into JSON or PDF summaries for reporting. + +--- + + +## 6. Usage + +1. Clone the Blueprint & Install Dependencies + + +Bash + + git clone + cd + docker build -t gpu-healthcheck . + + +2. Run the Pre-Check + - Single Node Example (fp16): + +Bash + + + docker run --gpus all -it -v $(pwd)/results:/app/testing_results gpu-healthcheck --dtype float16 --expected_gpus A10:2,A100:0,H100:0 + + + - GPU-Burn Stress Test (float32): + +Bash + + + docker run --gpus all -it -v $(pwd)/results:/app/testing_results gpu-healthcheck --dtype float32 --expected_gpus A10:2,A100:0,H100:0 + + +3. Examine Results + - JSON output is located in the results/ directory. + - PDF summaries will also be generated. + +--- +## 7. Implementing it into OCI AI Blueprints + +This is an example of json file which be used to deploy into OCI AI Blueprints: + +```json +{ + "recipe_id": "healthcheck", + "recipe_mode": "job", + "deployment_name": "healthcheck", + "recipe_image_uri": "iad.ocir.io/iduyx1qnmway/corrino-devops-repository:healthcheck_v0.3", + "recipe_node_shape": "VM.GPU.A10.2", + "output_object_storage": [ + { + "bucket_name": "healthcheck2", + "mount_location": "/healthcheck_results", + "volume_size_in_gbs": 20 + } + ], + "recipe_container_command_args": [ + "--dtype", "float16", "--output_dir", "/healthcheck_results", "--expected_gpus", "A10:2,A100:0,H100:0" + ], + "recipe_replica_count": 1, + "recipe_nvidia_gpu_count": 2, + "recipe_node_pool_size": 1, + "recipe_node_boot_volume_size_in_gbs": 200, + "recipe_ephemeral_storage_size": 100, + "recipe_shared_memory_volume_size_limit_in_mb": 1000, + "recipe_use_shared_node_pool": true +} +``` +--- + +## Explanation of Healthcheck Recipe Fields + +| Field | Type | Example Value | Description | +|---------------------------------------|-------------|-------------------------------------------------------------------------------|-------------| +| `recipe_id` | string | `"healthcheck"` | Identifier for the recipe | +| `recipe_mode` | string | `"job"` | Whether the recipe runs as a one-time job or a service | +| `deployment_name` | string | `"healthcheck"` | Name of the deployment/job | +| `recipe_image_uri` | string | `"iad.ocir.io/.../healthcheck_v0.3"` | URI of the container image stored in OCI Container Registry | +| `recipe_node_shape` | string | `"VM.GPU.A10.2"` | Compute shape to use for this job | +| `output_object_storage.bucket_name` | string | `"healthcheck2"` | Name of the Object Storage bucket to write results | +| `output_object_storage.mount_location`| string | `"/healthcheck_results"` | Directory inside the container where the bucket will be mounted | +| `output_object_storage.volume_size_in_gbs` | integer | `20` | Storage volume size (GB) for the mounted bucket | +| `recipe_container_command_args` | list | `[--dtype, float16, --output_dir, /healthcheck_results, --expected_gpus, A10:2,A100:0,H100:0]` | Arguments passed to the container | +| `--dtype` | string | `"float16"` | Precision type for computations (e.g. float16, float32) | +| `--output_dir` | string | `"/healthcheck_results"` | Directory for writing output (maps to mounted bucket) | +| `--expected_gpus` | string | `"A10:2,A100:0,H100:0"` | Expected GPU types and counts | +| `recipe_replica_count` | integer | `1` | Number of replicas (containers) to run | +| `recipe_nvidia_gpu_count` | integer | `2` | Number of GPUs to allocate | +| `recipe_node_pool_size` | integer | `1` | Number of nodes to provision | +| `recipe_node_boot_volume_size_in_gbs`| integer | `200` | Size of the boot volume (GB) | +| `recipe_ephemeral_storage_size` | integer | `100` | Ephemeral scratch storage size (GB) | +| `recipe_shared_memory_volume_size_limit_in_mb` | integer | `1000` | Size of shared memory volume (`/dev/shm`) in MB | +| `recipe_use_shared_node_pool` | boolean | `true` | Whether to run on a shared node pool | + +## 8. Contact + +For questions or additional information, open an issue in this blueprint or contact the maintainers directly. diff --git a/docs/whisper_transcription/Docker/Dockerfile b/docs/whisper_transcription/Docker/Dockerfile new file mode 100644 index 0000000..6092636 --- /dev/null +++ b/docs/whisper_transcription/Docker/Dockerfile @@ -0,0 +1,38 @@ +# Base image with Python and CUDA (for H100/A100 GPUs with CUDA 12.1) +FROM nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04 + + +# Set environment variables +ENV DEBIAN_FRONTEND=noninteractive +ENV TRANSFORMERS_CACHE=/root/.cache/huggingface/transformers + +# Install system packages and cuDNN 9.10.1 manually (for CUDA 12) +RUN apt-get update && apt-get install -y \ + python3 python3-pip python3-venv python3-dev \ + ffmpeg git curl wget ca-certificates gnupg libsndfile1 && \ + mkdir -p /tmp/cudnn && cd /tmp/cudnn && \ + wget https://developer.download.nvidia.com/compute/cudnn/9.10.1/local_installers/cudnn-local-repo-ubuntu2204-9.10.1_1.0-1_amd64.deb && \ + dpkg -i cudnn-local-repo-ubuntu2204-9.10.1_1.0-1_amd64.deb && \ + cp /var/cudnn-local-repo-ubuntu2204-9.10.1/cudnn-*-keyring.gpg /usr/share/keyrings/ && \ + apt-get update && \ + apt-get -y install cudnn-cuda-12 && \ + rm -rf /var/lib/apt/lists/* /tmp/cudnn + +# Upgrade pip and install torch separately for CUDA 12.1 +RUN python3 -m pip install --upgrade pip && \ + pip install torch==2.2.2+cu121 torchaudio==2.2.2+cu121 -f https://download.pytorch.org/whl/torch_stable.html + +# Copy requirements and install them +COPY requirements.txt /app/requirements.txt +WORKDIR /app +RUN pip install -r requirements.txt + +# Copy application code +COPY whisper_code.py whisper_code.py +COPY whisper_api_server.py whisper_api_server.py + +# Expose port +EXPOSE 8000 + +# Run API +CMD ["uvicorn", "whisper_api_server:app", "--host", "0.0.0.0", "--port", "8000", "--limit-max-requests", "500000000"] diff --git a/docs/whisper_transcription/Docker/requirements.txt b/docs/whisper_transcription/Docker/requirements.txt new file mode 100644 index 0000000..2352f28 --- /dev/null +++ b/docs/whisper_transcription/Docker/requirements.txt @@ -0,0 +1,37 @@ +# Core frameworks +fastapi==0.115.12 +uvicorn==0.34.2 + +# Whisper + speech processing +faster-whisper==1.1.1 +librosa==0.11.0 +pydub==0.25.1 +soundfile==0.13.1 +noisereduce==3.0.3 +demucs==4.0.1 +ffmpeg-python==0.2.0 + +# Diarization (PyAnnote) +pyannote.audio==3.3.2 +pyannote.core==5.0.0 +pyannote.database==5.1.3 +pyannote.metrics==3.2.1 +pyannote.pipeline==3.0.1 + +# Transformers + Summarization +transformers==4.51.3 +huggingface-hub==0.31.4 +sentencepiece==0.2.0 + +# Evaluation +jiwer==3.1.0 + +# Utilities +numpy==1.26.4 +scikit-learn==1.6.1 +requests==2.32.3 +tqdm==4.67.1 +typing_extensions==4.13.2 +pydantic==2.11.4 +python-multipart==0.0.20 +accelerate==1.7.0 \ No newline at end of file diff --git a/docs/whisper_transcription/Docker/whisper_api_server.py b/docs/whisper_transcription/Docker/whisper_api_server.py new file mode 100644 index 0000000..4487cea --- /dev/null +++ b/docs/whisper_transcription/Docker/whisper_api_server.py @@ -0,0 +1,187 @@ +import os +import tempfile +import json +import threading +import glob +from queue import Queue +from threading import Lock +from fastapi import FastAPI, UploadFile, File, Form +from fastapi.responses import StreamingResponse, JSONResponse, FileResponse +from whisper_code import transcribe_and_summarize, load_whisper_model_faster +import logging +import time + +app = FastAPI() + +# Global model cache and lock +model_cache = {} +model_lock = Lock() + +@app.post("/transcribe") +async def transcribe_audio_api( + audio_file: UploadFile = File(None), + audio_url: str = Form(None), + model: str = Form("base"), + summarized_model: str = Form("mistralai/Mistral-7B-Instruct-v0.1"), + denoise: bool = Form(False), + prop_decrease: float = Form(0.7), + summary: bool = Form(True), + speaker: bool = Form(False), + hf_token: str = Form(None), + max_speakers: int = Form(None), + streaming: bool = Form(False) +): + # Setup dynamic logging per request + from datetime import datetime + basename = "remote_audio" if not audio_file else os.path.splitext(os.path.basename(audio_file.filename))[0] + timestamp = datetime.now().strftime('%Y%m%d_%H%M%S') + log_file = f"transcription_log_{basename}_{timestamp}.log" + + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(levelname)s - %(message)s", + filename=log_file, + filemode="w", + force=True + ) + + # Load audio either from upload or URL + if audio_file: + temp_audio_path = tempfile.mktemp(suffix=f"_{audio_file.filename}") + with open(temp_audio_path, "wb") as f: + f.write(await audio_file.read()) + elif audio_url: + import requests + temp_audio_path = tempfile.mktemp(suffix=".m4a") + try: + with requests.get(audio_url, stream=True) as r: + r.raise_for_status() + with open(temp_audio_path, 'wb') as f: + for chunk in r.iter_content(chunk_size=1024*1024): + f.write(chunk) + + if r.status_code != 200: + raise Exception(f"Failed to download audio file: HTTP {r.status_code}") + except Exception as e: + from fastapi import HTTPException + raise HTTPException(status_code=400, detail=str(e)) + else: + from fastapi import HTTPException + raise HTTPException(status_code=400, detail="Must provide either audio_file or audio_url") + + output_dir = tempfile.mkdtemp() + + if model not in model_cache: + model_cache[model] = load_whisper_model_faster(model) + + whisper_model = model_cache[model] + + if streaming: + def generator(): + q = Queue() + q.put(f"data: {json.dumps({'meta': 'logfile_name', 'logfile': log_file})}\n\n") + + def api_callback(result): + if result.get("chunk_id", "").startswith("chunk_"): + logging.info(f"\n====== Streaming {result['chunk_id']} ======\n{result.get('text', '').strip()}\n") + elif result.get("chunk_id") == "summary": + logging.info("\n====== Final Summary ======\n") + logging.info(result.get("summary", "").strip()) + result["logfile"] = log_file + q.put(f"data: {json.dumps(result)}\n\n") + + def run_pipeline(): + try: + with model_lock: + transcribe_and_summarize( + path=temp_audio_path, + model_name=model, + output_dir=output_dir, + summarized_model_id=summarized_model, + denoise=denoise, + prop_decrease=prop_decrease, + summary=summary, + speaker=speaker, + hf_token=hf_token, + max_speakers=max_speakers, + streaming=True, + api_callback=api_callback, + model_instance=whisper_model + ) + except Exception as e: + import traceback + traceback.print_exc() + q.put(json.dumps({"error": f"Streaming transcription failed: {str(e)}"})) + finally: + q.put(None) + + threading.Thread(target=run_pipeline).start() + while True: + chunk = q.get() + if chunk is None: + break + yield chunk + + return StreamingResponse(generator(), media_type="text/event-stream") + + else: + try: + with model_lock: + transcribe_and_summarize( + path=temp_audio_path, + model_name=model, + output_dir=output_dir, + summarized_model_id=summarized_model, + denoise=denoise, + prop_decrease=prop_decrease, + summary=summary, + speaker=speaker, + hf_token=hf_token, + max_speakers=max_speakers, + streaming=False, + api_callback=None, + model_instance=whisper_model + ) + except Exception as e: + import traceback + traceback.print_exc() + return JSONResponse( + content={"error": f"Transcription failed: {str(e)}"}, + status_code=500 + ) + + try: + json_files = sorted( + glob.glob(os.path.join(output_dir, "*.json")), + key=os.path.getmtime, + reverse=True + ) + if not json_files: + raise FileNotFoundError("No output JSON file found.") + with open(json_files[0]) as f: + return JSONResponse(content=json.load(f)) + except Exception as e: + import traceback + traceback.print_exc() + return JSONResponse( + content={"error": f"Failed to read output JSON: {str(e)}"}, + status_code=500 + ) + +@app.get("/stream_log/{filename}") +def stream_log(filename: str): + log_path = os.path.join(".", filename) + if not os.path.exists(log_path): + return JSONResponse(content={"error": "Log file not found."}, status_code=404) + + def stream_lines(): + with open(log_path, "r") as f: + f.seek(0, 2) + while True: + line = f.readline() + if line: + yield f"data: {line.strip()}\n\n" + else: + time.sleep(0.2) + + return StreamingResponse(stream_lines(), media_type="text/event-stream") diff --git a/docs/whisper_transcription/Docker/whisper_code.py b/docs/whisper_transcription/Docker/whisper_code.py new file mode 100644 index 0000000..4e83282 --- /dev/null +++ b/docs/whisper_transcription/Docker/whisper_code.py @@ -0,0 +1,1013 @@ +import re +import os +import torch +#import whisper +import tempfile +import wave +import json +import librosa +import noisereduce as nr +import argparse +from pydub import AudioSegment, silence +from transformers import AutoTokenizer, AutoModelForCausalLM +import logging +from collections import OrderedDict +from jiwer import wer +from pyannote.audio import Pipeline +import soundfile as sf +import shutil +import subprocess +from datetime import datetime, timedelta +from collections import Counter +from faster_whisper import WhisperModel +from pydub.utils import make_chunks +from concurrent.futures import ThreadPoolExecutor, as_completed +from difflib import get_close_matches +import threading +thread_local = threading.local() +import traceback + + +SUPPORTED_EXTENSIONS = ('.flac', '.wav', '.mp3', '.m4a', '.aac', '.ogg', '.webm', '.opus', '.mp4', '.mov', '.mkv', '.avi') +CHUNK_FOLDER = "chunks" + +# Detects GPU type and suggests best Whisper model and token limit accordingly +def detect_gpu_and_optimize(): + """ + Detect the current GPU and return Whisper model + token limit. + Returns: (gpu_name, whisper_model, token_limit) + """ + if not torch.cuda.is_available(): + return "cpu", "small", 30 + device = torch.cuda.current_device() + name = torch.cuda.get_device_name(device).lower() + if "a10" in name: + return "a10", "medium", 60 + elif "a100" in name: + return "a100", "large", 120 + elif "h100" in name: + return "h100", "turbo", 180 + return "unknown_gpu", "base", 60 + +def load_whisper_model_faster(model_name, gpu_id=0): + """ + Loads a WhisperModel from faster-whisper for a specified GPU. + + Args: + model_name (str): Name of the Whisper model to load. + gpu_id (int): The GPU index to load the model on. + + Returns: + WhisperModel: An instance of the loaded WhisperModel ready for transcription. + """ + logging.info(f"[load_whisper_model_faster] Loading WhisperModel '{model_name}' on GPU {gpu_id}") + return WhisperModel( + model_name, + device="cuda", + device_index=gpu_id, + compute_type="float16", + cpu_threads=8 + ) + + +# Converts input audio/video file to 16kHz, mono WAV format (Whisper compatible) +def convert_to_wav(input_path): + """ + Converts any audio/video file into 16kHz, mono-channel WAV format using ffmpeg. + + Args: + input_path (str): Path to the input file. + + Returns: + str: Path to the converted WAV file or None on failure. + """ + output_path = tempfile.mktemp(suffix=".wav") + try: + cmd = [ + "ffmpeg", "-y", "-threads", "4", + "-i", input_path, + "-ac", "1", + "-ar", "16000", + "-sample_fmt", "s16", + output_path + ] + subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=True) + except Exception as e: + logging.error(f"Chunk processing failed: {e}") + return None + logging.error(f"ffmpeg conversion failed on {input_path}: {e}") + raise + return output_path + + + + + +def run_diarization(full_audio_path, hf_token, max_speakers=None): + """ + Runs speaker diarization on the full audio file using PyAnnote's pre-trained pipeline. + + This function loads a HuggingFace diarization pipeline and applies it to the audio, + optionally constraining the number of speakers. It logs and saves the diarization + results to a debug JSON file for inspection. + + Args: + full_audio_path (str): Path to the input audio file (must be supported by PyAnnote). + hf_token (str): HuggingFace access token for loading the diarization model. + max_speakers (int, optional): If provided, constrains diarization to a fixed number of speakers. + + Returns: + list[Tuple[Segment, None, str]]: A list of diarized segments, each represented as a tuple: + - track (Segment): Start and end time of the segment. + - _: Reserved (not used, always None). + - label (str): Speaker label (e.g., "SPEAKER_00", "SPEAKER_01"). + """ + + logging.info(f"Running speaker diarization on: {full_audio_path}") + try: + pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1", use_auth_token=hf_token) + if torch.cuda.is_available(): + pipeline.to(torch.device("cuda")) + + if max_speakers: + diarization = pipeline({"audio": full_audio_path}, num_speakers=max_speakers) + else: + diarization = pipeline({"audio": full_audio_path}) + + global_segments = list(diarization.itertracks(yield_label=True)) + + if not global_segments: + logging.warning("Diarization returned an empty result.") + else: + output_dir = os.path.dirname(full_audio_path) + debug_path = os.path.join(output_dir, "diarization_debug.json") + debug_dump = [ + {"start": float(track.start), "end": float(track.end), "label": str(label)} + for track, _, label in global_segments + ] + with open(debug_path, "w") as df: + json.dump(debug_dump, df, indent=2) + logging.info(f"Diarization returned {len(global_segments)} segments. Dumped to {debug_path}.") + for track, _, label in global_segments: + logging.info(f"Diarized segment: {track.start:.2f}s - {track.end:.2f}s -> {label}") + + return global_segments + except Exception as e: + logging.error(f"Diarization failed: {e}") + return [] + + + + +def assign_speakers_to_segments_from_global(chunk_path, segments, diarized_segments, chunk_offset_seconds): + """ + Assign speaker labels to each Whisper segment by comparing midpoints to global diarization tracks. + + Args: + chunk_path (str): Path to the audio chunk (not used directly but kept for interface consistency). + segments (list): List of Whisper segments (dict or Segment object with 'start'/'end' or .start/.end). + diarized_segments (list): List of (track, _, label) from pyannote speaker diarization. + chunk_offset_seconds (float): Offset of this chunk in full audio timeline. + + Returns: + list[str]: List of speaker labels (e.g., "Speaker 1", "Speaker 2", ...) + """ + assigned_speakers = [] + + for seg in segments: + # Support both dict-style and object-style segments + try: + seg_start = seg['start'] + seg_end = seg['end'] + except TypeError: + seg_start = seg.start + seg_end = seg.end + + midpoint = float((seg_start + seg_end) / 2 + chunk_offset_seconds) + match = None + + # Match midpoint against global diarized tracks + for track, _, label in diarized_segments: + if not (float(track.end) < seg_start + chunk_offset_seconds or float(track.start) > seg_end + chunk_offset_seconds): + match = label + break + + if not match: + match = "SPEAKER_00" + + # Convert SPEAKER_00 → Speaker 1, etc. + if match.startswith("SPEAKER_"): + try: + spk_number = int(match.split("_")[1]) + assigned_speakers.append(f"Speaker {spk_number + 1}") + except (IndexError, ValueError): + assigned_speakers.append("Speaker 1") + else: + assigned_speakers.append(match) + + logging.info(f"Speaker assignment breakdown: {Counter(assigned_speakers)}") + return assigned_speakers + + + +# Applies noise reduction using noisereduce (conservative settings) +def denoise_audio(input_file, output_file, prop_decrease=0.7): + """ + Applies denoising using Demucs and noise reduction techniques. + + Args: + input_file (str): Path to the input noisy audio. + output_file (str): Output path for the denoised audio. + prop_decrease (float): Aggressiveness of denoising. + + Returns: + None + """ + temp_dir = tempfile.mkdtemp() + try: + logging.info(f"Running Demucs on: {input_file}") + demucs_cmd = f"python3 -m demucs.separate -d cuda -n htdemucs --two-stems=vocals -o \"{temp_dir}\" \"{input_file}\"" + subprocess.run(demucs_cmd, shell=True, check=True) + + base = os.path.splitext(os.path.basename(input_file))[0] + vocals_path = os.path.join(temp_dir, "htdemucs", base, "vocals.wav") + + if not os.path.exists(vocals_path): + raise FileNotFoundError(f"Demucs output not found: {vocals_path}") + + y, sr = librosa.load(vocals_path, sr=None) + y_denoised = nr.reduce_noise(y=y, sr=sr, prop_decrease=prop_decrease, stationary=False) + sf.write(output_file, y_denoised, sr) + logging.info(f"Denoised file saved to: {output_file}") + except Exception as e: + logging.error(f"Denoising failed on {input_file}: {e}. Falling back to original.") + shutil.copy(input_file, output_file) # FALLBACK + finally: + shutil.rmtree(temp_dir, ignore_errors=True) + + + +# Calculates duration (in seconds) of the audio file using wave or pydub +def get_audio_duration(path): + """ + Calculates duration of a WAV or general audio file. + + Args: + path (str): File path to audio. + + Returns: + float: Duration in seconds. + """ + try: + if path.endswith('.wav'): + with wave.open(path, 'r') as f: + frames = f.getnframes() + rate = f.getframerate() + return frames / float(rate) + else: + audio = AudioSegment.from_file(path) + return len(audio) / 1000.0 # seconds + except Exception as e: + logging.error(f"Chunk processing failed: {e}") + return None + logging.warning(f"Failed to get duration of {path}: {e}") + return 0 + + +def fallback_fixed_chunks(audio, chunk_length_ms=30000): + """ + Splits audio into fixed-length chunks (30s default). + + Args: + audio (AudioSegment): Loaded audio object. + chunk_length_ms (int): Length in milliseconds. + + Returns: + list[AudioSegment]: List of chunks. + """ + return make_chunks(audio, chunk_length_ms) + + +def smart_chunk_audio(audio_path, output_dir, model, language="en"): + """ + Performs intelligent audio chunking based on speech segments detected by a transcription model. + + This function uses a transcription model (e.g., Whisper or faster-whisper) to identify + speech segments in the audio file. Each segment is then extracted as a separate audio chunk + and saved to the specified output directory. + + Args: + audio_path (str): Path to the input audio file (any format supported by pydub/ffmpeg). + output_dir (str): Directory where chunked audio files will be saved. + model: Transcription model object with a `.transcribe()` method (must yield segments with `.start` and `.end` attributes). + language (str, optional): Language code to guide transcription. Defaults to "en". + + Returns: + list[Tuple[str, float]]: List of tuples containing: + - chunk_path (str): Path to the exported audio chunk. + - start_offset (float): Start time of the chunk in seconds. + """ + logging.info("Running smart chunking...") + + segments_generator, _ = model.transcribe(audio_path, language=language, beam_size=5, word_timestamps=True) + segments = list(segments_generator) + audio = AudioSegment.from_file(audio_path) + + chunk_infos = [] # List of (chunk_path, start_offset) + for i, seg in enumerate(segments): + start = seg.start + end = seg.end + chunk_audio = audio[start * 1000:end * 1000] # Convert to ms + chunk_path = os.path.join(output_dir, f"chunk_{i:03d}.wav") + chunk_audio.export(chunk_path, format="wav") + chunk_infos.append((chunk_path, start)) + logging.info(f"Exported chunk: {chunk_path} ({start:.2f}s → {end:.2f}s)") + + return chunk_infos + + + +# Converts and transcribes an audio file using Whisper model +def transcribe_file(model, audio_path): + """ + Transcribes a WAV file using a Whisper model. + + Args: + model (WhisperModel): Loaded Whisper model. + audio_path (str): Path to WAV file. + beam_size (int): Beam size for decoding. + language (str): Optional forced language. + + Returns: + tuple[list[dict], str]: List of segments and detected language. + """ + try: + wav_path = convert_to_wav(audio_path) + + segments_generator, info = model.transcribe(wav_path, beam_size=1, language=None) + segments = list(segments_generator) + text = " ".join([seg.text.strip() for seg in segments]) + os.remove(wav_path) + + detected_lang = info.language + return text.strip(), detected_lang, segments + except Exception as e: + logging.error(f"Chunk processing failed: {e}") + return None + logging.error(f"Failed: {audio_path} — {e}") + return "", "unknown", [] + + +# Loads the summarization model (default: Mistral) from HuggingFace +def load_mistral_model(model_id, hf_token): + """ + Loads a HuggingFace LLM model and tokenizer. + + Args: + model_name (str): Name of model. + + Returns: + tuple: (model, tokenizer) + """ + if os.path.isdir(model_id): + logging.info(f"Loading summarizer model from local path: {model_id}") + tokenizer = AutoTokenizer.from_pretrained(model_id, use_auth_token=hf_token) + model = AutoModelForCausalLM.from_pretrained( + model_id, + torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32, + device_map="auto", + use_auth_token=hf_token + ) + else: + logging.info(f"Loading summarizer model from HuggingFace: {model_id}") + tokenizer = AutoTokenizer.from_pretrained(model_id, use_auth_token=hf_token) + model = AutoModelForCausalLM.from_pretrained( + model_id, + torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32, + device_map="auto", + use_auth_token=hf_token + ) + return model, tokenizer + + +# Splits long transcript into smaller LLM-compatible chunks based on token count +def split_into_chunks(text, tokenizer, max_tokens=3000): + """ + Splits long text into overlapping token-aware chunks. + + Args: + text (str): Input string. + tokenizer: HuggingFace tokenizer. + max_tokens (int): Max tokens per chunk. + stride (int): Overlap for continuity. + + Returns: + list[str]: List of text chunks. + """ + words = text.split() + chunks = [] + current = [] + for word in words: + current.append(word) + if len(tokenizer.encode(" ".join(current))) > max_tokens: + chunks.append(" ".join(current[:-1])) + current = [word] + if current: + chunks.append(" ".join(current)) + return chunks + + +# Summarizes a chunk of transcript using the loaded summarizer model +def summarize_with_mistral(model, tokenizer, text): + """ + Summarizes text using an LLM like Mistral. + + Args: + model: HuggingFace model. + tokenizer: Corresponding tokenizer. + text (str): Text to summarize. + + Returns: + str: Summary string. + """ + prompt = f"[INST] Summarize the following meeting transcript into a concise list of key points, decisions, and action items:\n\n{text} [/INST]" + inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=4096).to(model.device) + with torch.no_grad(): + output = model.generate(**inputs, max_new_tokens=512) + + result = tokenizer.decode(output[0], skip_special_tokens=True) + + # Strip echoed prompt (everything before [/INST]) + if "[/INST]" in result: + result = result.split("[/INST]", 1)[-1].strip() + + return result + + +# Main pipeline for one file: validate → denoise → chunk → transcribe → summarize → save outputs + +def assign_speakers_to_segments(converted_path, segments, hf_token, max_speakers=None): + """ + Runs full-audio diarization and assigns speakers to Whisper segments using midpoint matching. + + Args: + converted_path (str): Path to audio file. + segments (list): List of Whisper segments. + hf_token (str): HuggingFace access token. + max_speakers (int): Optional number of speakers to constrain diarization. + + Returns: + list[str]: Speaker label per segment, normalized as "Speaker 1", "Speaker 2", etc. + """ + logging.info(f"Running speaker diarization on: {converted_path}") + + try: + pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1", use_auth_token=hf_token) + if torch.cuda.is_available(): + pipeline.to(torch.device("cuda")) + + global_diarized_segments = run_diarization(converted_path, hf_token, max_speakers, output_dir) + + diarized_segments = list(diarization.itertracks(yield_label=True)) + logging.info(f"First 5 diarized segments:") + for i, (track, _, label) in enumerate(diarized_segments[:5]): + logging.info(f" {i}: {track.start:.2f} - {track.end:.2f} --> {label}") + if not diarized_segments: + logging.warning("No diarized speaker segments were found.") + return ["Speaker 1"] * len(segments) + + except Exception as e: + logging.error(f"Diarization pipeline failed: {e}") + return ["Speaker 1"] * len(segments) + + logging.info(f"Total diarized segments: {len(diarized_segments)}") + + speaker_mapping = {} + next_speaker_id = 1 + speaker_map = [] + + for segment in segments: + mid_point = (segment["start"] + segment["end"]) / 2.0 + assigned_speaker = "Speaker 1" + for turn, _, label in diarized_segments: + if turn.start <= mid_point <= turn.end: + if label not in speaker_mapping: + speaker_mapping[label] = f"Speaker {next_speaker_id}" + next_speaker_id += 1 + assigned_speaker = speaker_mapping[label] + break + speaker_map.append(assigned_speaker) + logging.info(f"Segment midpoint={mid_point:.2f} assigned to: {assigned_speaker} — {segment['text'][:30]}") + + logging.info(f"Speaker assignment breakdown: {Counter(speaker_map)}") + return speaker_map + + + +def transcribe_and_summarize( + path, + model_name="base", + output_dir="outputs", + summarized_model_id="mistralai/Mistral-7B-Instruct-v0.1", + ground_truth_path=None, + denoise=False, + prop_decrease=0.7, + summary=True, + speaker=False, + hf_token=None, + session_timestamp=None, + max_speakers=None, + streaming=False, + api_callback=None, + model_instance=None): + """ + Args: + path (str): Path to input audio file. + model_name (str): Whisper model name. + output_dir (str): Directory to save outputs. + denoise (bool): Whether to apply denoising. + summary (bool): Whether to generate summaries. + speaker (bool): Whether to run speaker diarization. + streaming (bool): Whether to stream results. + hf_token (str): HuggingFace token. + max_speakers (int): Max speakers for diarization. + + Returns: + None + + """ + os.makedirs(output_dir, exist_ok=True) + + if not isinstance(session_timestamp, str) or not re.match(r"^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}$", session_timestamp): + session_timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + base_time = datetime.strptime(session_timestamp, "%Y-%m-%d %H:%M:%S") + + if denoise: + denoised_path = tempfile.mktemp(suffix=".wav") + logging.info(f"Applying noise reduction to: {path}") + denoise_audio(path, denoised_path, prop_decrease=prop_decrease) + path = denoised_path + + converted_path = convert_to_wav(path) + if not converted_path or not os.path.exists(converted_path): + logging.error("Conversion failed — aborting transcription.") + return + + global_diarized_segments = [] + if speaker: + try: + logging.info("Running global speaker diarization...") + global_diarized_segments = run_diarization(converted_path, hf_token, max_speakers) + + for i, (track, _, label) in enumerate(global_diarized_segments): + logging.info(f"[DIAR] {i}: {track.start:.2f}s - {track.end:.2f}s --> {label}") + logging.info(f"Global diarization complete. {len(global_diarized_segments)} segments.") + except Exception as e: + logging.warning(f"Global diarization failed on {converted_path}: {e}") + global_diarized_segments = [] + + chunks_dir = os.path.join(output_dir, "chunks") + os.makedirs(chunks_dir, exist_ok=True) + + language = "en" # Default fallback; optionally auto-detect later + + # FIXED: Build whisper_models BEFORE calling smart_chunk_audio + num_gpus = torch.cuda.device_count() + if num_gpus == 0: + raise RuntimeError("No CUDA devices available") + gpus = list(range(num_gpus)) + + if model_instance is not None: + whisper_models = {0: model_instance} + else: + whisper_models = { + gpu_id: WhisperModel( + model_name, + device="cuda", + device_index=gpu_id, + compute_type="float16", + cpu_threads=4 + ) for gpu_id in gpus + } + + model = whisper_models[0] + chunk_infos = smart_chunk_audio(converted_path, chunks_dir, model, language) + + base = os.path.splitext(os.path.basename(path))[0] + structured_chunks = {} + temp_transcript_path = os.path.join(output_dir, f"{base}_partial_transcript.txt") + if os.path.exists(temp_transcript_path): + os.remove(temp_transcript_path) + logging.info(f"Total chunks: {len(chunk_infos)}") + + failed_chunks = [] + + + def clean_audio_with_ffmpeg(input_path, cleaned_path): + """ + Cleans audio using ffmpeg (mono, 16-bit PCM, 16kHz). + + Args: + input_path (str): Path to input audio. + cleaned_path (str): Output path for cleaned audio. + + Returns: + None + """ + try: + cmd = [ + "ffmpeg", "-y", "-i", input_path, + "-ar", "16000", "-ac", "1", "-c:a", "pcm_s16le", + cleaned_path + ] + subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + return cleaned_path + except subprocess.CalledProcessError: + logging.error(f"FFmpeg failed to clean {input_path}") + return None + + + + def transcribe_on_gpu(gpu_id, chunk_path, chunk_offset_seconds, global_diarized_segments, model_name): + """ + Transcribes a single chunk using a specified GPU, optionally performing speaker diarization. + + Args: + gpu_id (int): GPU index to use for transcription. + chunk_path (str): Path to audio chunk file. + chunk_offset_seconds (float): Offset of chunk in original audio. + global_diarized_segments (list): List of diarized segments (start, end, label). + model_name (str): Name of Whisper model to load. + + Returns: + dict: Dictionary containing transcription, chunk path, and speaker labels. + """ + try: + torch.cuda.set_device(gpu_id) + + # Lazily initialize model per thread + if not hasattr(thread_local, "model"): + thread_local.model = WhisperModel( + model_name, + device="cuda", + device_index=gpu_id, + compute_type="float16" + ) + logging.info(f"WhisperModel '{model_name}' loaded on GPU {gpu_id} for thread {threading.get_ident()}") + + model = thread_local.model + + logging.info(f"Processing audio with duration {AudioSegment.from_file(chunk_path).duration_seconds:.3f} seconds") + result = transcribe_file(model, chunk_path) + + if not result or not isinstance(result, tuple) or len(result) != 3: + logging.error(f"Invalid result returned by transcribe_file() for chunk {chunk_path}: {result}") + return None + + chunk_text, lang, segments = result + + if not isinstance(segments, list): + logging.error(f"Expected segments to be a list, got {type(segments)} in chunk {chunk_path}") + return None + + assigned_speakers = assign_speakers_to_segments_from_global( + chunk_path, + segments, + global_diarized_segments, + chunk_offset_seconds + ) + + + + text_with_speakers = [] + for seg, speaker in zip(segments, assigned_speakers): + seg_text = seg.text.strip() + raw_speaker = str(speaker) + + try: + start_time = float(seg.start) + except (ValueError, TypeError): + logging.error(f"Invalid seg.start: {seg.start} ({type(seg.start)})") + start_time = 0.0 + + try: + base_time = datetime.strptime(session_timestamp, "%Y-%m-%d %H:%M:%S") + except (ValueError, TypeError) as e: + logging.error(f"Invalid session_timestamp: {session_timestamp} ({type(session_timestamp)}): {e}") + base_time = datetime.fromtimestamp(0) + + timestamp = base_time + timedelta(seconds=start_time) + timestamp_str = timestamp.strftime("%Y-%m-%d %H:%M:%S") + + text_with_speakers.append(f"[{timestamp_str}] {raw_speaker}: {seg_text}") + + + + final_chunk_transcript = "\n".join(text_with_speakers) + + return { + "chunk_path": chunk_path, + "offset_seconds": chunk_offset_seconds, + "language": lang, + "segments": segments, + "speaker_labels": assigned_speakers, + "transcript": final_chunk_transcript + } + + except Exception as e: + logging.error(f"Chunk {chunk_path} failed on GPU {gpu_id}: {e}") + return None + + + structured_chunks = OrderedDict() + full_transcript = "" + + with ThreadPoolExecutor(max_workers=len(gpus)) as executor: + future_to_idx = {} + + for i, (chunk_path, chunk_offset_seconds) in enumerate(chunk_infos): + gpu_id = gpus[i % len(gpus)] + future = executor.submit( + transcribe_on_gpu, + gpu_id, + chunk_path, + chunk_offset_seconds, + global_diarized_segments, + model_name + ) + + + future_to_idx[future] = (i, chunk_path) + + for future in as_completed(future_to_idx): + idx, chunk = future_to_idx[future] + result = future.result() + if result: + logging.info(f"Got result for chunk: {result['chunk_path']}") + else: + logging.warning('Future returned None (likely failed transcription)') + if result is None: + continue + + + chunk_filename = os.path.basename(result["chunk_path"]) + text_with_speakers = result["transcript"] + lang = result["language"] + segments = result["segments"] + + + chunk_id = f"{idx+1:03d}" + + # this is new — gather full transcript across all chunks + full_transcript += text_with_speakers.strip() + "\n" + + structured_chunks[chunk_filename] = { + "chunk_id": chunk_id, + "transcript": text_with_speakers.strip(), + + "segments": [ + { + "text": seg.text.strip(), + "timestamp": ( + datetime.strptime(session_timestamp, "%Y-%m-%d %H:%M:%S") + timedelta(seconds=seg.start) + ).strftime("%Y-%m-%d %H:%M:%S"), + "speaker": speaker + } + for seg, speaker in zip(segments, result["speaker_labels"]) + ], + + "language": lang + } + + if streaming: + logging.info(f"\n====== Streaming Chunk {chunk_id} ======\n{text_with_speakers.strip()}\n") + + + # Save transcript to temp file + try: + with open(temp_transcript_path, "w") as f: + f.write(full_transcript.strip()) + except Exception as e: + logging.error(f"Failed to write temp transcript: {e}") + return + + # Ensure temp transcript file exists and re-read + if not os.path.exists(temp_transcript_path): + logging.error(f"Transcript file not found: {temp_transcript_path}") + return + + with open(temp_transcript_path, "r") as tf: + full_transcript = tf.read() + + if not full_transcript.strip(): + logging.warning(" Skipping save: empty transcript detected.") + return + + # Define final output paths + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + txt_path = os.path.join(output_dir, f"{base}_all_transcripts_{timestamp}.txt") + json_path = os.path.join(output_dir, f"{base}_all_transcripts_{timestamp}.json") + + # Write transcript to final TXT file + with open(txt_path, "w") as f: + f.write(full_transcript) + logging.info(f"Transcript saved to: {txt_path}") + + try: + logging.info(f"CUDA available before summarization: {torch.cuda.is_available()}") + model_llm, tokenizer = load_mistral_model(summarized_model_id, hf_token=hf_token) + except Exception as e: + logging.error(f"Failed to load summarization model: {e}") + model_llm = tokenizer = None + + # Generate summary if requested + if not model_llm: + combined_summary = "(Summarization failed due to model load issue.)" + elif len(full_transcript.split()) < 50: + logging.info("Skipping summarization: transcript too short.") + combined_summary = "(Transcript too short for meaningful summarization.)" + else: + summary_chunks = split_into_chunks(full_transcript, tokenizer) + summaries = [] + for i, chunk in enumerate(summary_chunks): + logging.info(f"Summarizing chunk {i+1}/{len(summary_chunks)}") + summary = summarize_with_mistral(model_llm, tokenizer, chunk) + summary = summary.replace(chunk.strip(), '').strip() + summaries.append(summary) + combined_summary = "\n\n---\n\n".join(summaries) + # Safeguard against None summary + if not isinstance(combined_summary, str) or not combined_summary.strip(): + combined_summary = "(Summary generation failed or was skipped.)" + + # Write summary to TXT file + try: + with open(txt_path, "a") as f: + f.write("\n\n====== Summary ======\n\n") + f.write(combined_summary.strip()) + logging.info(" Summary successfully appended to transcript TXT.") + except Exception as e: + logging.error(f" Failed to write summary to TXT: {e}") + + # Write final JSON + try: + ordered_output = OrderedDict() + ordered_output["chunks"] = structured_chunks + ordered_output["transcript"] = full_transcript.strip() + ordered_output["summary"] = combined_summary.strip() + with open(json_path, "w") as f: + json.dump(ordered_output, f, indent=2) + logging.info(f"Final JSON saved to: {json_path}") + except Exception as e: + logging.error(f" Failed to save JSON: {e}") + + if streaming and api_callback: + logging.info(f"\n====== Streaming Chunk {chunk_id} ======\n{text_with_speakers.strip()}\n") + api_callback({ + "chunk_id": "summary", + "transcript": full_transcript.strip(), + "summary": combined_summary.strip(), + }) + + if ground_truth_path and os.path.isfile(ground_truth_path): + with open(ground_truth_path, 'r') as gt: + reference = gt.read().strip() + error = wer(reference, full_transcript.strip()) + logging.info(f"WER (Word Error Rate) against ground truth: {error:.2%}") + elif ground_truth_path: + logging.warning(f"Ground truth file not found at: {ground_truth_path}") + + if failed_chunks: + failed_path = os.path.join(output_dir, f"{base}_failed_chunks.txt") + with open(failed_path, "w") as f: + for path in failed_chunks: + f.write(path + "\n") + logging.warning(f"{len(failed_chunks)} chunks failed. Paths saved to: {failed_path}") + + + + +local_session_timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + +# Handles single file or directory input, validates each audio, and runs full pipeline +def process_audio_batch(input_path, model_name, output_dir, summarized_model_id, ground_truth_path=None, denoise=False, prop_decrease=0.7, summary=True, speaker=False, hf_token=None, max_speakers=None, streaming=False, api_callback=None): + """ + Processes a file or a folder of audio through the full pipeline. + + Args: + input_path (str): File or directory path. + model (str): Whisper model name. + output_dir (str): Where to write output. + summary (bool): Enable summarization. + speaker (bool): Enable speaker diarization. + denoise (bool): Enable denoising. + streaming (bool): Enable chunk-wise streaming. + hf_token (str): HuggingFace token. + max_speakers (int): Maximum number of speakers. + + Returns: + None + """ + session_timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + timestamp = datetime.now().strftime('%Y%m%d_%H%M%S') + + if not output_dir: + output_dir = os.path.join("outputs", f"run_{timestamp}") + os.makedirs(output_dir, exist_ok=True) + + # Initialize logging + log_file = os.path.join(output_dir, f"transcription_log_{timestamp}.log") + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(levelname)s - %(message)s", + handlers=[ + logging.FileHandler(log_file), + logging.StreamHandler() + ] + ) + + # Get list of files + if os.path.isdir(input_path): + audio_files = [ + os.path.join(dp, f) + for dp, dn, filenames in os.walk(input_path) + for f in filenames if f.endswith(SUPPORTED_EXTENSIONS) + ] + else: + audio_files = [input_path] if input_path.endswith(SUPPORTED_EXTENSIONS) else [] + + if not audio_files: + logging.warning(f"No valid audio files found in input: {input_path}") + return + + logging.info(f"Total audio files to process: {len(audio_files)}") + + for audio_file in audio_files: + logging.info(f"Validating: {audio_file}") + if not os.path.exists(audio_file): + logging.warning(f"File does not exist: {audio_file}") + continue + + duration = get_audio_duration(audio_file) + logging.info(f"Duration of {audio_file}: {duration:.2f}s") + if duration < 1: + logging.warning(f"Skipping {audio_file}: too short (<1s)") + continue + elif duration > 3600: + logging.warning(f"{audio_file} is too long (>60 min), may require chunking") + + logging.info(f"Processing: {audio_file}") + temp_transcript_path = os.path.join(output_dir, f"{os.path.splitext(os.path.basename(audio_file))[0]}_partial_transcript.txt") + try: + transcribe_and_summarize( + audio_file, + model_name, + output_dir, + summarized_model_id, + model_instance=None, + ground_truth_path=ground_truth_path, + denoise=denoise, + prop_decrease=prop_decrease, + summary=summary, + speaker=speaker, + hf_token=hf_token, + session_timestamp=session_timestamp, + max_speakers=max_speakers, + streaming=streaming, + api_callback=api_callback + ) + + except Exception as e: + logging.error(f"Failed on {audio_file}: {e}") + logging.error(traceback.format_exc()) + continue # or pass if this is not inside a loop + + +# Script entry: parse CLI args and run processing pipeline +if __name__ == "__main__": + # Entry point: parse args and run batch + parser = argparse.ArgumentParser() + parser.add_argument("--input", required=True, help="Path to audio file or directory") + parser.add_argument("--model", help="Override model selection", default=None) + parser.add_argument("--output-dir", help="Directory to store outputs", default=None) + parser.add_argument("--summarized-model", help="HuggingFace model for summarization or provide the path of you own model", default="mistralai/Mistral-7B-Instruct-v0.1") + parser.add_argument("--ground-truth", help="Path to ground truth transcript file for WER evaluation", default=None) + parser.add_argument("--denoise", action="store_true", help="Apply noise reduction to audio before transcription") + parser.add_argument("--prop-decrease", type=float, default=0.7, help="Noise reduction aggressiveness (0.0 to 1.0)") + parser.add_argument("--summary", action="store_true", help="Include summarization after transcription") + parser.add_argument("--speaker", action="store_true", help="Enable speaker diarization using pyannote.audio") + parser.add_argument("--hf-token", help="Hugging Face token for diarization", default=None) + parser.add_argument("--max-speakers", type=int, help="Maximum number of speakers to force diarization into", default=None) + parser.add_argument("--streaming", action="store_true", help="Stream transcript results chunk by chunk") + + + args = parser.parse_args() + + gpu, suggested_model, _ = detect_gpu_and_optimize() + model_name = args.model if args.model else suggested_model + print(f"GPU Detected: {gpu.upper()} — Using model: {model_name}") + process_audio_batch( + args.input, model_name, args.output_dir, args.summarized_model, + args.ground_truth, args.denoise, args.prop_decrease, args.summary, + args.speaker, args.hf_token , args.max_speakers ,args.streaming, api_callback=None +) + diff --git a/docs/whisper_transcription/Examples/Test1/test.wav b/docs/whisper_transcription/Examples/Test1/test.wav new file mode 100644 index 0000000..3c6f1da Binary files /dev/null and b/docs/whisper_transcription/Examples/Test1/test.wav differ diff --git a/docs/whisper_transcription/Examples/Test1/test_all_transcripts_20250601_201349.txt b/docs/whisper_transcription/Examples/Test1/test_all_transcripts_20250601_201349.txt new file mode 100644 index 0000000..c0f631f --- /dev/null +++ b/docs/whisper_transcription/Examples/Test1/test_all_transcripts_20250601_201349.txt @@ -0,0 +1,21 @@ +[2025-06-01 20:13:40] Speaker 2: So, Aaron, in your email you said you wanted to talk about the exam. +[2025-06-01 20:13:40] Speaker 1: Yeah, um, I've just never taken a class with so many different readings. +[2025-06-01 20:13:40] Speaker 1: I've managed to keep up with all the assignments, but I'm not sure how to... +[2025-06-01 20:13:45] Speaker 1: How to... +[2025-06-01 20:13:40] Speaker 2: How to review everything. +[2025-06-01 20:13:40] Speaker 1: Yeah, in other classes I've had, there's usually just one book to review, not three different books, plus all those other text excerpts and video... + +====== Summary ====== + +Key points: + +* Speaker 1 is struggling to keep up with the readings in a class with multiple books and other materials. +* Speaker 2 suggests reviewing everything to prepare for the exam. + +Decisions: + +* None made during the meeting. + +Action items: + +* Speaker 1 needs to find a strategy for reviewing all the materials and preparing for the exam. \ No newline at end of file diff --git a/docs/whisper_transcription/Examples/Test1/transcription_log_20250601_201340.log b/docs/whisper_transcription/Examples/Test1/transcription_log_20250601_201340.log new file mode 100644 index 0000000..670d700 --- /dev/null +++ b/docs/whisper_transcription/Examples/Test1/transcription_log_20250601_201340.log @@ -0,0 +1,99 @@ +2025-06-01 20:13:40,071 - INFO - Total audio files to process: 1 +2025-06-01 20:13:40,071 - INFO - Validating: /home/ubuntu/whisper/dd_speaker/test.wav +2025-06-01 20:13:40,072 - INFO - Duration of /home/ubuntu/whisper/dd_speaker/test.wav: 27.30s +2025-06-01 20:13:40,072 - INFO - Processing: /home/ubuntu/whisper/dd_speaker/test.wav +2025-06-01 20:13:40,150 - INFO - Running global speaker diarization... +2025-06-01 20:13:40,150 - INFO - Running speaker diarization on: /tmp/tmp96aglawi.wav +2025-06-01 20:13:40,562 - DEBUG - Registered checkpoint save hook for _speechbrain_save +2025-06-01 20:13:40,562 - DEBUG - Registered checkpoint load hook for _speechbrain_load +2025-06-01 20:13:40,562 - DEBUG - Registered checkpoint save hook for save +2025-06-01 20:13:40,562 - DEBUG - Registered checkpoint load hook for load +2025-06-01 20:13:40,707 - INFO - Applied quirks (see `speechbrain.utils.quirks`): [disable_jit_profiling, allow_tf32] +2025-06-01 20:13:40,708 - INFO - Excluded quirks specified by the `SB_DISABLE_QUIRKS` environment (comma-separated list): [] +2025-06-01 20:13:40,708 - DEBUG - Registered checkpoint save hook for _save +2025-06-01 20:13:40,708 - DEBUG - Registered checkpoint load hook for _recover +2025-06-01 20:13:42,919 - INFO - Diarization returned 10 segments. Dumped to output_test3/diarization_debug.json. +2025-06-01 20:13:42,919 - INFO - Diarized segment: 0.32s - 4.18s -> SPEAKER_01 +2025-06-01 20:13:42,919 - INFO - Diarized segment: 4.89s - 9.92s -> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - Diarized segment: 9.97s - 10.04s -> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - Diarized segment: 10.09s - 12.55s -> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - Diarized segment: 12.97s - 14.29s -> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - Diarized segment: 14.97s - 15.56s -> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - Diarized segment: 16.21s - 18.07s -> SPEAKER_01 +2025-06-01 20:13:42,920 - INFO - Diarized segment: 18.42s - 19.05s -> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - Diarized segment: 19.35s - 20.87s -> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - Diarized segment: 21.07s - 27.40s -> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - [DIAR] 0: 0.32s - 4.18s --> SPEAKER_01 +2025-06-01 20:13:42,920 - INFO - [DIAR] 1: 4.89s - 9.92s --> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - [DIAR] 2: 9.97s - 10.04s --> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - [DIAR] 3: 10.09s - 12.55s --> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - [DIAR] 4: 12.97s - 14.29s --> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - [DIAR] 5: 14.97s - 15.56s --> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - [DIAR] 6: 16.21s - 18.07s --> SPEAKER_01 +2025-06-01 20:13:42,920 - INFO - [DIAR] 7: 18.42s - 19.05s --> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - [DIAR] 8: 19.35s - 20.87s --> SPEAKER_00 +2025-06-01 20:13:42,920 - INFO - [DIAR] 9: 21.07s - 27.40s --> SPEAKER_00 +2025-06-01 20:13:42,921 - INFO - Global diarization complete. 10 segments. +2025-06-01 20:13:45,108 - INFO - Running smart chunking... +2025-06-01 20:13:45,232 - INFO - Processing audio with duration 00:27.303 +2025-06-01 20:13:45,932 - INFO - Exported chunk: output_test4/chunks/chunk_000.wav (0.00s → 3.94s) +2025-06-01 20:13:45,933 - INFO - Exported chunk: output_test4/chunks/chunk_001.wav (4.64s → 9.34s) +2025-06-01 20:13:45,933 - INFO - Exported chunk: output_test4/chunks/chunk_002.wav (9.82s → 15.70s) +2025-06-01 20:13:45,933 - INFO - Exported chunk: output_test4/chunks/chunk_003.wav (15.70s → 17.84s) +2025-06-01 20:13:45,934 - INFO - Exported chunk: output_test4/chunks/chunk_004.wav (18.22s → 26.86s) +2025-06-01 20:13:45,934 - INFO - Total chunks: 5 +2025-06-01 20:13:47,224 - INFO - WhisperModel 'turbo' loaded on GPU 0 for thread 126507588519488 +2025-06-01 20:13:47,224 - INFO - Processing audio with duration 3.940 seconds +2025-06-01 20:13:47,314 - INFO - WhisperModel 'turbo' loaded on GPU 1 for thread 126507578033728 +2025-06-01 20:13:47,314 - INFO - Processing audio with duration 4.700 seconds +2025-06-01 20:13:47,432 - INFO - Processing audio with duration 00:03.940 +2025-06-01 20:13:47,535 - INFO - Processing audio with duration 00:04.700 +2025-06-01 20:13:47,569 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:13:47,728 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:13:47,736 - INFO - Speaker assignment breakdown: Counter({'Speaker 2': 1}) +2025-06-01 20:13:47,736 - INFO - Got result for chunk: output_test4/chunks/chunk_000.wav +2025-06-01 20:13:47,736 - INFO - +====== Streaming Chunk 001 ====== +[2025-06-01 20:13:40] Speaker 2: So, Aaron, in your email you said you wanted to talk about the exam. + +2025-06-01 20:13:47,736 - INFO - Processing audio with duration 5.880 seconds +2025-06-01 20:13:47,906 - INFO - Processing audio with duration 00:05.880 +2025-06-01 20:13:47,947 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:13:47,948 - INFO - Got result for chunk: output_test4/chunks/chunk_001.wav +2025-06-01 20:13:47,948 - INFO - +====== Streaming Chunk 002 ====== +[2025-06-01 20:13:40] Speaker 1: Yeah, um, I've just never taken a class with so many different readings. + +2025-06-01 20:13:47,948 - INFO - Processing audio with duration 2.140 seconds +2025-06-01 20:13:48,114 - INFO - Processing audio with duration 00:02.140 +2025-06-01 20:13:48,117 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:13:48,242 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:13:48,283 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:13:48,283 - INFO - Got result for chunk: output_test4/chunks/chunk_002.wav +2025-06-01 20:13:48,283 - INFO - +====== Streaming Chunk 003 ====== +[2025-06-01 20:13:40] Speaker 1: I've managed to keep up with all the assignments, but I'm not sure how to... +[2025-06-01 20:13:45] Speaker 1: How to... + +2025-06-01 20:13:48,283 - INFO - Processing audio with duration 8.640 seconds +2025-06-01 20:13:48,454 - INFO - Processing audio with duration 00:08.640 +2025-06-01 20:13:48,733 - INFO - Speaker assignment breakdown: Counter({'Speaker 2': 1}) +2025-06-01 20:13:48,733 - INFO - Got result for chunk: output_test4/chunks/chunk_003.wav +2025-06-01 20:13:48,733 - INFO - +====== Streaming Chunk 004 ====== +[2025-06-01 20:13:40] Speaker 2: How to review everything. + +2025-06-01 20:13:48,844 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:13:49,021 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:13:49,022 - INFO - Got result for chunk: output_test4/chunks/chunk_004.wav +2025-06-01 20:13:49,022 - INFO - +====== Streaming Chunk 005 ====== +[2025-06-01 20:13:40] Speaker 1: Yeah, in other classes I've had, there's usually just one book to review, not three different books, plus all those other text excerpts and video... + +2025-06-01 20:13:49,091 - INFO - Transcript saved to: output_test4/test_all_transcripts_20250601_201349.txt +2025-06-01 20:13:49,091 - INFO - CUDA available before summarization: True +2025-06-01 20:13:49,091 - INFO - Loading summarizer model from HuggingFace: mistralai/Mistral-7B-Instruct-v0.1 +2025-06-01 20:13:54,825 - INFO - Summarizing chunk 1/1 +2025-06-01 20:13:58,358 - INFO - Summary successfully appended to transcript TXT. +2025-06-01 20:13:58,358 - INFO - Final JSON saved to: output_test4/test_all_transcripts_20250601_201349.json +2025-06-01 20:13:58,359 - INFO - WER (Word Error Rate) against ground truth: 61.64% diff --git a/docs/whisper_transcription/Examples/Test2/transcription_log_20250601_203611.log b/docs/whisper_transcription/Examples/Test2/transcription_log_20250601_203611.log new file mode 100644 index 0000000..26239c6 --- /dev/null +++ b/docs/whisper_transcription/Examples/Test2/transcription_log_20250601_203611.log @@ -0,0 +1,1550 @@ +2025-06-01 20:36:11,212 - INFO - Total audio files to process: 1 +2025-06-01 20:36:11,212 - INFO - Validating: /home/ubuntu/whisper/docker3/dd_press/video1591686795.mp4 +2025-06-01 20:36:12,609 - INFO - Duration of /home/ubuntu/whisper/docker3/dd_press/video1591686795.mp4: 982.08s +2025-06-01 20:36:12,609 - INFO - Processing: /home/ubuntu/whisper/docker3/dd_press/video1591686795.mp4 +2025-06-01 20:36:13,670 - INFO - Running global speaker diarization... +2025-06-01 20:36:13,670 - INFO - Running speaker diarization on: /tmp/tmp1587zm1s.wav +2025-06-01 20:36:14,048 - DEBUG - Registered checkpoint save hook for _speechbrain_save +2025-06-01 20:36:14,048 - DEBUG - Registered checkpoint load hook for _speechbrain_load +2025-06-01 20:36:14,048 - DEBUG - Registered checkpoint save hook for save +2025-06-01 20:36:14,048 - DEBUG - Registered checkpoint load hook for load +2025-06-01 20:36:14,191 - INFO - Applied quirks (see `speechbrain.utils.quirks`): [disable_jit_profiling, allow_tf32] +2025-06-01 20:36:14,191 - INFO - Excluded quirks specified by the `SB_DISABLE_QUIRKS` environment (comma-separated list): [] +2025-06-01 20:36:14,192 - DEBUG - Registered checkpoint save hook for _save +2025-06-01 20:36:14,192 - DEBUG - Registered checkpoint load hook for _recover +2025-06-01 20:36:34,535 - INFO - Diarization returned 172 segments. Dumped to output_test3/diarization_debug.json. +2025-06-01 20:36:34,535 - INFO - Diarized segment: 5.35s - 6.16s -> SPEAKER_00 +2025-06-01 20:36:34,535 - INFO - Diarized segment: 6.93s - 13.01s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 13.94s - 14.93s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 15.02s - 23.40s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 24.43s - 29.48s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 31.18s - 34.64s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 34.93s - 42.56s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 43.30s - 53.81s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 54.15s - 59.46s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 59.97s - 62.25s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 62.45s - 69.51s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 71.67s - 73.56s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 76.51s - 80.98s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 81.72s - 87.09s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 87.58s - 95.02s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 95.63s - 97.15s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 97.79s - 103.22s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 104.12s - 117.53s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 118.21s - 123.71s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 125.04s - 126.27s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 126.51s - 133.97s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 134.31s - 136.55s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 136.65s - 152.92s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 153.37s - 162.69s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 163.33s - 164.01s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 166.20s - 170.30s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 168.81s - 169.29s -> SPEAKER_01 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 171.06s - 174.03s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 174.59s - 180.02s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 180.24s - 199.36s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 199.73s - 209.62s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 210.26s - 212.98s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 213.23s - 220.00s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 220.30s - 233.06s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 233.48s - 240.11s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 240.48s - 246.63s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 248.35s - 250.98s -> SPEAKER_00 +2025-06-01 20:36:34,536 - INFO - Diarized segment: 251.05s - 252.57s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 254.42s - 255.35s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 256.45s - 259.10s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 262.42s - 263.74s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 264.12s - 265.93s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 272.14s - 280.46s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 280.61s - 281.32s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 281.83s - 282.33s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 282.87s - 285.24s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 285.66s - 289.79s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 289.96s - 296.12s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 296.42s - 303.06s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 303.09s - 305.65s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 305.97s - 309.60s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 310.75s - 319.98s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 320.28s - 328.86s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 329.68s - 333.16s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 334.02s - 337.31s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 338.04s - 340.52s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 341.51s - 354.39s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 355.35s - 361.76s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 362.34s - 364.08s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 364.40s - 371.16s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 371.55s - 381.03s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 381.17s - 389.93s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 390.42s - 395.73s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 397.29s - 413.55s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 413.79s - 416.12s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 416.42s - 418.08s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 418.28s - 423.44s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 423.80s - 429.13s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 429.52s - 430.38s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 431.17s - 434.21s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 434.70s - 435.51s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 437.03s - 447.20s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 447.44s - 448.91s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 449.53s - 455.01s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 455.45s - 458.66s -> SPEAKER_00 +2025-06-01 20:36:34,537 - INFO - Diarized segment: 459.06s - 466.10s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 466.93s - 467.70s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 468.30s - 479.97s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 480.33s - 483.43s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 483.85s - 484.46s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 484.48s - 486.54s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 486.81s - 494.47s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 495.06s - 496.46s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 497.51s - 504.64s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 505.10s - 511.50s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 511.78s - 513.59s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 514.31s - 518.70s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 519.63s - 526.94s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 527.04s - 536.27s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 537.06s - 541.43s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 541.82s - 544.74s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 545.13s - 546.19s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 547.12s - 552.92s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 554.54s - 555.56s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 555.78s - 557.75s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 558.59s - 561.28s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 561.46s - 562.19s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 562.83s - 566.36s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 566.59s - 572.19s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 572.52s - 575.70s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 576.14s - 582.59s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 584.90s - 586.57s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 587.50s - 610.25s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 610.55s - 614.38s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 615.23s - 616.93s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 617.01s - 620.69s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 620.96s - 630.62s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 631.27s - 632.13s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 633.11s - 636.10s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 639.32s - 651.86s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 652.27s - 659.64s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 660.69s - 665.09s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 665.55s - 676.31s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 677.75s - 681.39s -> SPEAKER_00 +2025-06-01 20:36:34,538 - INFO - Diarized segment: 681.70s - 682.32s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 682.46s - 687.97s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 699.38s - 705.52s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 707.85s - 709.41s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 711.31s - 719.67s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 720.15s - 725.12s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 725.54s - 727.58s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 728.02s - 733.45s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 734.26s - 734.77s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 735.22s - 741.74s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 742.21s - 753.11s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 753.63s - 762.66s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 763.30s - 767.98s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 768.67s - 772.62s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 774.56s - 777.33s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 777.73s - 780.06s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 780.60s - 792.46s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 792.75s - 796.24s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 796.56s - 797.70s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 799.65s - 807.26s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 808.01s - 813.36s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 813.86s - 825.39s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 816.07s - 816.24s -> SPEAKER_01 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 825.84s - 829.91s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 831.07s - 835.63s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 835.90s - 836.56s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 836.90s - 839.29s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 841.38s - 845.86s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 848.07s - 852.71s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 852.98s - 856.12s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 856.18s - 857.28s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 857.84s - 861.38s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 862.07s - 863.63s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 863.96s - 867.95s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 868.32s - 871.22s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 871.91s - 875.74s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 876.16s - 880.26s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 884.10s - 887.03s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 887.50s - 890.68s -> SPEAKER_00 +2025-06-01 20:36:34,539 - INFO - Diarized segment: 891.03s - 891.71s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 892.28s - 892.84s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 893.58s - 895.06s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 895.50s - 901.81s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 902.27s - 913.07s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 915.53s - 925.91s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 926.27s - 926.87s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 927.35s - 928.02s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 929.15s - 930.54s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 931.41s - 932.81s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 933.35s - 934.60s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 934.82s - 939.56s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 939.87s - 941.27s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 941.45s - 943.02s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 943.23s - 946.41s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 946.97s - 954.08s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 956.89s - 963.32s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 964.17s - 968.82s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - Diarized segment: 969.42s - 976.91s -> SPEAKER_00 +2025-06-01 20:36:34,540 - INFO - [DIAR] 0: 5.35s - 6.16s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 1: 6.93s - 13.01s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 2: 13.94s - 14.93s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 3: 15.02s - 23.40s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 4: 24.43s - 29.48s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 5: 31.18s - 34.64s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 6: 34.93s - 42.56s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 7: 43.30s - 53.81s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 8: 54.15s - 59.46s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 9: 59.97s - 62.25s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 10: 62.45s - 69.51s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 11: 71.67s - 73.56s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 12: 76.51s - 80.98s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 13: 81.72s - 87.09s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 14: 87.58s - 95.02s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 15: 95.63s - 97.15s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 16: 97.79s - 103.22s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 17: 104.12s - 117.53s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 18: 118.21s - 123.71s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 19: 125.04s - 126.27s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 20: 126.51s - 133.97s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 21: 134.31s - 136.55s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 22: 136.65s - 152.92s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 23: 153.37s - 162.69s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 24: 163.33s - 164.01s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 25: 166.20s - 170.30s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 26: 168.81s - 169.29s --> SPEAKER_01 +2025-06-01 20:36:34,541 - INFO - [DIAR] 27: 171.06s - 174.03s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 28: 174.59s - 180.02s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 29: 180.24s - 199.36s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 30: 199.73s - 209.62s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 31: 210.26s - 212.98s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 32: 213.23s - 220.00s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 33: 220.30s - 233.06s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 34: 233.48s - 240.11s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 35: 240.48s - 246.63s --> SPEAKER_00 +2025-06-01 20:36:34,541 - INFO - [DIAR] 36: 248.35s - 250.98s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 37: 251.05s - 252.57s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 38: 254.42s - 255.35s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 39: 256.45s - 259.10s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 40: 262.42s - 263.74s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 41: 264.12s - 265.93s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 42: 272.14s - 280.46s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 43: 280.61s - 281.32s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 44: 281.83s - 282.33s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 45: 282.87s - 285.24s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 46: 285.66s - 289.79s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 47: 289.96s - 296.12s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 48: 296.42s - 303.06s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 49: 303.09s - 305.65s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 50: 305.97s - 309.60s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 51: 310.75s - 319.98s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 52: 320.28s - 328.86s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 53: 329.68s - 333.16s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 54: 334.02s - 337.31s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 55: 338.04s - 340.52s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 56: 341.51s - 354.39s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 57: 355.35s - 361.76s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 58: 362.34s - 364.08s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 59: 364.40s - 371.16s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 60: 371.55s - 381.03s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 61: 381.17s - 389.93s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 62: 390.42s - 395.73s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 63: 397.29s - 413.55s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 64: 413.79s - 416.12s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 65: 416.42s - 418.08s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 66: 418.28s - 423.44s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 67: 423.80s - 429.13s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 68: 429.52s - 430.38s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 69: 431.17s - 434.21s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 70: 434.70s - 435.51s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 71: 437.03s - 447.20s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 72: 447.44s - 448.91s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 73: 449.53s - 455.01s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 74: 455.45s - 458.66s --> SPEAKER_00 +2025-06-01 20:36:34,542 - INFO - [DIAR] 75: 459.06s - 466.10s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 76: 466.93s - 467.70s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 77: 468.30s - 479.97s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 78: 480.33s - 483.43s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 79: 483.85s - 484.46s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 80: 484.48s - 486.54s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 81: 486.81s - 494.47s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 82: 495.06s - 496.46s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 83: 497.51s - 504.64s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 84: 505.10s - 511.50s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 85: 511.78s - 513.59s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 86: 514.31s - 518.70s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 87: 519.63s - 526.94s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 88: 527.04s - 536.27s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 89: 537.06s - 541.43s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 90: 541.82s - 544.74s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 91: 545.13s - 546.19s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 92: 547.12s - 552.92s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 93: 554.54s - 555.56s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 94: 555.78s - 557.75s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 95: 558.59s - 561.28s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 96: 561.46s - 562.19s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 97: 562.83s - 566.36s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 98: 566.59s - 572.19s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 99: 572.52s - 575.70s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 100: 576.14s - 582.59s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 101: 584.90s - 586.57s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 102: 587.50s - 610.25s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 103: 610.55s - 614.38s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 104: 615.23s - 616.93s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 105: 617.01s - 620.69s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 106: 620.96s - 630.62s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 107: 631.27s - 632.13s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 108: 633.11s - 636.10s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 109: 639.32s - 651.86s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 110: 652.27s - 659.64s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 111: 660.69s - 665.09s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 112: 665.55s - 676.31s --> SPEAKER_00 +2025-06-01 20:36:34,543 - INFO - [DIAR] 113: 677.75s - 681.39s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 114: 681.70s - 682.32s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 115: 682.46s - 687.97s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 116: 699.38s - 705.52s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 117: 707.85s - 709.41s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 118: 711.31s - 719.67s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 119: 720.15s - 725.12s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 120: 725.54s - 727.58s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 121: 728.02s - 733.45s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 122: 734.26s - 734.77s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 123: 735.22s - 741.74s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 124: 742.21s - 753.11s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 125: 753.63s - 762.66s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 126: 763.30s - 767.98s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 127: 768.67s - 772.62s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 128: 774.56s - 777.33s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 129: 777.73s - 780.06s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 130: 780.60s - 792.46s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 131: 792.75s - 796.24s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 132: 796.56s - 797.70s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 133: 799.65s - 807.26s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 134: 808.01s - 813.36s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 135: 813.86s - 825.39s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 136: 816.07s - 816.24s --> SPEAKER_01 +2025-06-01 20:36:34,544 - INFO - [DIAR] 137: 825.84s - 829.91s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 138: 831.07s - 835.63s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 139: 835.90s - 836.56s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 140: 836.90s - 839.29s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 141: 841.38s - 845.86s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 142: 848.07s - 852.71s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 143: 852.98s - 856.12s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 144: 856.18s - 857.28s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 145: 857.84s - 861.38s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 146: 862.07s - 863.63s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 147: 863.96s - 867.95s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 148: 868.32s - 871.22s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 149: 871.91s - 875.74s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 150: 876.16s - 880.26s --> SPEAKER_00 +2025-06-01 20:36:34,544 - INFO - [DIAR] 151: 884.10s - 887.03s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 152: 887.50s - 890.68s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 153: 891.03s - 891.71s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 154: 892.28s - 892.84s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 155: 893.58s - 895.06s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 156: 895.50s - 901.81s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 157: 902.27s - 913.07s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 158: 915.53s - 925.91s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 159: 926.27s - 926.87s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 160: 927.35s - 928.02s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 161: 929.15s - 930.54s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 162: 931.41s - 932.81s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 163: 933.35s - 934.60s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 164: 934.82s - 939.56s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 165: 939.87s - 941.27s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 166: 941.45s - 943.02s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 167: 943.23s - 946.41s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 168: 946.97s - 954.08s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 169: 956.89s - 963.32s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 170: 964.17s - 968.82s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - [DIAR] 171: 969.42s - 976.91s --> SPEAKER_00 +2025-06-01 20:36:34,545 - INFO - Global diarization complete. 172 segments. +2025-06-01 20:36:36,542 - INFO - Running smart chunking... +2025-06-01 20:36:36,770 - INFO - Processing audio with duration 16:22.080 +2025-06-01 20:36:53,075 - INFO - Exported chunk: output_test6/chunks/chunk_000.wav (4.72s → 11.94s) +2025-06-01 20:36:53,075 - INFO - Exported chunk: output_test6/chunks/chunk_001.wav (11.94s → 22.20s) +2025-06-01 20:36:53,076 - INFO - Exported chunk: output_test6/chunks/chunk_002.wav (22.20s → 33.80s) +2025-06-01 20:36:53,076 - INFO - Exported chunk: output_test6/chunks/chunk_003.wav (33.80s → 42.48s) +2025-06-01 20:36:53,076 - INFO - Exported chunk: output_test6/chunks/chunk_004.wav (43.16s → 48.74s) +2025-06-01 20:36:53,077 - INFO - Exported chunk: output_test6/chunks/chunk_005.wav (48.74s → 55.12s) +2025-06-01 20:36:53,077 - INFO - Exported chunk: output_test6/chunks/chunk_006.wav (55.12s → 64.12s) +2025-06-01 20:36:53,077 - INFO - Exported chunk: output_test6/chunks/chunk_007.wav (64.12s → 72.48s) +2025-06-01 20:36:53,078 - INFO - Exported chunk: output_test6/chunks/chunk_008.wav (72.48s → 84.28s) +2025-06-01 20:36:53,078 - INFO - Exported chunk: output_test6/chunks/chunk_009.wav (84.28s → 92.92s) +2025-06-01 20:36:53,078 - INFO - Exported chunk: output_test6/chunks/chunk_010.wav (92.92s → 100.36s) +2025-06-01 20:36:53,079 - INFO - Exported chunk: output_test6/chunks/chunk_011.wav (100.36s → 109.00s) +2025-06-01 20:36:53,079 - INFO - Exported chunk: output_test6/chunks/chunk_012.wav (109.00s → 116.26s) +2025-06-01 20:36:53,079 - INFO - Exported chunk: output_test6/chunks/chunk_013.wav (116.26s → 123.62s) +2025-06-01 20:36:53,080 - INFO - Exported chunk: output_test6/chunks/chunk_014.wav (125.08s → 130.06s) +2025-06-01 20:36:53,080 - INFO - Exported chunk: output_test6/chunks/chunk_015.wav (130.06s → 136.34s) +2025-06-01 20:36:53,080 - INFO - Exported chunk: output_test6/chunks/chunk_016.wav (136.34s → 145.72s) +2025-06-01 20:36:53,081 - INFO - Exported chunk: output_test6/chunks/chunk_017.wav (145.72s → 154.78s) +2025-06-01 20:36:53,081 - INFO - Exported chunk: output_test6/chunks/chunk_018.wav (154.78s → 162.66s) +2025-06-01 20:36:53,081 - INFO - Exported chunk: output_test6/chunks/chunk_019.wav (165.72s → 173.78s) +2025-06-01 20:36:53,082 - INFO - Exported chunk: output_test6/chunks/chunk_020.wav (173.78s → 184.66s) +2025-06-01 20:36:53,082 - INFO - Exported chunk: output_test6/chunks/chunk_021.wav (184.66s → 192.10s) +2025-06-01 20:36:53,082 - INFO - Exported chunk: output_test6/chunks/chunk_022.wav (192.10s → 198.50s) +2025-06-01 20:36:53,082 - INFO - Exported chunk: output_test6/chunks/chunk_023.wav (198.50s → 207.68s) +2025-06-01 20:36:53,083 - INFO - Exported chunk: output_test6/chunks/chunk_024.wav (207.68s → 216.72s) +2025-06-01 20:36:53,083 - INFO - Exported chunk: output_test6/chunks/chunk_025.wav (216.72s → 226.24s) +2025-06-01 20:36:53,083 - INFO - Exported chunk: output_test6/chunks/chunk_026.wav (226.24s → 238.26s) +2025-06-01 20:36:53,084 - INFO - Exported chunk: output_test6/chunks/chunk_027.wav (238.26s → 246.56s) +2025-06-01 20:36:53,084 - INFO - Exported chunk: output_test6/chunks/chunk_028.wav (246.56s → 258.92s) +2025-06-01 20:36:53,084 - INFO - Exported chunk: output_test6/chunks/chunk_029.wav (262.06s → 265.82s) +2025-06-01 20:36:53,085 - INFO - Exported chunk: output_test6/chunks/chunk_030.wav (271.90s → 275.36s) +2025-06-01 20:36:53,085 - INFO - Exported chunk: output_test6/chunks/chunk_031.wav (275.36s → 285.10s) +2025-06-01 20:36:53,085 - INFO - Exported chunk: output_test6/chunks/chunk_032.wav (285.10s → 289.34s) +2025-06-01 20:36:53,085 - INFO - Exported chunk: output_test6/chunks/chunk_033.wav (289.92s → 298.38s) +2025-06-01 20:36:53,086 - INFO - Exported chunk: output_test6/chunks/chunk_034.wav (298.38s → 305.28s) +2025-06-01 20:36:53,086 - INFO - Exported chunk: output_test6/chunks/chunk_035.wav (305.90s → 312.96s) +2025-06-01 20:36:53,086 - INFO - Exported chunk: output_test6/chunks/chunk_036.wav (312.96s → 319.10s) +2025-06-01 20:36:53,087 - INFO - Exported chunk: output_test6/chunks/chunk_037.wav (319.10s → 327.34s) +2025-06-01 20:36:53,087 - INFO - Exported chunk: output_test6/chunks/chunk_038.wav (327.34s → 336.60s) +2025-06-01 20:36:53,087 - INFO - Exported chunk: output_test6/chunks/chunk_039.wav (336.60s → 345.20s) +2025-06-01 20:36:53,087 - INFO - Exported chunk: output_test6/chunks/chunk_040.wav (345.20s → 353.06s) +2025-06-01 20:36:53,088 - INFO - Exported chunk: output_test6/chunks/chunk_041.wav (353.06s → 363.30s) +2025-06-01 20:36:53,088 - INFO - Exported chunk: output_test6/chunks/chunk_042.wav (363.30s → 369.46s) +2025-06-01 20:36:53,088 - INFO - Exported chunk: output_test6/chunks/chunk_043.wav (369.46s → 376.04s) +2025-06-01 20:36:53,089 - INFO - Exported chunk: output_test6/chunks/chunk_044.wav (376.04s → 384.02s) +2025-06-01 20:36:53,089 - INFO - Exported chunk: output_test6/chunks/chunk_045.wav (384.02s → 393.68s) +2025-06-01 20:36:53,089 - INFO - Exported chunk: output_test6/chunks/chunk_046.wav (393.68s → 403.92s) +2025-06-01 20:36:53,089 - INFO - Exported chunk: output_test6/chunks/chunk_047.wav (403.92s → 411.66s) +2025-06-01 20:36:53,090 - INFO - Exported chunk: output_test6/chunks/chunk_048.wav (411.66s → 417.16s) +2025-06-01 20:36:53,090 - INFO - Exported chunk: output_test6/chunks/chunk_049.wav (417.16s → 429.02s) +2025-06-01 20:36:53,090 - INFO - Exported chunk: output_test6/chunks/chunk_050.wav (429.02s → 440.98s) +2025-06-01 20:36:53,091 - INFO - Exported chunk: output_test6/chunks/chunk_051.wav (440.98s → 448.32s) +2025-06-01 20:36:53,091 - INFO - Exported chunk: output_test6/chunks/chunk_052.wav (449.10s → 456.10s) +2025-06-01 20:36:53,091 - INFO - Exported chunk: output_test6/chunks/chunk_053.wav (456.10s → 463.00s) +2025-06-01 20:36:53,092 - INFO - Exported chunk: output_test6/chunks/chunk_054.wav (463.00s → 471.92s) +2025-06-01 20:36:53,092 - INFO - Exported chunk: output_test6/chunks/chunk_055.wav (471.92s → 478.44s) +2025-06-01 20:36:53,092 - INFO - Exported chunk: output_test6/chunks/chunk_056.wav (478.44s → 487.30s) +2025-06-01 20:36:53,092 - INFO - Exported chunk: output_test6/chunks/chunk_057.wav (487.30s → 493.92s) +2025-06-01 20:36:53,093 - INFO - Exported chunk: output_test6/chunks/chunk_058.wav (493.92s → 501.32s) +2025-06-01 20:36:53,093 - INFO - Exported chunk: output_test6/chunks/chunk_059.wav (501.32s → 508.32s) +2025-06-01 20:36:53,093 - INFO - Exported chunk: output_test6/chunks/chunk_060.wav (508.32s → 513.54s) +2025-06-01 20:36:53,093 - INFO - Exported chunk: output_test6/chunks/chunk_061.wav (514.08s → 518.74s) +2025-06-01 20:36:53,094 - INFO - Exported chunk: output_test6/chunks/chunk_062.wav (519.48s → 526.74s) +2025-06-01 20:36:53,094 - INFO - Exported chunk: output_test6/chunks/chunk_063.wav (526.74s → 539.02s) +2025-06-01 20:36:53,094 - INFO - Exported chunk: output_test6/chunks/chunk_064.wav (539.02s → 549.68s) +2025-06-01 20:36:53,095 - INFO - Exported chunk: output_test6/chunks/chunk_065.wav (549.68s → 561.94s) +2025-06-01 20:36:53,095 - INFO - Exported chunk: output_test6/chunks/chunk_066.wav (562.72s → 570.06s) +2025-06-01 20:36:53,095 - INFO - Exported chunk: output_test6/chunks/chunk_067.wav (570.06s → 578.68s) +2025-06-01 20:36:53,096 - INFO - Exported chunk: output_test6/chunks/chunk_068.wav (578.68s → 591.64s) +2025-06-01 20:36:53,096 - INFO - Exported chunk: output_test6/chunks/chunk_069.wav (591.64s → 601.02s) +2025-06-01 20:36:53,096 - INFO - Exported chunk: output_test6/chunks/chunk_070.wav (601.02s → 606.24s) +2025-06-01 20:36:53,097 - INFO - Exported chunk: output_test6/chunks/chunk_071.wav (606.24s → 612.78s) +2025-06-01 20:36:53,097 - INFO - Exported chunk: output_test6/chunks/chunk_072.wav (612.78s → 621.98s) +2025-06-01 20:36:53,097 - INFO - Exported chunk: output_test6/chunks/chunk_073.wav (621.98s → 634.42s) +2025-06-01 20:36:53,098 - INFO - Exported chunk: output_test6/chunks/chunk_074.wav (634.42s → 645.00s) +2025-06-01 20:36:53,098 - INFO - Exported chunk: output_test6/chunks/chunk_075.wav (645.00s → 654.00s) +2025-06-01 20:36:53,098 - INFO - Exported chunk: output_test6/chunks/chunk_076.wav (654.00s → 659.54s) +2025-06-01 20:36:53,099 - INFO - Exported chunk: output_test6/chunks/chunk_077.wav (659.54s → 669.38s) +2025-06-01 20:36:53,099 - INFO - Exported chunk: output_test6/chunks/chunk_078.wav (669.38s → 681.28s) +2025-06-01 20:36:53,099 - INFO - Exported chunk: output_test6/chunks/chunk_079.wav (681.28s → 687.72s) +2025-06-01 20:36:53,099 - INFO - Exported chunk: output_test6/chunks/chunk_080.wav (699.16s → 705.22s) +2025-06-01 20:36:53,100 - INFO - Exported chunk: output_test6/chunks/chunk_081.wav (707.78s → 709.30s) +2025-06-01 20:36:53,100 - INFO - Exported chunk: output_test6/chunks/chunk_082.wav (711.02s → 717.66s) +2025-06-01 20:36:53,100 - INFO - Exported chunk: output_test6/chunks/chunk_083.wav (717.66s → 725.00s) +2025-06-01 20:36:53,100 - INFO - Exported chunk: output_test6/chunks/chunk_084.wav (725.00s → 732.50s) +2025-06-01 20:36:53,101 - INFO - Exported chunk: output_test6/chunks/chunk_085.wav (732.50s → 740.30s) +2025-06-01 20:36:53,101 - INFO - Exported chunk: output_test6/chunks/chunk_086.wav (741.06s → 749.64s) +2025-06-01 20:36:53,101 - INFO - Exported chunk: output_test6/chunks/chunk_087.wav (750.18s → 756.74s) +2025-06-01 20:36:53,102 - INFO - Exported chunk: output_test6/chunks/chunk_088.wav (756.74s → 764.30s) +2025-06-01 20:36:53,102 - INFO - Exported chunk: output_test6/chunks/chunk_089.wav (764.30s → 770.72s) +2025-06-01 20:36:53,102 - INFO - Exported chunk: output_test6/chunks/chunk_090.wav (770.72s → 779.52s) +2025-06-01 20:36:53,102 - INFO - Exported chunk: output_test6/chunks/chunk_091.wav (779.52s → 785.76s) +2025-06-01 20:36:53,103 - INFO - Exported chunk: output_test6/chunks/chunk_092.wav (786.48s → 794.72s) +2025-06-01 20:36:53,103 - INFO - Exported chunk: output_test6/chunks/chunk_093.wav (794.72s → 806.90s) +2025-06-01 20:36:53,103 - INFO - Exported chunk: output_test6/chunks/chunk_094.wav (806.90s → 817.46s) +2025-06-01 20:36:53,104 - INFO - Exported chunk: output_test6/chunks/chunk_095.wav (817.46s → 824.44s) +2025-06-01 20:36:53,104 - INFO - Exported chunk: output_test6/chunks/chunk_096.wav (824.44s → 833.88s) +2025-06-01 20:36:53,104 - INFO - Exported chunk: output_test6/chunks/chunk_097.wav (833.88s → 839.26s) +2025-06-01 20:36:53,104 - INFO - Exported chunk: output_test6/chunks/chunk_098.wav (841.24s → 845.84s) +2025-06-01 20:36:53,105 - INFO - Exported chunk: output_test6/chunks/chunk_099.wav (847.74s → 852.44s) +2025-06-01 20:36:53,105 - INFO - Exported chunk: output_test6/chunks/chunk_100.wav (852.44s → 861.16s) +2025-06-01 20:36:53,105 - INFO - Exported chunk: output_test6/chunks/chunk_101.wav (861.96s → 870.92s) +2025-06-01 20:36:53,106 - INFO - Exported chunk: output_test6/chunks/chunk_102.wav (871.90s → 878.44s) +2025-06-01 20:36:53,106 - INFO - Exported chunk: output_test6/chunks/chunk_103.wav (878.44s → 880.16s) +2025-06-01 20:36:53,106 - INFO - Exported chunk: output_test6/chunks/chunk_104.wav (883.56s → 894.82s) +2025-06-01 20:36:53,106 - INFO - Exported chunk: output_test6/chunks/chunk_105.wav (895.44s → 903.72s) +2025-06-01 20:36:53,107 - INFO - Exported chunk: output_test6/chunks/chunk_106.wav (903.72s → 911.82s) +2025-06-01 20:36:53,107 - INFO - Exported chunk: output_test6/chunks/chunk_107.wav (911.82s → 912.80s) +2025-06-01 20:36:53,107 - INFO - Exported chunk: output_test6/chunks/chunk_108.wav (915.08s → 924.46s) +2025-06-01 20:36:53,107 - INFO - Exported chunk: output_test6/chunks/chunk_109.wav (924.46s → 932.64s) +2025-06-01 20:36:53,108 - INFO - Exported chunk: output_test6/chunks/chunk_110.wav (933.26s → 946.22s) +2025-06-01 20:36:53,108 - INFO - Exported chunk: output_test6/chunks/chunk_111.wav (946.94s → 953.88s) +2025-06-01 20:36:53,108 - INFO - Exported chunk: output_test6/chunks/chunk_112.wav (956.38s → 963.22s) +2025-06-01 20:36:53,109 - INFO - Exported chunk: output_test6/chunks/chunk_113.wav (964.32s → 974.04s) +2025-06-01 20:36:53,109 - INFO - Exported chunk: output_test6/chunks/chunk_114.wav (974.04s → 976.76s) +2025-06-01 20:36:53,109 - INFO - Total chunks: 115 +2025-06-01 20:36:54,452 - INFO - WhisperModel 'turbo' loaded on GPU 1 for thread 126640547956288 +2025-06-01 20:36:54,452 - INFO - WhisperModel 'turbo' loaded on GPU 0 for thread 126640558442048 +2025-06-01 20:36:54,452 - INFO - Processing audio with duration 10.260 seconds +2025-06-01 20:36:54,453 - INFO - Processing audio with duration 7.220 seconds +2025-06-01 20:36:54,638 - INFO - Processing audio with duration 00:10.260 +2025-06-01 20:36:54,750 - INFO - Processing audio with duration 00:07.220 +2025-06-01 20:36:54,890 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:54,963 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:55,051 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:55,051 - INFO - Got result for chunk: output_test6/chunks/chunk_000.wav +2025-06-01 20:36:55,051 - INFO - +====== Streaming Chunk 001 ====== +[2025-06-01 20:36:11] Speaker 1: hello everyone thank you for joining to this session this session we will talk about health + +2025-06-01 20:36:55,052 - INFO - Processing audio with duration 11.600 seconds +2025-06-01 20:36:55,223 - INFO - Processing audio with duration 00:11.600 +2025-06-01 20:36:55,224 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:55,224 - INFO - Got result for chunk: output_test6/chunks/chunk_001.wav +2025-06-01 20:36:55,224 - INFO - Processing audio with duration 8.680 seconds +2025-06-01 20:36:55,225 - INFO - +====== Streaming Chunk 002 ====== +[2025-06-01 20:36:11] Speaker 1: recipe that we provided in seeds group Ammar, Soumya and me etc. we are working on this. + +2025-06-01 20:36:55,424 - INFO - Processing audio with duration 00:08.680 +2025-06-01 20:36:55,426 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:55,554 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:55,588 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:55,589 - INFO - Got result for chunk: output_test6/chunks/chunk_002.wav +2025-06-01 20:36:55,589 - INFO - +====== Streaming Chunk 003 ====== +[2025-06-01 20:36:11] Speaker 1: a resume and with that we can start to talk about that so the main idea of the health check + +2025-06-01 20:36:55,589 - INFO - Processing audio with duration 5.580 seconds +2025-06-01 20:36:55,764 - INFO - Processing audio with duration 00:05.580 +2025-06-01 20:36:55,798 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:55,799 - INFO - Got result for chunk: output_test6/chunks/chunk_003.wav +2025-06-01 20:36:55,799 - INFO - +====== Streaming Chunk 004 ====== +[2025-06-01 20:36:11] Speaker 1: recipe is to make the customer confidence that they have enough resources to run heavy workloads + +2025-06-01 20:36:55,799 - INFO - Processing audio with duration 6.380 seconds +2025-06-01 20:36:55,970 - INFO - Processing audio with duration 00:06.380 +2025-06-01 20:36:55,972 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:56,102 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:56,144 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:36:56,145 - INFO - Got result for chunk: output_test6/chunks/chunk_004.wav +2025-06-01 20:36:56,145 - INFO - +====== Streaming Chunk 005 ====== +[2025-06-01 20:36:11] Speaker 1: what i mean by that is that most of the customers when they want to run the llm's training find +[2025-06-01 20:36:29] Speaker 1: you + +2025-06-01 20:36:56,145 - INFO - Processing audio with duration 9.000 seconds +2025-06-01 20:36:56,317 - INFO - Processing audio with duration 00:09.000 +2025-06-01 20:36:56,572 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:56,573 - INFO - Got result for chunk: output_test6/chunks/chunk_005.wav +2025-06-01 20:36:56,573 - INFO - +====== Streaming Chunk 006 ====== +[2025-06-01 20:36:11] Speaker 1: tuning or any other things they need which assumed to be a heavy workload they need to have a lot of + +2025-06-01 20:36:56,573 - INFO - Processing audio with duration 8.360 seconds +2025-06-01 20:36:56,745 - INFO - Processing audio with duration 00:08.360 +2025-06-01 20:36:56,747 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:56,875 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:56,912 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:56,912 - INFO - Got result for chunk: output_test6/chunks/chunk_006.wav +2025-06-01 20:36:56,912 - INFO - +====== Streaming Chunk 007 ====== +[2025-06-01 20:36:11] Speaker 1: gpus and we want to make sure that these gpus work well so they can run this recipe before starting + +2025-06-01 20:36:56,913 - INFO - Processing audio with duration 11.800 seconds +2025-06-01 20:36:57,097 - INFO - Processing audio with duration 00:11.800 +2025-06-01 20:36:57,138 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:57,138 - INFO - Got result for chunk: output_test6/chunks/chunk_007.wav +2025-06-01 20:36:57,138 - INFO - +====== Streaming Chunk 008 ====== +[2025-06-01 20:36:11] Speaker 1: this task and to just make sure that you know everything is working very well i'll go ahead + +2025-06-01 20:36:57,139 - INFO - Processing audio with duration 8.640 seconds +2025-06-01 20:36:57,332 - INFO - Processing audio with duration 00:08.640 +2025-06-01 20:36:57,334 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:57,462 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:57,498 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:57,498 - INFO - Got result for chunk: output_test6/chunks/chunk_008.wav +2025-06-01 20:36:57,498 - INFO - +====== Streaming Chunk 009 ====== +[2025-06-01 20:36:11] Speaker 1: share my screen and yeah i will start with the result of this recipe i can say that this you know + +2025-06-01 20:36:57,498 - INFO - Processing audio with duration 7.440 seconds +2025-06-01 20:36:57,670 - INFO - Processing audio with duration 00:07.440 +2025-06-01 20:36:57,709 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:57,710 - INFO - Got result for chunk: output_test6/chunks/chunk_009.wav +2025-06-01 20:36:57,710 - INFO - +====== Streaming Chunk 010 ====== +[2025-06-01 20:36:11] Speaker 1: recipe the health check recipe will provide you two different formats one is the pdf which i'm showing + +2025-06-01 20:36:57,710 - INFO - Processing audio with duration 8.640 seconds +2025-06-01 20:36:57,885 - INFO - Processing audio with duration 00:08.640 +2025-06-01 20:36:57,887 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:58,015 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:58,056 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:58,056 - INFO - Got result for chunk: output_test6/chunks/chunk_010.wav +2025-06-01 20:36:58,056 - INFO - +====== Streaming Chunk 011 ====== +[2025-06-01 20:36:11] Speaker 1: to you and the other one is json one i will also go through that as well but here you can see uh what + +2025-06-01 20:36:58,057 - INFO - Processing audio with duration 7.260 seconds +2025-06-01 20:36:58,232 - INFO - Processing audio with duration 00:07.260 +2025-06-01 20:36:58,270 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:58,270 - INFO - Got result for chunk: output_test6/chunks/chunk_011.wav +2025-06-01 20:36:58,270 - INFO - +====== Streaming Chunk 012 ====== +[2025-06-01 20:36:11] Speaker 1: test we are evaluating for the gpus uh we are doing bunch of uh testing like the background + +2025-06-01 20:36:58,270 - INFO - Processing audio with duration 7.360 seconds +2025-06-01 20:36:58,446 - INFO - Processing audio with duration 00:07.360 +2025-06-01 20:36:58,448 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:58,577 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:58,614 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:36:58,615 - INFO - Got result for chunk: output_test6/chunks/chunk_012.wav +2025-06-01 20:36:58,615 - INFO - +====== Streaming Chunk 013 ====== +[2025-06-01 20:36:11] Speaker 1: compare computation compute throughput memory bandwidth error detection tensor core utilization +[2025-06-01 20:36:17] Speaker 1: sustain workload + +2025-06-01 20:36:58,615 - INFO - Processing audio with duration 4.980 seconds +2025-06-01 20:36:58,788 - INFO - Processing audio with duration 00:04.980 +2025-06-01 20:36:59,005 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:59,005 - INFO - Got result for chunk: output_test6/chunks/chunk_013.wav +2025-06-01 20:36:59,005 - INFO - +====== Streaming Chunk 014 ====== +[2025-06-01 20:36:11] Speaker 1: mixed precision, power, temperature and utilization. So these are the main tests that we are + +2025-06-01 20:36:59,006 - INFO - Processing audio with duration 6.280 seconds +2025-06-01 20:36:59,176 - INFO - Processing audio with duration 00:06.280 +2025-06-01 20:36:59,178 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:59,306 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:59,338 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:36:59,339 - INFO - Got result for chunk: output_test6/chunks/chunk_014.wav +2025-06-01 20:36:59,339 - INFO - +====== Streaming Chunk 015 ====== +[2025-06-01 20:36:11] Speaker 1: evaluating you can see that in this format once you know you run the health +[2025-06-01 20:36:15] Speaker 1: show + +2025-06-01 20:36:59,339 - INFO - Processing audio with duration 9.380 seconds +2025-06-01 20:36:59,510 - INFO - Processing audio with duration 00:09.380 +2025-06-01 20:36:59,544 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:36:59,545 - INFO - Got result for chunk: output_test6/chunks/chunk_015.wav +2025-06-01 20:36:59,545 - INFO - +====== Streaming Chunk 016 ====== +[2025-06-01 20:36:11] Speaker 1: recipe you will get the result you know everything is in a table the duration of each + +2025-06-01 20:36:59,545 - INFO - Processing audio with duration 9.060 seconds +2025-06-01 20:36:59,721 - INFO - Processing audio with duration 00:09.060 +2025-06-01 20:36:59,724 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:59,853 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:36:59,886 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:36:59,887 - INFO - Got result for chunk: output_test6/chunks/chunk_016.wav +2025-06-01 20:36:59,887 - INFO - +====== Streaming Chunk 017 ====== +[2025-06-01 20:36:11] Speaker 1: test, whether it has failed or not. +[2025-06-01 20:36:18] Speaker 1: For example, here for compute throughput, + +2025-06-01 20:36:59,887 - INFO - Processing audio with duration 7.880 seconds +2025-06-01 20:37:00,058 - INFO - Processing audio with duration 00:07.880 +2025-06-01 20:37:00,106 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:00,107 - INFO - Got result for chunk: output_test6/chunks/chunk_017.wav +2025-06-01 20:37:00,107 - INFO - +====== Streaming Chunk 018 ====== +[2025-06-01 20:36:11] Speaker 1: has failed and i will show you uh in detail that how it works and for those customers who you know +[2025-06-01 20:36:19] Speaker 1: maybe + +2025-06-01 20:37:00,107 - INFO - Processing audio with duration 8.060 seconds +2025-06-01 20:37:00,278 - INFO - Processing audio with duration 00:08.060 +2025-06-01 20:37:00,281 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:00,409 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:00,451 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:00,451 - INFO - Got result for chunk: output_test6/chunks/chunk_018.wav +2025-06-01 20:37:00,451 - INFO - +====== Streaming Chunk 019 ====== +[2025-06-01 20:36:11] Speaker 1: don't know what are these numbers you know they can go to the end of that we +[2025-06-01 20:36:17] Speaker 1: have also provided a graph + +2025-06-01 20:37:00,451 - INFO - Processing audio with duration 10.880 seconds +2025-06-01 20:37:00,622 - INFO - Processing audio with duration 00:10.880 +2025-06-01 20:37:00,666 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:00,667 - INFO - Got result for chunk: output_test6/chunks/chunk_019.wav +2025-06-01 20:37:00,667 - INFO - +====== Streaming Chunk 020 ====== +[2025-06-01 20:36:11] Speaker 1: I'm waiting for that. We have also provided a graph. In this graph, we have provided, you know, + +2025-06-01 20:37:00,668 - INFO - Processing audio with duration 7.440 seconds +2025-06-01 20:37:00,854 - INFO - Processing audio with duration 00:07.440 +2025-06-01 20:37:00,856 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:00,985 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:01,025 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:01,025 - INFO - Got result for chunk: output_test6/chunks/chunk_020.wav +2025-06-01 20:37:01,025 - INFO - +====== Streaming Chunk 021 ====== +[2025-06-01 20:36:11] Speaker 1: So first what we did was that we did an extensive research from NVIDIA website articles to see what are the... + +2025-06-01 20:37:01,025 - INFO - Processing audio with duration 6.400 seconds +2025-06-01 20:37:01,194 - INFO - Processing audio with duration 00:06.400 +2025-06-01 20:37:01,447 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:01,448 - INFO - Got result for chunk: output_test6/chunks/chunk_021.wav +2025-06-01 20:37:01,448 - INFO - +====== Streaming Chunk 022 ====== +[2025-06-01 20:36:11] Speaker 1: uh you know ideal uh range for for example temperature the compute throughput memory bandwidth +[2025-06-01 20:36:17] Speaker 1: and this type of + +2025-06-01 20:37:01,448 - INFO - Processing audio with duration 9.180 seconds +2025-06-01 20:37:01,618 - INFO - Processing audio with duration 00:09.180 +2025-06-01 20:37:01,620 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:01,748 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:01,784 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:01,784 - INFO - Got result for chunk: output_test6/chunks/chunk_022.wav +2025-06-01 20:37:01,785 - INFO - +====== Streaming Chunk 023 ====== +[2025-06-01 20:36:11] Speaker 1: things. What is the upper threshold? What is the lower threshold? We added a 10% to each of + +2025-06-01 20:37:01,785 - INFO - Processing audio with duration 9.040 seconds +2025-06-01 20:37:01,956 - INFO - Processing audio with duration 00:09.040 +2025-06-01 20:37:02,005 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:02,005 - INFO - Got result for chunk: output_test6/chunks/chunk_023.wav +2025-06-01 20:37:02,006 - INFO - +====== Streaming Chunk 024 ====== +[2025-06-01 20:36:11] Speaker 1: this once and uh we got the average of that these average are shown here as a baseline for example +[2025-06-01 20:36:18] Speaker 1: 3000 is + +2025-06-01 20:37:02,006 - INFO - Processing audio with duration 9.520 seconds +2025-06-01 20:37:02,191 - INFO - Processing audio with duration 00:09.520 +2025-06-01 20:37:02,194 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:02,322 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:02,364 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:02,365 - INFO - Got result for chunk: output_test6/chunks/chunk_024.wav +2025-06-01 20:37:02,365 - INFO - +====== Streaming Chunk 025 ====== +[2025-06-01 20:36:11] Speaker 1: shown here as a baseline and the measured one is the for example uh the yellow one it shows that uh +[2025-06-01 20:36:19] Speaker 1: you know + +2025-06-01 20:37:02,365 - INFO - Processing audio with duration 12.020 seconds +2025-06-01 20:37:02,539 - INFO - Processing audio with duration 00:12.020 +2025-06-01 20:37:02,585 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:02,585 - INFO - Got result for chunk: output_test6/chunks/chunk_025.wav +2025-06-01 20:37:02,585 - INFO - +====== Streaming Chunk 026 ====== +[2025-06-01 20:36:11] Speaker 1: So once we are using the test compute throughput, the measure was around 651.92, which is around 630. + +2025-06-01 20:37:02,586 - INFO - Processing audio with duration 8.300 seconds +2025-06-01 20:37:02,775 - INFO - Processing audio with duration 00:08.300 +2025-06-01 20:37:02,777 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:02,905 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:02,939 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:02,940 - INFO - Got result for chunk: output_test6/chunks/chunk_026.wav +2025-06-01 20:37:02,940 - INFO - +====== Streaming Chunk 027 ====== +[2025-06-01 20:36:11] Speaker 1: uh 30 which was the baseline so the main purpose of this graph is to make the customer sure that + +2025-06-01 20:37:02,940 - INFO - Processing audio with duration 12.360 seconds +2025-06-01 20:37:03,119 - INFO - Processing audio with duration 00:12.360 +2025-06-01 20:37:03,164 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:03,165 - INFO - Got result for chunk: output_test6/chunks/chunk_027.wav +2025-06-01 20:37:03,165 - INFO - Processing audio with duration 3.760 seconds +2025-06-01 20:37:03,165 - INFO - +====== Streaming Chunk 028 ====== +[2025-06-01 20:36:11] Speaker 1: numbers that they are getting are in the same kind of range of you know what the +[2025-06-01 20:36:18] Speaker 1: number should be + +2025-06-01 20:37:03,357 - INFO - Processing audio with duration 00:03.760 +2025-06-01 20:37:03,360 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:03,848 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:03,870 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:03,870 - INFO - Got result for chunk: output_test6/chunks/chunk_028.wav +2025-06-01 20:37:03,871 - INFO - +====== Streaming Chunk 029 ====== +[2025-06-01 20:36:11] Speaker 1: With that, I can also go to the JSON file. + +2025-06-01 20:37:03,871 - INFO - Processing audio with duration 3.460 seconds +2025-06-01 20:37:04,039 - INFO - Processing audio with duration 00:03.460 +2025-06-01 20:37:04,065 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:04,066 - INFO - Got result for chunk: output_test6/chunks/chunk_029.wav +2025-06-01 20:37:04,066 - INFO - +====== Streaming Chunk 030 ====== +[2025-06-01 20:36:11] Speaker 1: which we have here. I will just open one of them. + +2025-06-01 20:37:04,066 - INFO - Processing audio with duration 9.740 seconds +2025-06-01 20:37:04,245 - INFO - Processing audio with duration 00:09.740 +2025-06-01 20:37:04,248 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:04,376 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:04,397 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:04,397 - INFO - Got result for chunk: output_test6/chunks/chunk_030.wav +2025-06-01 20:37:04,397 - INFO - +====== Streaming Chunk 031 ====== +[2025-06-01 20:36:11] Speaker 1: So the way that they know the JSON file. + +2025-06-01 20:37:04,397 - INFO - Processing audio with duration 4.240 seconds +2025-06-01 20:37:04,566 - INFO - Processing audio with duration 00:04.240 +2025-06-01 20:37:04,601 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:04,602 - INFO - Got result for chunk: output_test6/chunks/chunk_031.wav +2025-06-01 20:37:04,602 - INFO - +====== Streaming Chunk 032 ====== +[2025-06-01 20:36:11] Speaker 1: is working is that you know basically the same numbers are both in the PDF and the JSON file. + +2025-06-01 20:37:04,602 - INFO - Processing audio with duration 8.460 seconds +2025-06-01 20:37:04,777 - INFO - Processing audio with duration 00:08.460 +2025-06-01 20:37:04,780 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:04,907 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:04,941 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:04,941 - INFO - Got result for chunk: output_test6/chunks/chunk_032.wav +2025-06-01 20:37:04,941 - INFO - +====== Streaming Chunk 033 ====== +[2025-06-01 20:36:11] Speaker 1: The good thing about JSON file here is that, you know, for example, for compute throughput, + +2025-06-01 20:37:04,941 - INFO - Processing audio with duration 6.900 seconds +2025-06-01 20:37:05,112 - INFO - Processing audio with duration 00:06.900 +2025-06-01 20:37:05,155 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:05,155 - INFO - Got result for chunk: output_test6/chunks/chunk_033.wav +2025-06-01 20:37:05,155 - INFO - Processing audio with duration 7.060 seconds +2025-06-01 20:37:05,155 - INFO - +====== Streaming Chunk 034 ====== +[2025-06-01 20:36:11] Speaker 1: it has failed if it go on each gpu one by one um based on the tensor and you know duration + +2025-06-01 20:37:05,326 - INFO - Processing audio with duration 00:07.060 +2025-06-01 20:37:05,329 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:05,456 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:05,495 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:05,495 - INFO - Got result for chunk: output_test6/chunks/chunk_034.wav +2025-06-01 20:37:05,495 - INFO - +====== Streaming Chunk 035 ====== +[2025-06-01 20:36:11] Speaker 1: flops bandwidth and it will uh make the comparison with the baseline and if uh for all of them it's not + +2025-06-01 20:37:05,495 - INFO - Processing audio with duration 6.140 seconds +2025-06-01 20:37:05,669 - INFO - Processing audio with duration 00:06.140 +2025-06-01 20:37:05,709 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:05,709 - INFO - Got result for chunk: output_test6/chunks/chunk_035.wav +2025-06-01 20:37:05,709 - INFO - +====== Streaming Chunk 036 ====== +[2025-06-01 20:36:11] Speaker 1: in the range that you know we specify it will give us the fail. Similarly for the memory band + +2025-06-01 20:37:05,709 - INFO - Processing audio with duration 8.240 seconds +2025-06-01 20:37:05,881 - INFO - Processing audio with duration 00:08.240 +2025-06-01 20:37:05,883 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:06,324 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:06,359 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:06,360 - INFO - Got result for chunk: output_test6/chunks/chunk_036.wav +2025-06-01 20:37:06,360 - INFO - +====== Streaming Chunk 037 ====== +[2025-06-01 20:36:11] Speaker 1: it you know it has some for each gpu it will say that you know what does what are the information + +2025-06-01 20:37:06,360 - INFO - Processing audio with duration 9.260 seconds +2025-06-01 20:37:06,531 - INFO - Processing audio with duration 00:09.260 +2025-06-01 20:37:06,572 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:06,573 - INFO - Got result for chunk: output_test6/chunks/chunk_037.wav +2025-06-01 20:37:06,573 - INFO - +====== Streaming Chunk 038 ====== +[2025-06-01 20:36:11] Speaker 1: at the end it will give whether it is ideal or not so this is the way that uh you know the + +2025-06-01 20:37:06,573 - INFO - Processing audio with duration 8.600 seconds +2025-06-01 20:37:06,745 - INFO - Processing audio with duration 00:08.600 +2025-06-01 20:37:06,747 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:06,875 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:06,914 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:06,914 - INFO - Got result for chunk: output_test6/chunks/chunk_038.wav +2025-06-01 20:37:06,914 - INFO - +====== Streaming Chunk 039 ====== +[2025-06-01 20:36:11] Speaker 1: JSON file is working. With that, I can go inside the code and I can talk a little bit more about + +2025-06-01 20:37:06,915 - INFO - Processing audio with duration 7.860 seconds +2025-06-01 20:37:07,088 - INFO - Processing audio with duration 00:07.860 +2025-06-01 20:37:07,126 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:07,126 - INFO - Got result for chunk: output_test6/chunks/chunk_039.wav +2025-06-01 20:37:07,127 - INFO - +====== Streaming Chunk 040 ====== +[2025-06-01 20:36:11] Speaker 1: the code what we have provided for the customer so we are we have enabled the customer to choose + +2025-06-01 20:37:07,127 - INFO - Processing audio with duration 10.240 seconds +2025-06-01 20:37:07,300 - INFO - Processing audio with duration 00:10.240 +2025-06-01 20:37:07,302 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:07,431 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:07,489 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:07,489 - INFO - Got result for chunk: output_test6/chunks/chunk_040.wav +2025-06-01 20:37:07,489 - INFO - +====== Streaming Chunk 041 ====== +[2025-06-01 20:36:11] Speaker 1: what type, what D-type, you know, they can choose. +[2025-06-01 20:36:14] Speaker 1: They can go with FV32, FV16, which is the most common one. + +2025-06-01 20:37:07,489 - INFO - Processing audio with duration 6.160 seconds +2025-06-01 20:37:07,661 - INFO - Processing audio with duration 00:06.160 +2025-06-01 20:37:07,706 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:07,706 - INFO - Got result for chunk: output_test6/chunks/chunk_041.wav +2025-06-01 20:37:07,707 - INFO - +====== Streaming Chunk 042 ====== +[2025-06-01 20:36:11] Speaker 1: uh fp8 uh we have also enabled them that whether they want to just choose one um uh you know one of + +2025-06-01 20:37:07,707 - INFO - Processing audio with duration 6.580 seconds +2025-06-01 20:37:07,897 - INFO - Processing audio with duration 00:06.580 +2025-06-01 20:37:07,900 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:08,027 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:08,062 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:08,062 - INFO - Got result for chunk: output_test6/chunks/chunk_042.wav +2025-06-01 20:37:08,063 - INFO - +====== Streaming Chunk 043 ====== +[2025-06-01 20:36:11] Speaker 1: these uh uh functions or you know all of them the default is on all of them also they can choose + +2025-06-01 20:37:08,063 - INFO - Processing audio with duration 7.980 seconds +2025-06-01 20:37:08,234 - INFO - Processing audio with duration 00:07.980 +2025-06-01 20:37:08,276 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:08,276 - INFO - Got result for chunk: output_test6/chunks/chunk_043.wav +2025-06-01 20:37:08,276 - INFO - +====== Streaming Chunk 044 ====== +[2025-06-01 20:36:11] Speaker 1: to do the direction so we have provided a couple of uh you know uh parser uh so that you know + +2025-06-01 20:37:08,277 - INFO - Processing audio with duration 9.660 seconds +2025-06-01 20:37:08,453 - INFO - Processing audio with duration 00:09.660 +2025-06-01 20:37:08,455 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:08,875 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:08,910 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:08,910 - INFO - Got result for chunk: output_test6/chunks/chunk_044.wav +2025-06-01 20:37:08,910 - INFO - +====== Streaming Chunk 045 ====== +[2025-06-01 20:36:11] Speaker 1: can choose uh uh you know uh to play with them uh code one important thing is the tensor shape + +2025-06-01 20:37:08,910 - INFO - Processing audio with duration 10.240 seconds +2025-06-01 20:37:09,082 - INFO - Processing audio with duration 00:10.240 +2025-06-01 20:37:09,135 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:09,135 - INFO - Got result for chunk: output_test6/chunks/chunk_045.wav +2025-06-01 20:37:09,135 - INFO - Processing audio with duration 7.740 seconds +2025-06-01 20:37:09,135 - INFO - +====== Streaming Chunk 046 ====== +[2025-06-01 20:36:11] Speaker 1: example for a10 the tensor shape is 8192 this is the default that we have provided for all of the +[2025-06-01 20:36:20] Speaker 1: a10 + +2025-06-01 20:37:09,322 - INFO - Processing audio with duration 00:07.740 +2025-06-01 20:37:09,324 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:09,453 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:09,494 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:09,494 - INFO - Got result for chunk: output_test6/chunks/chunk_046.wav +2025-06-01 20:37:09,494 - INFO - +====== Streaming Chunk 047 ====== +[2025-06-01 20:36:11] Speaker 1: a hundred and h hundred um with this i will give you some information that we can we also measure +[2025-06-01 20:36:20] Speaker 1: the highest + +2025-06-01 20:37:09,495 - INFO - Processing audio with duration 5.500 seconds +2025-06-01 20:37:09,664 - INFO - Processing audio with duration 00:05.500 +2025-06-01 20:37:09,708 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:09,708 - INFO - Got result for chunk: output_test6/chunks/chunk_047.wav +2025-06-01 20:37:09,708 - INFO - +====== Streaming Chunk 048 ====== +[2025-06-01 20:36:11] Speaker 1: uh tensor for example for 800 we can also use 65k for that that will take a long time so that was + +2025-06-01 20:37:09,708 - INFO - Processing audio with duration 11.860 seconds +2025-06-01 20:37:09,876 - INFO - Processing audio with duration 00:11.860 +2025-06-01 20:37:09,878 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:10,007 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:10,036 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:10,036 - INFO - Got result for chunk: output_test6/chunks/chunk_048.wav +2025-06-01 20:37:10,037 - INFO - +====== Streaming Chunk 049 ====== +[2025-06-01 20:36:11] Speaker 1: reason why we wanted to go to go with something which take less time so for example + +2025-06-01 20:37:10,037 - INFO - Processing audio with duration 11.960 seconds +2025-06-01 20:37:10,208 - INFO - Processing audio with duration 00:11.960 +2025-06-01 20:37:10,256 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:10,257 - INFO - Got result for chunk: output_test6/chunks/chunk_049.wav +2025-06-01 20:37:10,257 - INFO - Processing audio with duration 7.340 seconds +2025-06-01 20:37:10,257 - INFO - +====== Streaming Chunk 050 ====== +[2025-06-01 20:36:11] Speaker 1: the H100 with 8192 it will take around 110 seconds. A100 it will take around 492 or similar like that. + +2025-06-01 20:37:10,444 - INFO - Processing audio with duration 00:07.340 +2025-06-01 20:37:10,446 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:10,575 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:10,620 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:10,620 - INFO - Got result for chunk: output_test6/chunks/chunk_050.wav +2025-06-01 20:37:10,621 - INFO - +====== Streaming Chunk 051 ====== +[2025-06-01 20:36:11] Speaker 1: and for 810 it will take around 649 seconds and with that we can start to talk about the functions +[2025-06-01 20:36:22] Speaker 1: I'm not + +2025-06-01 20:37:10,621 - INFO - Processing audio with duration 7.000 seconds +2025-06-01 20:37:10,793 - INFO - Processing audio with duration 00:07.000 +2025-06-01 20:37:10,832 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:10,832 - INFO - Got result for chunk: output_test6/chunks/chunk_051.wav +2025-06-01 20:37:10,832 - INFO - +====== Streaming Chunk 052 ====== +[2025-06-01 20:36:11] Speaker 1: going about the function again the first thing is the power the temperature and utilization + +2025-06-01 20:37:10,833 - INFO - Processing audio with duration 6.900 seconds +2025-06-01 20:37:11,004 - INFO - Processing audio with duration 00:06.900 +2025-06-01 20:37:11,226 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:11,339 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:11,384 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:11,384 - INFO - Got result for chunk: output_test6/chunks/chunk_052.wav +2025-06-01 20:37:11,384 - INFO - Processing audio with duration 8.920 seconds +2025-06-01 20:37:11,384 - INFO - +====== Streaming Chunk 053 ====== +[2025-06-01 20:36:11] Speaker 1: So we found out what is the range of that and if the temperature, the power... + +2025-06-01 20:37:11,553 - INFO - Processing audio with duration 00:08.920 +2025-06-01 20:37:11,593 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:11,593 - INFO - Got result for chunk: output_test6/chunks/chunk_053.wav +2025-06-01 20:37:11,593 - INFO - +====== Streaming Chunk 054 ====== +[2025-06-01 20:36:11] Speaker 1: visualization is in in that range you know we'll say that it has passed if not uh you know we'll say + +2025-06-01 20:37:11,594 - INFO - Processing audio with duration 6.520 seconds +2025-06-01 20:37:11,766 - INFO - Processing audio with duration 00:06.520 +2025-06-01 20:37:11,769 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:11,896 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:11,934 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:11,934 - INFO - Got result for chunk: output_test6/chunks/chunk_054.wav +2025-06-01 20:37:11,934 - INFO - +====== Streaming Chunk 055 ====== +[2025-06-01 20:36:11] Speaker 1: that you know uh it has uh you know fade um these are some of the information about the gpus uh what + +2025-06-01 20:37:11,935 - INFO - Processing audio with duration 8.860 seconds +2025-06-01 20:37:12,105 - INFO - Processing audio with duration 00:08.860 +2025-06-01 20:37:12,140 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:12,141 - INFO - Got result for chunk: output_test6/chunks/chunk_055.wav +2025-06-01 20:37:12,141 - INFO - +====== Streaming Chunk 056 ====== +[2025-06-01 20:36:11] Speaker 1: shape is that and this type of things till we get to the background computation and compute on gpu + +2025-06-01 20:37:12,141 - INFO - Processing audio with duration 6.620 seconds +2025-06-01 20:37:12,314 - INFO - Processing audio with duration 00:06.620 +2025-06-01 20:37:12,317 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:12,445 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:12,477 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:12,477 - INFO - Got result for chunk: output_test6/chunks/chunk_056.wav +2025-06-01 20:37:12,478 - INFO - +====== Streaming Chunk 057 ====== +[2025-06-01 20:36:11] Speaker 1: background computation is that it's a function that it will compute uh you know a simple math mall + +2025-06-01 20:37:12,478 - INFO - Processing audio with duration 7.400 seconds +2025-06-01 20:37:12,646 - INFO - Processing audio with duration 00:07.400 +2025-06-01 20:37:12,681 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:12,681 - INFO - Got result for chunk: output_test6/chunks/chunk_057.wav +2025-06-01 20:37:12,681 - INFO - +====== Streaming Chunk 058 ====== +[2025-06-01 20:36:11] Speaker 1: back end so that you know we can measure the for temperature for uh you know for uh utilization + +2025-06-01 20:37:12,681 - INFO - Processing audio with duration 7.000 seconds +2025-06-01 20:37:12,854 - INFO - Processing audio with duration 00:07.000 +2025-06-01 20:37:12,856 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:12,984 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:13,021 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:13,022 - INFO - Got result for chunk: output_test6/chunks/chunk_058.wav +2025-06-01 20:37:13,022 - INFO - +====== Streaming Chunk 059 ====== +[2025-06-01 20:36:11] Speaker 1: for the power one thing that i want to mention here is that matmon why we are using in most of these + +2025-06-01 20:37:13,022 - INFO - Processing audio with duration 5.220 seconds +2025-06-01 20:37:13,190 - INFO - Processing audio with duration 00:05.220 +2025-06-01 20:37:13,237 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:13,237 - INFO - Got result for chunk: output_test6/chunks/chunk_059.wav +2025-06-01 20:37:13,238 - INFO - +====== Streaming Chunk 060 ====== +[2025-06-01 20:36:11] Speaker 1: we are using MathMol. Why we are using MathMol here? MathMol is a good simulation of fine-tuning + +2025-06-01 20:37:13,238 - INFO - Processing audio with duration 4.660 seconds +2025-06-01 20:37:13,409 - INFO - Processing audio with duration 00:04.660 +2025-06-01 20:37:13,662 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:13,778 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:13,823 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:13,824 - INFO - Got result for chunk: output_test6/chunks/chunk_060.wav +2025-06-01 20:37:13,824 - INFO - +====== Streaming Chunk 061 ====== +[2025-06-01 20:36:11] Speaker 1: training llms and these type of things and it's very common for that so that's the reason why we + +2025-06-01 20:37:13,824 - INFO - Processing audio with duration 7.260 seconds +2025-06-01 20:37:13,993 - INFO - Processing audio with duration 00:07.260 +2025-06-01 20:37:14,029 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:14,029 - INFO - Got result for chunk: output_test6/chunks/chunk_061.wav +2025-06-01 20:37:14,029 - INFO - +====== Streaming Chunk 062 ====== +[2025-06-01 20:36:11] Speaker 1: I really focus on MatMod as one of the functions that we have used here a lot. + +2025-06-01 20:37:14,029 - INFO - Processing audio with duration 12.280 seconds +2025-06-01 20:37:14,206 - INFO - Processing audio with duration 00:12.280 +2025-06-01 20:37:14,208 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:14,336 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:14,367 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:14,367 - INFO - Got result for chunk: output_test6/chunks/chunk_062.wav +2025-06-01 20:37:14,367 - INFO - +====== Streaming Chunk 063 ====== +[2025-06-01 20:36:11] Speaker 1: i can uh the next function is about compute and gpu or compute throughput as i mentioned + +2025-06-01 20:37:14,367 - INFO - Processing audio with duration 10.660 seconds +2025-06-01 20:37:14,537 - INFO - Processing audio with duration 00:10.660 +2025-06-01 20:37:14,579 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:14,579 - INFO - Got result for chunk: output_test6/chunks/chunk_063.wav +2025-06-01 20:37:14,579 - INFO - +====== Streaming Chunk 064 ====== +[2025-06-01 20:36:11] Speaker 1: previously and this will measure the performance of the GPU cores under stress. +[2025-06-01 20:36:22] Speaker 1: Mainly it will + +2025-06-01 20:37:14,579 - INFO - Processing audio with duration 12.260 seconds +2025-06-01 20:37:14,768 - INFO - Processing audio with duration 00:12.260 +2025-06-01 20:37:14,771 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:14,898 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:14,939 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:14,939 - INFO - Got result for chunk: output_test6/chunks/chunk_064.wav +2025-06-01 20:37:14,940 - INFO - +====== Streaming Chunk 065 ====== +[2025-06-01 20:36:11] Speaker 1: say that how the gpus can handle the computation and again you can see here that you know this is +[2025-06-01 20:36:21] Speaker 1: also + +2025-06-01 20:37:14,940 - INFO - Processing audio with duration 7.340 seconds +2025-06-01 20:37:15,110 - INFO - Processing audio with duration 00:07.340 +2025-06-01 20:37:15,144 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:15,144 - INFO - Got result for chunk: output_test6/chunks/chunk_065.wav +2025-06-01 20:37:15,144 - INFO - +====== Streaming Chunk 066 ====== +[2025-06-01 20:36:11] Speaker 1: again based on matmul function the next one is the memory bandwidth the memory bandwidth is mainly + +2025-06-01 20:37:15,145 - INFO - Processing audio with duration 8.620 seconds +2025-06-01 20:37:15,323 - INFO - Processing audio with duration 00:08.620 +2025-06-01 20:37:15,325 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:15,453 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:15,491 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:15,491 - INFO - Got result for chunk: output_test6/chunks/chunk_066.wav +2025-06-01 20:37:15,491 - INFO - +====== Streaming Chunk 067 ====== +[2025-06-01 20:36:11] Speaker 1: um how the gpu can handle the heavy memory uh you know bound it means that you know how we can move + +2025-06-01 20:37:15,492 - INFO - Processing audio with duration 12.960 seconds +2025-06-01 20:37:15,668 - INFO - Processing audio with duration 00:12.960 +2025-06-01 20:37:15,706 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:15,707 - INFO - Got result for chunk: output_test6/chunks/chunk_067.wav +2025-06-01 20:37:15,707 - INFO - +====== Streaming Chunk 068 ====== +[2025-06-01 20:36:11] Speaker 1: the volume of data between the global memory and the compute you need. So here we are using + +2025-06-01 20:37:15,708 - INFO - Processing audio with duration 9.380 seconds +2025-06-01 20:37:15,938 - INFO - Processing audio with duration 00:09.380 +2025-06-01 20:37:16,128 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:16,242 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:16,297 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:16,298 - INFO - Got result for chunk: output_test6/chunks/chunk_068.wav +2025-06-01 20:37:16,298 - INFO - +====== Streaming Chunk 069 ====== +[2025-06-01 20:36:11] Speaker 1: uh addition b equal to a plus a +[2025-06-01 20:36:17] Speaker 1: uh and yeah the next one is about error detection on gpus so uh here + +2025-06-01 20:37:16,298 - INFO - Processing audio with duration 5.220 seconds +2025-06-01 20:37:16,468 - INFO - Processing audio with duration 00:05.220 +2025-06-01 20:37:16,507 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:16,507 - INFO - Got result for chunk: output_test6/chunks/chunk_069.wav +2025-06-01 20:37:16,507 - INFO - +====== Streaming Chunk 070 ====== +[2025-06-01 20:36:11] Speaker 1: this is mainly based on the accuracy it means that one time we are using CPU for +[2025-06-01 20:36:19] Speaker 1: you know + +2025-06-01 20:37:16,507 - INFO - Processing audio with duration 6.540 seconds +2025-06-01 20:37:16,678 - INFO - Processing audio with duration 00:06.540 +2025-06-01 20:37:16,680 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:16,807 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:16,848 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:16,848 - INFO - Got result for chunk: output_test6/chunks/chunk_070.wav +2025-06-01 20:37:16,848 - INFO - +====== Streaming Chunk 071 ====== +[2025-06-01 20:36:11] Speaker 1: matmol one time we are using gpus for matmol and you know we compare these numbers and we have some + +2025-06-01 20:37:16,848 - INFO - Processing audio with duration 9.200 seconds +2025-06-01 20:37:17,018 - INFO - Processing audio with duration 00:09.200 +2025-06-01 20:37:17,055 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:17,055 - INFO - Got result for chunk: output_test6/chunks/chunk_071.wav +2025-06-01 20:37:17,056 - INFO - +====== Streaming Chunk 072 ====== +[2025-06-01 20:36:11] Speaker 1: threshold for that as you can see here and if that's it's uh not significant we can say that + +2025-06-01 20:37:17,056 - INFO - Processing audio with duration 12.440 seconds +2025-06-01 20:37:17,228 - INFO - Processing audio with duration 00:12.440 +2025-06-01 20:37:17,231 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:17,358 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:17,389 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:17,389 - INFO - Got result for chunk: output_test6/chunks/chunk_072.wav +2025-06-01 20:37:17,389 - INFO - +====== Streaming Chunk 073 ====== +[2025-06-01 20:36:11] Speaker 1: the error detection is fine. Next thing is the tensor core utilization. This is the core + +2025-06-01 20:37:17,389 - INFO - Processing audio with duration 10.580 seconds +2025-06-01 20:37:17,560 - INFO - Processing audio with duration 00:10.580 +2025-06-01 20:37:17,603 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:17,604 - INFO - Got result for chunk: output_test6/chunks/chunk_073.wav +2025-06-01 20:37:17,604 - INFO - Processing audio with duration 9.000 seconds +2025-06-01 20:37:17,604 - INFO - +====== Streaming Chunk 074 ====== +[2025-06-01 20:36:11] Speaker 1: engagement and and it emphasizes on the low precision like the fp8 or fp16 um this is a mainly + +2025-06-01 20:37:17,795 - INFO - Processing audio with duration 00:09.000 +2025-06-01 20:37:17,798 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:17,925 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:17,955 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:17,955 - INFO - Got result for chunk: output_test6/chunks/chunk_074.wav +2025-06-01 20:37:17,956 - INFO - +====== Streaming Chunk 075 ====== +[2025-06-01 20:36:11] Speaker 1: hardware not on the software. After the Tensor Core utilization we did sustained workload and this + +2025-06-01 20:37:17,956 - INFO - Processing audio with duration 5.540 seconds +2025-06-01 20:37:18,125 - INFO - Processing audio with duration 00:05.540 +2025-06-01 20:37:18,166 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:18,166 - INFO - Got result for chunk: output_test6/chunks/chunk_075.wav +2025-06-01 20:37:18,166 - INFO - +====== Streaming Chunk 076 ====== +[2025-06-01 20:36:11] Speaker 1: test the long-term compute and stability. So for example here as an example we put 60 seconds. + +2025-06-01 20:37:18,166 - INFO - Processing audio with duration 9.840 seconds +2025-06-01 20:37:18,340 - INFO - Processing audio with duration 00:09.840 +2025-06-01 20:37:18,586 - INFO - Detected language 'en' with probability 0.99 +2025-06-01 20:37:18,700 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:18,749 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:18,750 - INFO - Got result for chunk: output_test6/chunks/chunk_076.wav +2025-06-01 20:37:18,750 - INFO - +====== Streaming Chunk 077 ====== +[2025-06-01 20:36:11] Speaker 1: and we are running matmod for 60 seconds and we will see that how the gpus are working based on that + +2025-06-01 20:37:18,750 - INFO - Processing audio with duration 11.900 seconds +2025-06-01 20:37:18,924 - INFO - Processing audio with duration 00:11.900 +2025-06-01 20:37:18,958 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:18,959 - INFO - Got result for chunk: output_test6/chunks/chunk_077.wav +2025-06-01 20:37:18,959 - INFO - Processing audio with duration 6.440 seconds +2025-06-01 20:37:18,959 - INFO - +====== Streaming Chunk 078 ====== +[2025-06-01 20:36:11] Speaker 1: and the last one is mixed precision testing and this will validate both the performance and the + +2025-06-01 20:37:19,152 - INFO - Processing audio with duration 00:06.440 +2025-06-01 20:37:19,155 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:19,282 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:19,330 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:19,330 - INFO - Got result for chunk: output_test6/chunks/chunk_078.wav +2025-06-01 20:37:19,330 - INFO - +====== Streaming Chunk 079 ====== +[2025-06-01 20:36:11] Speaker 1: correctness of the GPUs again when we are using matmult here. +[2025-06-01 20:36:19] Speaker 1: So with that, you know, we got to the + +2025-06-01 20:37:19,330 - INFO - Processing audio with duration 6.060 seconds +2025-06-01 20:37:19,504 - INFO - Processing audio with duration 00:06.060 +2025-06-01 20:37:19,537 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:19,537 - INFO - Got result for chunk: output_test6/chunks/chunk_079.wav +2025-06-01 20:37:19,537 - INFO - +====== Streaming Chunk 080 ====== +[2025-06-01 20:36:11] Speaker 1: end of this part. I will give you an example of how this has worked previously. + +2025-06-01 20:37:19,538 - INFO - Processing audio with duration 1.520 seconds +2025-06-01 20:37:19,703 - INFO - Processing audio with duration 00:01.520 +2025-06-01 20:37:19,705 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:19,832 - INFO - Detected language 'en' with probability 0.99 +2025-06-01 20:37:19,862 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:19,862 - INFO - Got result for chunk: output_test6/chunks/chunk_080.wav +2025-06-01 20:37:19,862 - INFO - +====== Streaming Chunk 081 ====== +[2025-06-01 20:36:11] Speaker 1: So we had an H100 and you can see the result of the H100 here + +2025-06-01 20:37:19,862 - INFO - Processing audio with duration 6.640 seconds +2025-06-01 20:37:20,032 - INFO - Processing audio with duration 00:06.640 +2025-06-01 20:37:20,049 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:20,050 - INFO - Got result for chunk: output_test6/chunks/chunk_081.wav +2025-06-01 20:37:20,050 - INFO - +====== Streaming Chunk 082 ====== +[2025-06-01 20:36:11] Speaker 1: And, um... + +2025-06-01 20:37:20,050 - INFO - Processing audio with duration 7.340 seconds +2025-06-01 20:37:20,217 - INFO - Processing audio with duration 00:07.340 +2025-06-01 20:37:20,220 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:20,346 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:20,379 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:20,379 - INFO - Got result for chunk: output_test6/chunks/chunk_082.wav +2025-06-01 20:37:20,380 - INFO - +====== Streaming Chunk 083 ====== +[2025-06-01 20:36:11] Speaker 1: We saw that the H100 is not performing very well. +[2025-06-01 20:36:15] Speaker 1: I went and saw + +2025-06-01 20:37:20,380 - INFO - Processing audio with duration 7.500 seconds +2025-06-01 20:37:20,550 - INFO - Processing audio with duration 00:07.500 +2025-06-01 20:37:20,595 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:20,596 - INFO - Got result for chunk: output_test6/chunks/chunk_083.wav +2025-06-01 20:37:20,596 - INFO - +====== Streaming Chunk 084 ====== +[2025-06-01 20:36:11] Speaker 1: specifically for gpu 4 and you can see that you know in the power the gpu 4 is uh going up very + +2025-06-01 20:37:20,596 - INFO - Processing audio with duration 7.800 seconds +2025-06-01 20:37:20,769 - INFO - Processing audio with duration 00:07.800 +2025-06-01 20:37:21,024 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:21,139 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:21,185 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:21,185 - INFO - Got result for chunk: output_test6/chunks/chunk_084.wav +2025-06-01 20:37:21,185 - INFO - +====== Streaming Chunk 085 ====== +[2025-06-01 20:36:11] Speaker 1: you know fast compared to other ones but you know after that the rest of that went sharply up but this + +2025-06-01 20:37:21,185 - INFO - Processing audio with duration 8.580 seconds +2025-06-01 20:37:21,357 - INFO - Processing audio with duration 00:08.580 +2025-06-01 20:37:21,408 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:21,409 - INFO - Got result for chunk: output_test6/chunks/chunk_085.wav +2025-06-01 20:37:21,409 - INFO - +====== Streaming Chunk 086 ====== +[2025-06-01 20:36:11] Speaker 1: one didn't go up uh so maybe that was the reason why the h100 was not performing very well then i +[2025-06-01 20:36:18] Speaker 1: also + +2025-06-01 20:37:21,409 - INFO - Processing audio with duration 6.560 seconds +2025-06-01 20:37:21,580 - INFO - Processing audio with duration 00:06.560 +2025-06-01 20:37:21,582 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:21,709 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:21,752 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:21,752 - INFO - Got result for chunk: output_test6/chunks/chunk_086.wav +2025-06-01 20:37:21,752 - INFO - +====== Streaming Chunk 087 ====== +[2025-06-01 20:36:11] Speaker 1: saw the um the temperature you can see that for gpu for the temperature again is sharply going up like +[2025-06-01 20:36:19] Speaker 1: 80. + +2025-06-01 20:37:21,752 - INFO - Processing audio with duration 7.560 seconds +2025-06-01 20:37:21,922 - INFO - Processing audio with duration 00:07.560 +2025-06-01 20:37:21,958 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:21,959 - INFO - Got result for chunk: output_test6/chunks/chunk_087.wav +2025-06-01 20:37:21,959 - INFO - +====== Streaming Chunk 088 ====== +[2025-06-01 20:36:11] Speaker 1: which is above the range that you know we specified and then you know it stays there so + +2025-06-01 20:37:21,959 - INFO - Processing audio with duration 6.420 seconds +2025-06-01 20:37:22,127 - INFO - Processing audio with duration 00:06.420 +2025-06-01 20:37:22,130 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:22,256 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:22,294 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:22,294 - INFO - Got result for chunk: output_test6/chunks/chunk_088.wav +2025-06-01 20:37:22,295 - INFO - +====== Streaming Chunk 089 ====== +[2025-06-01 20:36:11] Speaker 1: maybe that was the reason why you know it wasn't able to uh you know do power good and the rest of + +2025-06-01 20:37:22,295 - INFO - Processing audio with duration 8.800 seconds +2025-06-01 20:37:22,466 - INFO - Processing audio with duration 00:08.800 +2025-06-01 20:37:22,521 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:22,521 - INFO - Got result for chunk: output_test6/chunks/chunk_089.wav +2025-06-01 20:37:22,522 - INFO - +====== Streaming Chunk 090 ====== +[2025-06-01 20:36:11] Speaker 1: that you know you can see that you know it's not even close to 86 so we found out that you know the +[2025-06-01 20:36:16] Speaker 1: gpu 4 + +2025-06-01 20:37:22,522 - INFO - Processing audio with duration 6.240 seconds +2025-06-01 20:37:22,690 - INFO - Processing audio with duration 00:06.240 +2025-06-01 20:37:22,693 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:22,820 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:22,855 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:22,855 - INFO - Got result for chunk: output_test6/chunks/chunk_090.wav +2025-06-01 20:37:22,855 - INFO - +====== Streaming Chunk 091 ====== +[2025-06-01 20:36:11] Speaker 1: is not working very well but one thing that you know here i want to emphasize is that this is inside + +2025-06-01 20:37:22,855 - INFO - Processing audio with duration 8.240 seconds +2025-06-01 20:37:23,027 - INFO - Processing audio with duration 00:08.240 +2025-06-01 20:37:23,076 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:23,077 - INFO - Got result for chunk: output_test6/chunks/chunk_091.wav +2025-06-01 20:37:23,077 - INFO - +====== Streaming Chunk 092 ====== +[2025-06-01 20:36:11] Speaker 1: a node so what does it mean it means that you know for example in the h100 and knows we have eight gpu + +2025-06-01 20:37:23,077 - INFO - Processing audio with duration 12.180 seconds +2025-06-01 20:37:23,252 - INFO - Processing audio with duration 00:12.180 +2025-06-01 20:37:23,487 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:23,602 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:23,648 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:23,648 - INFO - Got result for chunk: output_test6/chunks/chunk_092.wav +2025-06-01 20:37:23,648 - INFO - +====== Streaming Chunk 093 ====== +[2025-06-01 20:36:11] Speaker 1: so it will give us the result for the hgpus but we have also extended this uh code for + +2025-06-01 20:37:23,648 - INFO - Processing audio with duration 10.560 seconds +2025-06-01 20:37:23,825 - INFO - Processing audio with duration 00:10.560 +2025-06-01 20:37:23,868 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:23,869 - INFO - Got result for chunk: output_test6/chunks/chunk_093.wav +2025-06-01 20:37:23,869 - INFO - Processing audio with duration 6.980 seconds +2025-06-01 20:37:23,869 - INFO - +====== Streaming Chunk 094 ====== +[2025-06-01 20:36:11] Speaker 1: multi-node and by that so this was this code was given to us by another team from John Shelley's + +2025-06-01 20:37:24,060 - INFO - Processing audio with duration 00:06.980 +2025-06-01 20:37:24,063 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:24,190 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:24,218 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:24,218 - INFO - Got result for chunk: output_test6/chunks/chunk_094.wav +2025-06-01 20:37:24,219 - INFO - +====== Streaming Chunk 095 ====== +[2025-06-01 20:36:11] Speaker 1: theme and it's mainly doing some more tests about nodes between the nodes specifically it + +2025-06-01 20:37:24,219 - INFO - Processing audio with duration 9.440 seconds +2025-06-01 20:37:24,391 - INFO - Processing audio with duration 00:09.440 +2025-06-01 20:37:24,439 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:24,439 - INFO - Got result for chunk: output_test6/chunks/chunk_095.wav +2025-06-01 20:37:24,439 - INFO - +====== Streaming Chunk 096 ====== +[2025-06-01 20:36:11] Speaker 1: It will evaluate the RDMA, the different type of errors like error correctness code. +[2025-06-01 20:36:17] Speaker 1: It will also... + +2025-06-01 20:37:24,440 - INFO - Processing audio with duration 5.380 seconds +2025-06-01 20:37:24,609 - INFO - Processing audio with duration 00:05.380 +2025-06-01 20:37:24,611 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:24,738 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:24,778 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:24,779 - INFO - Got result for chunk: output_test6/chunks/chunk_096.wav +2025-06-01 20:37:24,779 - INFO - +====== Streaming Chunk 097 ====== +[2025-06-01 20:36:11] Speaker 1: evaluate the PCIe link and this type of things. +[2025-06-01 20:36:17] Speaker 1: The way that our structure is working right now... + +2025-06-01 20:37:24,779 - INFO - Processing audio with duration 4.600 seconds +2025-06-01 20:37:24,954 - INFO - Processing audio with duration 00:04.600 +2025-06-01 20:37:24,985 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:24,985 - INFO - Got result for chunk: output_test6/chunks/chunk_097.wav +2025-06-01 20:37:24,986 - INFO - +====== Streaming Chunk 098 ====== +[2025-06-01 20:36:11] Speaker 1: is that first of all we run if you have a multi node we run this one + +2025-06-01 20:37:24,986 - INFO - Processing audio with duration 4.700 seconds +2025-06-01 20:37:25,158 - INFO - Processing audio with duration 00:04.700 +2025-06-01 20:37:25,161 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:25,287 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:25,313 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:25,313 - INFO - Got result for chunk: output_test6/chunks/chunk_098.wav +2025-06-01 20:37:25,313 - INFO - +====== Streaming Chunk 099 ====== +[2025-06-01 20:36:11] Speaker 1: uh let me show you the result or the log maybe that would help + +2025-06-01 20:37:25,314 - INFO - Processing audio with duration 8.720 seconds +2025-06-01 20:37:25,486 - INFO - Processing audio with duration 00:08.720 +2025-06-01 20:37:25,527 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:25,527 - INFO - Got result for chunk: output_test6/chunks/chunk_099.wav +2025-06-01 20:37:25,527 - INFO - Processing audio with duration 8.960 seconds +2025-06-01 20:37:25,527 - INFO - +====== Streaming Chunk 100 ====== +[2025-06-01 20:36:11] Speaker 1: First of all, we'll run that one. For example here I used two nodes and you can see + +2025-06-01 20:37:25,697 - INFO - Processing audio with duration 00:08.960 +2025-06-01 20:37:25,945 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:26,061 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:26,108 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:26,109 - INFO - Got result for chunk: output_test6/chunks/chunk_100.wav +2025-06-01 20:37:26,109 - INFO - +====== Streaming Chunk 101 ====== +[2025-06-01 20:36:11] Speaker 1: that you know it will go and run all of these ones one by one and it will get to this end. + +2025-06-01 20:37:26,109 - INFO - Processing audio with duration 6.540 seconds +2025-06-01 20:37:26,282 - INFO - Processing audio with duration 00:06.540 +2025-06-01 20:37:26,332 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:26,332 - INFO - Got result for chunk: output_test6/chunks/chunk_101.wav +2025-06-01 20:37:26,332 - INFO - Processing audio with duration 1.720 seconds +2025-06-01 20:37:26,332 - INFO - +====== Streaming Chunk 102 ====== +[2025-06-01 20:36:11] Speaker 1: Once this is done, it will go inside each node. +[2025-06-01 20:36:15] Speaker 1: For example, here we have the GPU 471. + +2025-06-01 20:37:26,493 - INFO - Processing audio with duration 00:01.720 +2025-06-01 20:37:26,496 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:26,622 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:26,661 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:26,661 - INFO - Got result for chunk: output_test6/chunks/chunk_102.wav +2025-06-01 20:37:26,661 - INFO - +====== Streaming Chunk 103 ====== +[2025-06-01 20:36:11] Speaker 1: here everything will be saved as a json file and this multi-node result will be saved as a json + +2025-06-01 20:37:26,661 - INFO - Processing audio with duration 11.260 seconds +2025-06-01 20:37:26,835 - INFO - Processing audio with duration 00:11.260 +2025-06-01 20:37:26,860 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:26,861 - INFO - Got result for chunk: output_test6/chunks/chunk_103.wav +2025-06-01 20:37:26,861 - INFO - +====== Streaming Chunk 104 ====== +[2025-06-01 20:36:11] Speaker 1: file and you can see that you know it's here + +2025-06-01 20:37:26,864 - INFO - Processing audio with duration 8.280 seconds +2025-06-01 20:37:27,062 - INFO - Processing audio with duration 00:08.280 +2025-06-01 20:37:27,064 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:27,191 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:27,227 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:27,227 - INFO - Got result for chunk: output_test6/chunks/chunk_104.wav +2025-06-01 20:37:27,227 - INFO - +====== Streaming Chunk 105 ====== +[2025-06-01 20:36:11] Speaker 1: Again, for the other one, I think it was another GPU, which was probably the 509. + +2025-06-01 20:37:27,228 - INFO - Processing audio with duration 8.100 seconds +2025-06-01 20:37:27,399 - INFO - Processing audio with duration 00:08.100 +2025-06-01 20:37:27,442 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:27,442 - INFO - Got result for chunk: output_test6/chunks/chunk_105.wav +2025-06-01 20:37:27,442 - INFO - Processing audio with duration 0.980 seconds +2025-06-01 20:37:27,442 - INFO - +====== Streaming Chunk 106 ====== +[2025-06-01 20:36:11] Speaker 1: and it will again all of the json file will uh you know save all of these results then it will go + +2025-06-01 20:37:27,600 - INFO - Processing audio with duration 00:00.980 +2025-06-01 20:37:27,603 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:27,729 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:27,771 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:27,771 - INFO - Got result for chunk: output_test6/chunks/chunk_106.wav +2025-06-01 20:37:27,771 - INFO - +====== Streaming Chunk 107 ====== +[2025-06-01 20:36:11] Speaker 1: inside the gpu 471 and inside that gpu it will run the health check that you know and we just we were + +2025-06-01 20:37:27,771 - INFO - Processing audio with duration 9.380 seconds +2025-06-01 20:37:27,945 - INFO - Processing audio with duration 00:09.380 +2025-06-01 20:37:27,961 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:27,961 - INFO - Got result for chunk: output_test6/chunks/chunk_107.wav +2025-06-01 20:37:27,962 - INFO - +====== Streaming Chunk 108 ====== +[2025-06-01 20:36:11] Speaker 1: just talking about. + +2025-06-01 20:37:27,962 - INFO - Processing audio with duration 8.180 seconds +2025-06-01 20:37:28,133 - INFO - Processing audio with duration 00:08.180 +2025-06-01 20:37:28,406 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:28,520 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:28,577 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:28,577 - INFO - Got result for chunk: output_test6/chunks/chunk_108.wav +2025-06-01 20:37:28,578 - INFO - +====== Streaming Chunk 109 ====== +[2025-06-01 20:36:11] Speaker 1: it will go it will do that for gpu 471 again it will do that for the other gpu gpu 507 which have + +2025-06-01 20:37:28,578 - INFO - Processing audio with duration 12.960 seconds +2025-06-01 20:37:28,750 - INFO - Processing audio with duration 00:12.960 +2025-06-01 20:37:28,798 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:28,799 - INFO - Got result for chunk: output_test6/chunks/chunk_109.wav +2025-06-01 20:37:28,799 - INFO - +====== Streaming Chunk 110 ====== +[2025-06-01 20:36:11] Speaker 1: which we have here, then it will give us the result and the testing result. + +2025-06-01 20:37:28,800 - INFO - Processing audio with duration 6.940 seconds +2025-06-01 20:37:28,991 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:28,992 - INFO - Processing audio with duration 00:06.940 +2025-06-01 20:37:29,119 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:29,167 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:29,167 - INFO - Got result for chunk: output_test6/chunks/chunk_110.wav +2025-06-01 20:37:29,167 - INFO - +====== Streaming Chunk 111 ====== +[2025-06-01 20:36:11] Speaker 1: so you will have um you know a pdf and a json file and a log for each of the gpu 431 and 507 here + +2025-06-01 20:37:29,167 - INFO - Processing audio with duration 6.840 seconds +2025-06-01 20:37:29,341 - INFO - Processing audio with duration 00:06.840 +2025-06-01 20:37:29,370 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:29,371 - INFO - Got result for chunk: output_test6/chunks/chunk_111.wav +2025-06-01 20:37:29,371 - INFO - +====== Streaming Chunk 112 ====== +[2025-06-01 20:36:11] Speaker 1: These PDFs will be saved based on the timing and calendar. + +2025-06-01 20:37:29,371 - INFO - Processing audio with duration 9.720 seconds +2025-06-01 20:37:29,539 - INFO - Processing audio with duration 00:09.720 +2025-06-01 20:37:29,541 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:29,667 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:29,703 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:29,703 - INFO - Got result for chunk: output_test6/chunks/chunk_112.wav +2025-06-01 20:37:29,703 - INFO - +====== Streaming Chunk 113 ====== +[2025-06-01 20:36:11] Speaker 1: uh with that uh this was the main idea about the health check and you know how we are running that + +2025-06-01 20:37:29,703 - INFO - Processing audio with duration 2.720 seconds +2025-06-01 20:37:29,878 - INFO - Processing audio with duration 00:02.720 +2025-06-01 20:37:29,921 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 20:37:29,921 - INFO - Got result for chunk: output_test6/chunks/chunk_113.wav +2025-06-01 20:37:29,921 - INFO - +====== Streaming Chunk 114 ====== +[2025-06-01 20:36:11] Speaker 1: And I really appreciate your time. +[2025-06-01 20:36:14] Speaker 1: And I would be happy if you can, you know, reach out. + +2025-06-01 20:37:30,005 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 20:37:30,153 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 20:37:30,153 - INFO - Got result for chunk: output_test6/chunks/chunk_114.wav +2025-06-01 20:37:30,153 - INFO - +====== Streaming Chunk 115 ====== +[2025-06-01 20:36:11] Speaker 1: any question about this. Thank you so much. + +2025-06-01 20:37:30,224 - INFO - Transcript saved to: output_test6/video1591686795_all_transcripts_20250601_203730.txt +2025-06-01 20:37:30,224 - INFO - CUDA available before summarization: True +2025-06-01 20:37:30,224 - INFO - Loading summarizer model from HuggingFace: mistralai/Mistral-7B-Instruct-v0.1 +2025-06-01 20:37:39,942 - INFO - Summarizing chunk 1/3 +2025-06-01 20:37:56,777 - INFO - Summarizing chunk 2/3 +2025-06-01 20:38:07,633 - INFO - Summarizing chunk 3/3 +2025-06-01 20:38:11,573 - INFO - Summary successfully appended to transcript TXT. +2025-06-01 20:38:11,575 - INFO - Final JSON saved to: output_test6/video1591686795_all_transcripts_20250601_203730.json diff --git a/docs/whisper_transcription/Examples/Test2/video1591686795.mp4 b/docs/whisper_transcription/Examples/Test2/video1591686795.mp4 new file mode 100644 index 0000000..a02a2e0 Binary files /dev/null and b/docs/whisper_transcription/Examples/Test2/video1591686795.mp4 differ diff --git a/docs/whisper_transcription/Examples/Test2/video1591686795_all_transcripts_20250601_203730.json b/docs/whisper_transcription/Examples/Test2/video1591686795_all_transcripts_20250601_203730.json new file mode 100644 index 0000000..0d4b447 --- /dev/null +++ b/docs/whisper_transcription/Examples/Test2/video1591686795_all_transcripts_20250601_203730.json @@ -0,0 +1,1521 @@ +{ + "chunks": { + "chunk_000.wav": { + "chunk_id": "001", + "transcript": "[2025-06-01 20:36:11] Speaker 1: hello everyone thank you for joining to this session this session we will talk about health", + "segments": [ + { + "text": "hello everyone thank you for joining to this session this session we will talk about health", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_001.wav": { + "chunk_id": "002", + "transcript": "[2025-06-01 20:36:11] Speaker 1: recipe that we provided in seeds group Ammar, Soumya and me etc. we are working on this.", + "segments": [ + { + "text": "recipe that we provided in seeds group Ammar, Soumya and me etc. we are working on this.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_002.wav": { + "chunk_id": "003", + "transcript": "[2025-06-01 20:36:11] Speaker 1: a resume and with that we can start to talk about that so the main idea of the health check", + "segments": [ + { + "text": "a resume and with that we can start to talk about that so the main idea of the health check", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_003.wav": { + "chunk_id": "004", + "transcript": "[2025-06-01 20:36:11] Speaker 1: recipe is to make the customer confidence that they have enough resources to run heavy workloads", + "segments": [ + { + "text": "recipe is to make the customer confidence that they have enough resources to run heavy workloads", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_004.wav": { + "chunk_id": "005", + "transcript": "[2025-06-01 20:36:11] Speaker 1: what i mean by that is that most of the customers when they want to run the llm's training find\n[2025-06-01 20:36:29] Speaker 1: you", + "segments": [ + { + "text": "what i mean by that is that most of the customers when they want to run the llm's training find", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "you", + "timestamp": "2025-06-01 20:36:29", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_005.wav": { + "chunk_id": "006", + "transcript": "[2025-06-01 20:36:11] Speaker 1: tuning or any other things they need which assumed to be a heavy workload they need to have a lot of", + "segments": [ + { + "text": "tuning or any other things they need which assumed to be a heavy workload they need to have a lot of", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_006.wav": { + "chunk_id": "007", + "transcript": "[2025-06-01 20:36:11] Speaker 1: gpus and we want to make sure that these gpus work well so they can run this recipe before starting", + "segments": [ + { + "text": "gpus and we want to make sure that these gpus work well so they can run this recipe before starting", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_007.wav": { + "chunk_id": "008", + "transcript": "[2025-06-01 20:36:11] Speaker 1: this task and to just make sure that you know everything is working very well i'll go ahead", + "segments": [ + { + "text": "this task and to just make sure that you know everything is working very well i'll go ahead", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_008.wav": { + "chunk_id": "009", + "transcript": "[2025-06-01 20:36:11] Speaker 1: share my screen and yeah i will start with the result of this recipe i can say that this you know", + "segments": [ + { + "text": "share my screen and yeah i will start with the result of this recipe i can say that this you know", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_009.wav": { + "chunk_id": "010", + "transcript": "[2025-06-01 20:36:11] Speaker 1: recipe the health check recipe will provide you two different formats one is the pdf which i'm showing", + "segments": [ + { + "text": "recipe the health check recipe will provide you two different formats one is the pdf which i'm showing", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_010.wav": { + "chunk_id": "011", + "transcript": "[2025-06-01 20:36:11] Speaker 1: to you and the other one is json one i will also go through that as well but here you can see uh what", + "segments": [ + { + "text": "to you and the other one is json one i will also go through that as well but here you can see uh what", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_011.wav": { + "chunk_id": "012", + "transcript": "[2025-06-01 20:36:11] Speaker 1: test we are evaluating for the gpus uh we are doing bunch of uh testing like the background", + "segments": [ + { + "text": "test we are evaluating for the gpus uh we are doing bunch of uh testing like the background", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_012.wav": { + "chunk_id": "013", + "transcript": "[2025-06-01 20:36:11] Speaker 1: compare computation compute throughput memory bandwidth error detection tensor core utilization\n[2025-06-01 20:36:17] Speaker 1: sustain workload", + "segments": [ + { + "text": "compare computation compute throughput memory bandwidth error detection tensor core utilization", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "sustain workload", + "timestamp": "2025-06-01 20:36:17", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_013.wav": { + "chunk_id": "014", + "transcript": "[2025-06-01 20:36:11] Speaker 1: mixed precision, power, temperature and utilization. So these are the main tests that we are", + "segments": [ + { + "text": "mixed precision, power, temperature and utilization. So these are the main tests that we are", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_014.wav": { + "chunk_id": "015", + "transcript": "[2025-06-01 20:36:11] Speaker 1: evaluating you can see that in this format once you know you run the health\n[2025-06-01 20:36:15] Speaker 1: show", + "segments": [ + { + "text": "evaluating you can see that in this format once you know you run the health", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "show", + "timestamp": "2025-06-01 20:36:15", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_015.wav": { + "chunk_id": "016", + "transcript": "[2025-06-01 20:36:11] Speaker 1: recipe you will get the result you know everything is in a table the duration of each", + "segments": [ + { + "text": "recipe you will get the result you know everything is in a table the duration of each", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_016.wav": { + "chunk_id": "017", + "transcript": "[2025-06-01 20:36:11] Speaker 1: test, whether it has failed or not.\n[2025-06-01 20:36:18] Speaker 1: For example, here for compute throughput,", + "segments": [ + { + "text": "test, whether it has failed or not.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "For example, here for compute throughput,", + "timestamp": "2025-06-01 20:36:18", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_017.wav": { + "chunk_id": "018", + "transcript": "[2025-06-01 20:36:11] Speaker 1: has failed and i will show you uh in detail that how it works and for those customers who you know\n[2025-06-01 20:36:19] Speaker 1: maybe", + "segments": [ + { + "text": "has failed and i will show you uh in detail that how it works and for those customers who you know", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "maybe", + "timestamp": "2025-06-01 20:36:19", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_018.wav": { + "chunk_id": "019", + "transcript": "[2025-06-01 20:36:11] Speaker 1: don't know what are these numbers you know they can go to the end of that we\n[2025-06-01 20:36:17] Speaker 1: have also provided a graph", + "segments": [ + { + "text": "don't know what are these numbers you know they can go to the end of that we", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "have also provided a graph", + "timestamp": "2025-06-01 20:36:17", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_019.wav": { + "chunk_id": "020", + "transcript": "[2025-06-01 20:36:11] Speaker 1: I'm waiting for that. We have also provided a graph. In this graph, we have provided, you know,", + "segments": [ + { + "text": "I'm waiting for that. We have also provided a graph. In this graph, we have provided, you know,", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_020.wav": { + "chunk_id": "021", + "transcript": "[2025-06-01 20:36:11] Speaker 1: So first what we did was that we did an extensive research from NVIDIA website articles to see what are the...", + "segments": [ + { + "text": "So first what we did was that we did an extensive research from NVIDIA website articles to see what are the...", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_021.wav": { + "chunk_id": "022", + "transcript": "[2025-06-01 20:36:11] Speaker 1: uh you know ideal uh range for for example temperature the compute throughput memory bandwidth\n[2025-06-01 20:36:17] Speaker 1: and this type of", + "segments": [ + { + "text": "uh you know ideal uh range for for example temperature the compute throughput memory bandwidth", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "and this type of", + "timestamp": "2025-06-01 20:36:17", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_022.wav": { + "chunk_id": "023", + "transcript": "[2025-06-01 20:36:11] Speaker 1: things. What is the upper threshold? What is the lower threshold? We added a 10% to each of", + "segments": [ + { + "text": "things. What is the upper threshold? What is the lower threshold? We added a 10% to each of", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_023.wav": { + "chunk_id": "024", + "transcript": "[2025-06-01 20:36:11] Speaker 1: this once and uh we got the average of that these average are shown here as a baseline for example\n[2025-06-01 20:36:18] Speaker 1: 3000 is", + "segments": [ + { + "text": "this once and uh we got the average of that these average are shown here as a baseline for example", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "3000 is", + "timestamp": "2025-06-01 20:36:18", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_024.wav": { + "chunk_id": "025", + "transcript": "[2025-06-01 20:36:11] Speaker 1: shown here as a baseline and the measured one is the for example uh the yellow one it shows that uh\n[2025-06-01 20:36:19] Speaker 1: you know", + "segments": [ + { + "text": "shown here as a baseline and the measured one is the for example uh the yellow one it shows that uh", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "you know", + "timestamp": "2025-06-01 20:36:19", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_025.wav": { + "chunk_id": "026", + "transcript": "[2025-06-01 20:36:11] Speaker 1: So once we are using the test compute throughput, the measure was around 651.92, which is around 630.", + "segments": [ + { + "text": "So once we are using the test compute throughput, the measure was around 651.92, which is around 630.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_026.wav": { + "chunk_id": "027", + "transcript": "[2025-06-01 20:36:11] Speaker 1: uh 30 which was the baseline so the main purpose of this graph is to make the customer sure that", + "segments": [ + { + "text": "uh 30 which was the baseline so the main purpose of this graph is to make the customer sure that", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_027.wav": { + "chunk_id": "028", + "transcript": "[2025-06-01 20:36:11] Speaker 1: numbers that they are getting are in the same kind of range of you know what the\n[2025-06-01 20:36:18] Speaker 1: number should be", + "segments": [ + { + "text": "numbers that they are getting are in the same kind of range of you know what the", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "number should be", + "timestamp": "2025-06-01 20:36:18", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_028.wav": { + "chunk_id": "029", + "transcript": "[2025-06-01 20:36:11] Speaker 1: With that, I can also go to the JSON file.", + "segments": [ + { + "text": "With that, I can also go to the JSON file.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_029.wav": { + "chunk_id": "030", + "transcript": "[2025-06-01 20:36:11] Speaker 1: which we have here. I will just open one of them.", + "segments": [ + { + "text": "which we have here. I will just open one of them.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_030.wav": { + "chunk_id": "031", + "transcript": "[2025-06-01 20:36:11] Speaker 1: So the way that they know the JSON file.", + "segments": [ + { + "text": "So the way that they know the JSON file.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_031.wav": { + "chunk_id": "032", + "transcript": "[2025-06-01 20:36:11] Speaker 1: is working is that you know basically the same numbers are both in the PDF and the JSON file.", + "segments": [ + { + "text": "is working is that you know basically the same numbers are both in the PDF and the JSON file.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_032.wav": { + "chunk_id": "033", + "transcript": "[2025-06-01 20:36:11] Speaker 1: The good thing about JSON file here is that, you know, for example, for compute throughput,", + "segments": [ + { + "text": "The good thing about JSON file here is that, you know, for example, for compute throughput,", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_033.wav": { + "chunk_id": "034", + "transcript": "[2025-06-01 20:36:11] Speaker 1: it has failed if it go on each gpu one by one um based on the tensor and you know duration", + "segments": [ + { + "text": "it has failed if it go on each gpu one by one um based on the tensor and you know duration", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_034.wav": { + "chunk_id": "035", + "transcript": "[2025-06-01 20:36:11] Speaker 1: flops bandwidth and it will uh make the comparison with the baseline and if uh for all of them it's not", + "segments": [ + { + "text": "flops bandwidth and it will uh make the comparison with the baseline and if uh for all of them it's not", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_035.wav": { + "chunk_id": "036", + "transcript": "[2025-06-01 20:36:11] Speaker 1: in the range that you know we specify it will give us the fail. Similarly for the memory band", + "segments": [ + { + "text": "in the range that you know we specify it will give us the fail. Similarly for the memory band", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_036.wav": { + "chunk_id": "037", + "transcript": "[2025-06-01 20:36:11] Speaker 1: it you know it has some for each gpu it will say that you know what does what are the information", + "segments": [ + { + "text": "it you know it has some for each gpu it will say that you know what does what are the information", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_037.wav": { + "chunk_id": "038", + "transcript": "[2025-06-01 20:36:11] Speaker 1: at the end it will give whether it is ideal or not so this is the way that uh you know the", + "segments": [ + { + "text": "at the end it will give whether it is ideal or not so this is the way that uh you know the", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_038.wav": { + "chunk_id": "039", + "transcript": "[2025-06-01 20:36:11] Speaker 1: JSON file is working. With that, I can go inside the code and I can talk a little bit more about", + "segments": [ + { + "text": "JSON file is working. With that, I can go inside the code and I can talk a little bit more about", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_039.wav": { + "chunk_id": "040", + "transcript": "[2025-06-01 20:36:11] Speaker 1: the code what we have provided for the customer so we are we have enabled the customer to choose", + "segments": [ + { + "text": "the code what we have provided for the customer so we are we have enabled the customer to choose", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_040.wav": { + "chunk_id": "041", + "transcript": "[2025-06-01 20:36:11] Speaker 1: what type, what D-type, you know, they can choose.\n[2025-06-01 20:36:14] Speaker 1: They can go with FV32, FV16, which is the most common one.", + "segments": [ + { + "text": "what type, what D-type, you know, they can choose.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "They can go with FV32, FV16, which is the most common one.", + "timestamp": "2025-06-01 20:36:14", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_041.wav": { + "chunk_id": "042", + "transcript": "[2025-06-01 20:36:11] Speaker 1: uh fp8 uh we have also enabled them that whether they want to just choose one um uh you know one of", + "segments": [ + { + "text": "uh fp8 uh we have also enabled them that whether they want to just choose one um uh you know one of", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_042.wav": { + "chunk_id": "043", + "transcript": "[2025-06-01 20:36:11] Speaker 1: these uh uh functions or you know all of them the default is on all of them also they can choose", + "segments": [ + { + "text": "these uh uh functions or you know all of them the default is on all of them also they can choose", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_043.wav": { + "chunk_id": "044", + "transcript": "[2025-06-01 20:36:11] Speaker 1: to do the direction so we have provided a couple of uh you know uh parser uh so that you know", + "segments": [ + { + "text": "to do the direction so we have provided a couple of uh you know uh parser uh so that you know", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_044.wav": { + "chunk_id": "045", + "transcript": "[2025-06-01 20:36:11] Speaker 1: can choose uh uh you know uh to play with them uh code one important thing is the tensor shape", + "segments": [ + { + "text": "can choose uh uh you know uh to play with them uh code one important thing is the tensor shape", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_045.wav": { + "chunk_id": "046", + "transcript": "[2025-06-01 20:36:11] Speaker 1: example for a10 the tensor shape is 8192 this is the default that we have provided for all of the\n[2025-06-01 20:36:20] Speaker 1: a10", + "segments": [ + { + "text": "example for a10 the tensor shape is 8192 this is the default that we have provided for all of the", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "a10", + "timestamp": "2025-06-01 20:36:20", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_046.wav": { + "chunk_id": "047", + "transcript": "[2025-06-01 20:36:11] Speaker 1: a hundred and h hundred um with this i will give you some information that we can we also measure\n[2025-06-01 20:36:20] Speaker 1: the highest", + "segments": [ + { + "text": "a hundred and h hundred um with this i will give you some information that we can we also measure", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "the highest", + "timestamp": "2025-06-01 20:36:20", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_047.wav": { + "chunk_id": "048", + "transcript": "[2025-06-01 20:36:11] Speaker 1: uh tensor for example for 800 we can also use 65k for that that will take a long time so that was", + "segments": [ + { + "text": "uh tensor for example for 800 we can also use 65k for that that will take a long time so that was", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_048.wav": { + "chunk_id": "049", + "transcript": "[2025-06-01 20:36:11] Speaker 1: reason why we wanted to go to go with something which take less time so for example", + "segments": [ + { + "text": "reason why we wanted to go to go with something which take less time so for example", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_049.wav": { + "chunk_id": "050", + "transcript": "[2025-06-01 20:36:11] Speaker 1: the H100 with 8192 it will take around 110 seconds. A100 it will take around 492 or similar like that.", + "segments": [ + { + "text": "the H100 with 8192 it will take around 110 seconds. A100 it will take around 492 or similar like that.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_050.wav": { + "chunk_id": "051", + "transcript": "[2025-06-01 20:36:11] Speaker 1: and for 810 it will take around 649 seconds and with that we can start to talk about the functions\n[2025-06-01 20:36:22] Speaker 1: I'm not", + "segments": [ + { + "text": "and for 810 it will take around 649 seconds and with that we can start to talk about the functions", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "I'm not", + "timestamp": "2025-06-01 20:36:22", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_051.wav": { + "chunk_id": "052", + "transcript": "[2025-06-01 20:36:11] Speaker 1: going about the function again the first thing is the power the temperature and utilization", + "segments": [ + { + "text": "going about the function again the first thing is the power the temperature and utilization", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_052.wav": { + "chunk_id": "053", + "transcript": "[2025-06-01 20:36:11] Speaker 1: So we found out what is the range of that and if the temperature, the power...", + "segments": [ + { + "text": "So we found out what is the range of that and if the temperature, the power...", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_053.wav": { + "chunk_id": "054", + "transcript": "[2025-06-01 20:36:11] Speaker 1: visualization is in in that range you know we'll say that it has passed if not uh you know we'll say", + "segments": [ + { + "text": "visualization is in in that range you know we'll say that it has passed if not uh you know we'll say", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_054.wav": { + "chunk_id": "055", + "transcript": "[2025-06-01 20:36:11] Speaker 1: that you know uh it has uh you know fade um these are some of the information about the gpus uh what", + "segments": [ + { + "text": "that you know uh it has uh you know fade um these are some of the information about the gpus uh what", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_055.wav": { + "chunk_id": "056", + "transcript": "[2025-06-01 20:36:11] Speaker 1: shape is that and this type of things till we get to the background computation and compute on gpu", + "segments": [ + { + "text": "shape is that and this type of things till we get to the background computation and compute on gpu", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_056.wav": { + "chunk_id": "057", + "transcript": "[2025-06-01 20:36:11] Speaker 1: background computation is that it's a function that it will compute uh you know a simple math mall", + "segments": [ + { + "text": "background computation is that it's a function that it will compute uh you know a simple math mall", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_057.wav": { + "chunk_id": "058", + "transcript": "[2025-06-01 20:36:11] Speaker 1: back end so that you know we can measure the for temperature for uh you know for uh utilization", + "segments": [ + { + "text": "back end so that you know we can measure the for temperature for uh you know for uh utilization", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_058.wav": { + "chunk_id": "059", + "transcript": "[2025-06-01 20:36:11] Speaker 1: for the power one thing that i want to mention here is that matmon why we are using in most of these", + "segments": [ + { + "text": "for the power one thing that i want to mention here is that matmon why we are using in most of these", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_059.wav": { + "chunk_id": "060", + "transcript": "[2025-06-01 20:36:11] Speaker 1: we are using MathMol. Why we are using MathMol here? MathMol is a good simulation of fine-tuning", + "segments": [ + { + "text": "we are using MathMol. Why we are using MathMol here? MathMol is a good simulation of fine-tuning", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_060.wav": { + "chunk_id": "061", + "transcript": "[2025-06-01 20:36:11] Speaker 1: training llms and these type of things and it's very common for that so that's the reason why we", + "segments": [ + { + "text": "training llms and these type of things and it's very common for that so that's the reason why we", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_061.wav": { + "chunk_id": "062", + "transcript": "[2025-06-01 20:36:11] Speaker 1: I really focus on MatMod as one of the functions that we have used here a lot.", + "segments": [ + { + "text": "I really focus on MatMod as one of the functions that we have used here a lot.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_062.wav": { + "chunk_id": "063", + "transcript": "[2025-06-01 20:36:11] Speaker 1: i can uh the next function is about compute and gpu or compute throughput as i mentioned", + "segments": [ + { + "text": "i can uh the next function is about compute and gpu or compute throughput as i mentioned", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_063.wav": { + "chunk_id": "064", + "transcript": "[2025-06-01 20:36:11] Speaker 1: previously and this will measure the performance of the GPU cores under stress.\n[2025-06-01 20:36:22] Speaker 1: Mainly it will", + "segments": [ + { + "text": "previously and this will measure the performance of the GPU cores under stress.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "Mainly it will", + "timestamp": "2025-06-01 20:36:22", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_064.wav": { + "chunk_id": "065", + "transcript": "[2025-06-01 20:36:11] Speaker 1: say that how the gpus can handle the computation and again you can see here that you know this is\n[2025-06-01 20:36:21] Speaker 1: also", + "segments": [ + { + "text": "say that how the gpus can handle the computation and again you can see here that you know this is", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "also", + "timestamp": "2025-06-01 20:36:21", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_065.wav": { + "chunk_id": "066", + "transcript": "[2025-06-01 20:36:11] Speaker 1: again based on matmul function the next one is the memory bandwidth the memory bandwidth is mainly", + "segments": [ + { + "text": "again based on matmul function the next one is the memory bandwidth the memory bandwidth is mainly", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_066.wav": { + "chunk_id": "067", + "transcript": "[2025-06-01 20:36:11] Speaker 1: um how the gpu can handle the heavy memory uh you know bound it means that you know how we can move", + "segments": [ + { + "text": "um how the gpu can handle the heavy memory uh you know bound it means that you know how we can move", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_067.wav": { + "chunk_id": "068", + "transcript": "[2025-06-01 20:36:11] Speaker 1: the volume of data between the global memory and the compute you need. So here we are using", + "segments": [ + { + "text": "the volume of data between the global memory and the compute you need. So here we are using", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_068.wav": { + "chunk_id": "069", + "transcript": "[2025-06-01 20:36:11] Speaker 1: uh addition b equal to a plus a\n[2025-06-01 20:36:17] Speaker 1: uh and yeah the next one is about error detection on gpus so uh here", + "segments": [ + { + "text": "uh addition b equal to a plus a", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "uh and yeah the next one is about error detection on gpus so uh here", + "timestamp": "2025-06-01 20:36:17", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_069.wav": { + "chunk_id": "070", + "transcript": "[2025-06-01 20:36:11] Speaker 1: this is mainly based on the accuracy it means that one time we are using CPU for\n[2025-06-01 20:36:19] Speaker 1: you know", + "segments": [ + { + "text": "this is mainly based on the accuracy it means that one time we are using CPU for", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "you know", + "timestamp": "2025-06-01 20:36:19", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_070.wav": { + "chunk_id": "071", + "transcript": "[2025-06-01 20:36:11] Speaker 1: matmol one time we are using gpus for matmol and you know we compare these numbers and we have some", + "segments": [ + { + "text": "matmol one time we are using gpus for matmol and you know we compare these numbers and we have some", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_071.wav": { + "chunk_id": "072", + "transcript": "[2025-06-01 20:36:11] Speaker 1: threshold for that as you can see here and if that's it's uh not significant we can say that", + "segments": [ + { + "text": "threshold for that as you can see here and if that's it's uh not significant we can say that", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_072.wav": { + "chunk_id": "073", + "transcript": "[2025-06-01 20:36:11] Speaker 1: the error detection is fine. Next thing is the tensor core utilization. This is the core", + "segments": [ + { + "text": "the error detection is fine. Next thing is the tensor core utilization. This is the core", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_073.wav": { + "chunk_id": "074", + "transcript": "[2025-06-01 20:36:11] Speaker 1: engagement and and it emphasizes on the low precision like the fp8 or fp16 um this is a mainly", + "segments": [ + { + "text": "engagement and and it emphasizes on the low precision like the fp8 or fp16 um this is a mainly", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_074.wav": { + "chunk_id": "075", + "transcript": "[2025-06-01 20:36:11] Speaker 1: hardware not on the software. After the Tensor Core utilization we did sustained workload and this", + "segments": [ + { + "text": "hardware not on the software. After the Tensor Core utilization we did sustained workload and this", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_075.wav": { + "chunk_id": "076", + "transcript": "[2025-06-01 20:36:11] Speaker 1: test the long-term compute and stability. So for example here as an example we put 60 seconds.", + "segments": [ + { + "text": "test the long-term compute and stability. So for example here as an example we put 60 seconds.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_076.wav": { + "chunk_id": "077", + "transcript": "[2025-06-01 20:36:11] Speaker 1: and we are running matmod for 60 seconds and we will see that how the gpus are working based on that", + "segments": [ + { + "text": "and we are running matmod for 60 seconds and we will see that how the gpus are working based on that", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_077.wav": { + "chunk_id": "078", + "transcript": "[2025-06-01 20:36:11] Speaker 1: and the last one is mixed precision testing and this will validate both the performance and the", + "segments": [ + { + "text": "and the last one is mixed precision testing and this will validate both the performance and the", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_078.wav": { + "chunk_id": "079", + "transcript": "[2025-06-01 20:36:11] Speaker 1: correctness of the GPUs again when we are using matmult here.\n[2025-06-01 20:36:19] Speaker 1: So with that, you know, we got to the", + "segments": [ + { + "text": "correctness of the GPUs again when we are using matmult here.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "So with that, you know, we got to the", + "timestamp": "2025-06-01 20:36:19", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_079.wav": { + "chunk_id": "080", + "transcript": "[2025-06-01 20:36:11] Speaker 1: end of this part. I will give you an example of how this has worked previously.", + "segments": [ + { + "text": "end of this part. I will give you an example of how this has worked previously.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_080.wav": { + "chunk_id": "081", + "transcript": "[2025-06-01 20:36:11] Speaker 1: So we had an H100 and you can see the result of the H100 here", + "segments": [ + { + "text": "So we had an H100 and you can see the result of the H100 here", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_081.wav": { + "chunk_id": "082", + "transcript": "[2025-06-01 20:36:11] Speaker 1: And, um...", + "segments": [ + { + "text": "And, um...", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_082.wav": { + "chunk_id": "083", + "transcript": "[2025-06-01 20:36:11] Speaker 1: We saw that the H100 is not performing very well.\n[2025-06-01 20:36:15] Speaker 1: I went and saw", + "segments": [ + { + "text": "We saw that the H100 is not performing very well.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "I went and saw", + "timestamp": "2025-06-01 20:36:15", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_083.wav": { + "chunk_id": "084", + "transcript": "[2025-06-01 20:36:11] Speaker 1: specifically for gpu 4 and you can see that you know in the power the gpu 4 is uh going up very", + "segments": [ + { + "text": "specifically for gpu 4 and you can see that you know in the power the gpu 4 is uh going up very", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_084.wav": { + "chunk_id": "085", + "transcript": "[2025-06-01 20:36:11] Speaker 1: you know fast compared to other ones but you know after that the rest of that went sharply up but this", + "segments": [ + { + "text": "you know fast compared to other ones but you know after that the rest of that went sharply up but this", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_085.wav": { + "chunk_id": "086", + "transcript": "[2025-06-01 20:36:11] Speaker 1: one didn't go up uh so maybe that was the reason why the h100 was not performing very well then i\n[2025-06-01 20:36:18] Speaker 1: also", + "segments": [ + { + "text": "one didn't go up uh so maybe that was the reason why the h100 was not performing very well then i", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "also", + "timestamp": "2025-06-01 20:36:18", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_086.wav": { + "chunk_id": "087", + "transcript": "[2025-06-01 20:36:11] Speaker 1: saw the um the temperature you can see that for gpu for the temperature again is sharply going up like\n[2025-06-01 20:36:19] Speaker 1: 80.", + "segments": [ + { + "text": "saw the um the temperature you can see that for gpu for the temperature again is sharply going up like", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "80.", + "timestamp": "2025-06-01 20:36:19", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_087.wav": { + "chunk_id": "088", + "transcript": "[2025-06-01 20:36:11] Speaker 1: which is above the range that you know we specified and then you know it stays there so", + "segments": [ + { + "text": "which is above the range that you know we specified and then you know it stays there so", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_088.wav": { + "chunk_id": "089", + "transcript": "[2025-06-01 20:36:11] Speaker 1: maybe that was the reason why you know it wasn't able to uh you know do power good and the rest of", + "segments": [ + { + "text": "maybe that was the reason why you know it wasn't able to uh you know do power good and the rest of", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_089.wav": { + "chunk_id": "090", + "transcript": "[2025-06-01 20:36:11] Speaker 1: that you know you can see that you know it's not even close to 86 so we found out that you know the\n[2025-06-01 20:36:16] Speaker 1: gpu 4", + "segments": [ + { + "text": "that you know you can see that you know it's not even close to 86 so we found out that you know the", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "gpu 4", + "timestamp": "2025-06-01 20:36:16", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_090.wav": { + "chunk_id": "091", + "transcript": "[2025-06-01 20:36:11] Speaker 1: is not working very well but one thing that you know here i want to emphasize is that this is inside", + "segments": [ + { + "text": "is not working very well but one thing that you know here i want to emphasize is that this is inside", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_091.wav": { + "chunk_id": "092", + "transcript": "[2025-06-01 20:36:11] Speaker 1: a node so what does it mean it means that you know for example in the h100 and knows we have eight gpu", + "segments": [ + { + "text": "a node so what does it mean it means that you know for example in the h100 and knows we have eight gpu", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_092.wav": { + "chunk_id": "093", + "transcript": "[2025-06-01 20:36:11] Speaker 1: so it will give us the result for the hgpus but we have also extended this uh code for", + "segments": [ + { + "text": "so it will give us the result for the hgpus but we have also extended this uh code for", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_093.wav": { + "chunk_id": "094", + "transcript": "[2025-06-01 20:36:11] Speaker 1: multi-node and by that so this was this code was given to us by another team from John Shelley's", + "segments": [ + { + "text": "multi-node and by that so this was this code was given to us by another team from John Shelley's", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_094.wav": { + "chunk_id": "095", + "transcript": "[2025-06-01 20:36:11] Speaker 1: theme and it's mainly doing some more tests about nodes between the nodes specifically it", + "segments": [ + { + "text": "theme and it's mainly doing some more tests about nodes between the nodes specifically it", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_095.wav": { + "chunk_id": "096", + "transcript": "[2025-06-01 20:36:11] Speaker 1: It will evaluate the RDMA, the different type of errors like error correctness code.\n[2025-06-01 20:36:17] Speaker 1: It will also...", + "segments": [ + { + "text": "It will evaluate the RDMA, the different type of errors like error correctness code.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "It will also...", + "timestamp": "2025-06-01 20:36:17", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_096.wav": { + "chunk_id": "097", + "transcript": "[2025-06-01 20:36:11] Speaker 1: evaluate the PCIe link and this type of things.\n[2025-06-01 20:36:17] Speaker 1: The way that our structure is working right now...", + "segments": [ + { + "text": "evaluate the PCIe link and this type of things.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "The way that our structure is working right now...", + "timestamp": "2025-06-01 20:36:17", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_097.wav": { + "chunk_id": "098", + "transcript": "[2025-06-01 20:36:11] Speaker 1: is that first of all we run if you have a multi node we run this one", + "segments": [ + { + "text": "is that first of all we run if you have a multi node we run this one", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_098.wav": { + "chunk_id": "099", + "transcript": "[2025-06-01 20:36:11] Speaker 1: uh let me show you the result or the log maybe that would help", + "segments": [ + { + "text": "uh let me show you the result or the log maybe that would help", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_099.wav": { + "chunk_id": "100", + "transcript": "[2025-06-01 20:36:11] Speaker 1: First of all, we'll run that one. For example here I used two nodes and you can see", + "segments": [ + { + "text": "First of all, we'll run that one. For example here I used two nodes and you can see", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_100.wav": { + "chunk_id": "101", + "transcript": "[2025-06-01 20:36:11] Speaker 1: that you know it will go and run all of these ones one by one and it will get to this end.", + "segments": [ + { + "text": "that you know it will go and run all of these ones one by one and it will get to this end.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_101.wav": { + "chunk_id": "102", + "transcript": "[2025-06-01 20:36:11] Speaker 1: Once this is done, it will go inside each node.\n[2025-06-01 20:36:15] Speaker 1: For example, here we have the GPU 471.", + "segments": [ + { + "text": "Once this is done, it will go inside each node.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "For example, here we have the GPU 471.", + "timestamp": "2025-06-01 20:36:15", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_102.wav": { + "chunk_id": "103", + "transcript": "[2025-06-01 20:36:11] Speaker 1: here everything will be saved as a json file and this multi-node result will be saved as a json", + "segments": [ + { + "text": "here everything will be saved as a json file and this multi-node result will be saved as a json", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_103.wav": { + "chunk_id": "104", + "transcript": "[2025-06-01 20:36:11] Speaker 1: file and you can see that you know it's here", + "segments": [ + { + "text": "file and you can see that you know it's here", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_104.wav": { + "chunk_id": "105", + "transcript": "[2025-06-01 20:36:11] Speaker 1: Again, for the other one, I think it was another GPU, which was probably the 509.", + "segments": [ + { + "text": "Again, for the other one, I think it was another GPU, which was probably the 509.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_105.wav": { + "chunk_id": "106", + "transcript": "[2025-06-01 20:36:11] Speaker 1: and it will again all of the json file will uh you know save all of these results then it will go", + "segments": [ + { + "text": "and it will again all of the json file will uh you know save all of these results then it will go", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_106.wav": { + "chunk_id": "107", + "transcript": "[2025-06-01 20:36:11] Speaker 1: inside the gpu 471 and inside that gpu it will run the health check that you know and we just we were", + "segments": [ + { + "text": "inside the gpu 471 and inside that gpu it will run the health check that you know and we just we were", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_107.wav": { + "chunk_id": "108", + "transcript": "[2025-06-01 20:36:11] Speaker 1: just talking about.", + "segments": [ + { + "text": "just talking about.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_108.wav": { + "chunk_id": "109", + "transcript": "[2025-06-01 20:36:11] Speaker 1: it will go it will do that for gpu 471 again it will do that for the other gpu gpu 507 which have", + "segments": [ + { + "text": "it will go it will do that for gpu 471 again it will do that for the other gpu gpu 507 which have", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_109.wav": { + "chunk_id": "110", + "transcript": "[2025-06-01 20:36:11] Speaker 1: which we have here, then it will give us the result and the testing result.", + "segments": [ + { + "text": "which we have here, then it will give us the result and the testing result.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_110.wav": { + "chunk_id": "111", + "transcript": "[2025-06-01 20:36:11] Speaker 1: so you will have um you know a pdf and a json file and a log for each of the gpu 431 and 507 here", + "segments": [ + { + "text": "so you will have um you know a pdf and a json file and a log for each of the gpu 431 and 507 here", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_111.wav": { + "chunk_id": "112", + "transcript": "[2025-06-01 20:36:11] Speaker 1: These PDFs will be saved based on the timing and calendar.", + "segments": [ + { + "text": "These PDFs will be saved based on the timing and calendar.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_112.wav": { + "chunk_id": "113", + "transcript": "[2025-06-01 20:36:11] Speaker 1: uh with that uh this was the main idea about the health check and you know how we are running that", + "segments": [ + { + "text": "uh with that uh this was the main idea about the health check and you know how we are running that", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_113.wav": { + "chunk_id": "114", + "transcript": "[2025-06-01 20:36:11] Speaker 1: And I really appreciate your time.\n[2025-06-01 20:36:14] Speaker 1: And I would be happy if you can, you know, reach out.", + "segments": [ + { + "text": "And I really appreciate your time.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + }, + { + "text": "And I would be happy if you can, you know, reach out.", + "timestamp": "2025-06-01 20:36:14", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_114.wav": { + "chunk_id": "115", + "transcript": "[2025-06-01 20:36:11] Speaker 1: any question about this. Thank you so much.", + "segments": [ + { + "text": "any question about this. Thank you so much.", + "timestamp": "2025-06-01 20:36:11", + "speaker": "Speaker 1" + } + ], + "language": "en" + } + }, + "transcript": "[2025-06-01 20:36:11] Speaker 1: hello everyone thank you for joining to this session this session we will talk about health\n[2025-06-01 20:36:11] Speaker 1: recipe that we provided in seeds group Ammar, Soumya and me etc. we are working on this.\n[2025-06-01 20:36:11] Speaker 1: a resume and with that we can start to talk about that so the main idea of the health check\n[2025-06-01 20:36:11] Speaker 1: recipe is to make the customer confidence that they have enough resources to run heavy workloads\n[2025-06-01 20:36:11] Speaker 1: what i mean by that is that most of the customers when they want to run the llm's training find\n[2025-06-01 20:36:29] Speaker 1: you\n[2025-06-01 20:36:11] Speaker 1: tuning or any other things they need which assumed to be a heavy workload they need to have a lot of\n[2025-06-01 20:36:11] Speaker 1: gpus and we want to make sure that these gpus work well so they can run this recipe before starting\n[2025-06-01 20:36:11] Speaker 1: this task and to just make sure that you know everything is working very well i'll go ahead\n[2025-06-01 20:36:11] Speaker 1: share my screen and yeah i will start with the result of this recipe i can say that this you know\n[2025-06-01 20:36:11] Speaker 1: recipe the health check recipe will provide you two different formats one is the pdf which i'm showing\n[2025-06-01 20:36:11] Speaker 1: to you and the other one is json one i will also go through that as well but here you can see uh what\n[2025-06-01 20:36:11] Speaker 1: test we are evaluating for the gpus uh we are doing bunch of uh testing like the background\n[2025-06-01 20:36:11] Speaker 1: compare computation compute throughput memory bandwidth error detection tensor core utilization\n[2025-06-01 20:36:17] Speaker 1: sustain workload\n[2025-06-01 20:36:11] Speaker 1: mixed precision, power, temperature and utilization. So these are the main tests that we are\n[2025-06-01 20:36:11] Speaker 1: evaluating you can see that in this format once you know you run the health\n[2025-06-01 20:36:15] Speaker 1: show\n[2025-06-01 20:36:11] Speaker 1: recipe you will get the result you know everything is in a table the duration of each\n[2025-06-01 20:36:11] Speaker 1: test, whether it has failed or not.\n[2025-06-01 20:36:18] Speaker 1: For example, here for compute throughput,\n[2025-06-01 20:36:11] Speaker 1: has failed and i will show you uh in detail that how it works and for those customers who you know\n[2025-06-01 20:36:19] Speaker 1: maybe\n[2025-06-01 20:36:11] Speaker 1: don't know what are these numbers you know they can go to the end of that we\n[2025-06-01 20:36:17] Speaker 1: have also provided a graph\n[2025-06-01 20:36:11] Speaker 1: I'm waiting for that. We have also provided a graph. In this graph, we have provided, you know,\n[2025-06-01 20:36:11] Speaker 1: So first what we did was that we did an extensive research from NVIDIA website articles to see what are the...\n[2025-06-01 20:36:11] Speaker 1: uh you know ideal uh range for for example temperature the compute throughput memory bandwidth\n[2025-06-01 20:36:17] Speaker 1: and this type of\n[2025-06-01 20:36:11] Speaker 1: things. What is the upper threshold? What is the lower threshold? We added a 10% to each of\n[2025-06-01 20:36:11] Speaker 1: this once and uh we got the average of that these average are shown here as a baseline for example\n[2025-06-01 20:36:18] Speaker 1: 3000 is\n[2025-06-01 20:36:11] Speaker 1: shown here as a baseline and the measured one is the for example uh the yellow one it shows that uh\n[2025-06-01 20:36:19] Speaker 1: you know\n[2025-06-01 20:36:11] Speaker 1: So once we are using the test compute throughput, the measure was around 651.92, which is around 630.\n[2025-06-01 20:36:11] Speaker 1: uh 30 which was the baseline so the main purpose of this graph is to make the customer sure that\n[2025-06-01 20:36:11] Speaker 1: numbers that they are getting are in the same kind of range of you know what the\n[2025-06-01 20:36:18] Speaker 1: number should be\n[2025-06-01 20:36:11] Speaker 1: With that, I can also go to the JSON file.\n[2025-06-01 20:36:11] Speaker 1: which we have here. I will just open one of them.\n[2025-06-01 20:36:11] Speaker 1: So the way that they know the JSON file.\n[2025-06-01 20:36:11] Speaker 1: is working is that you know basically the same numbers are both in the PDF and the JSON file.\n[2025-06-01 20:36:11] Speaker 1: The good thing about JSON file here is that, you know, for example, for compute throughput,\n[2025-06-01 20:36:11] Speaker 1: it has failed if it go on each gpu one by one um based on the tensor and you know duration\n[2025-06-01 20:36:11] Speaker 1: flops bandwidth and it will uh make the comparison with the baseline and if uh for all of them it's not\n[2025-06-01 20:36:11] Speaker 1: in the range that you know we specify it will give us the fail. Similarly for the memory band\n[2025-06-01 20:36:11] Speaker 1: it you know it has some for each gpu it will say that you know what does what are the information\n[2025-06-01 20:36:11] Speaker 1: at the end it will give whether it is ideal or not so this is the way that uh you know the\n[2025-06-01 20:36:11] Speaker 1: JSON file is working. With that, I can go inside the code and I can talk a little bit more about\n[2025-06-01 20:36:11] Speaker 1: the code what we have provided for the customer so we are we have enabled the customer to choose\n[2025-06-01 20:36:11] Speaker 1: what type, what D-type, you know, they can choose.\n[2025-06-01 20:36:14] Speaker 1: They can go with FV32, FV16, which is the most common one.\n[2025-06-01 20:36:11] Speaker 1: uh fp8 uh we have also enabled them that whether they want to just choose one um uh you know one of\n[2025-06-01 20:36:11] Speaker 1: these uh uh functions or you know all of them the default is on all of them also they can choose\n[2025-06-01 20:36:11] Speaker 1: to do the direction so we have provided a couple of uh you know uh parser uh so that you know\n[2025-06-01 20:36:11] Speaker 1: can choose uh uh you know uh to play with them uh code one important thing is the tensor shape\n[2025-06-01 20:36:11] Speaker 1: example for a10 the tensor shape is 8192 this is the default that we have provided for all of the\n[2025-06-01 20:36:20] Speaker 1: a10\n[2025-06-01 20:36:11] Speaker 1: a hundred and h hundred um with this i will give you some information that we can we also measure\n[2025-06-01 20:36:20] Speaker 1: the highest\n[2025-06-01 20:36:11] Speaker 1: uh tensor for example for 800 we can also use 65k for that that will take a long time so that was\n[2025-06-01 20:36:11] Speaker 1: reason why we wanted to go to go with something which take less time so for example\n[2025-06-01 20:36:11] Speaker 1: the H100 with 8192 it will take around 110 seconds. A100 it will take around 492 or similar like that.\n[2025-06-01 20:36:11] Speaker 1: and for 810 it will take around 649 seconds and with that we can start to talk about the functions\n[2025-06-01 20:36:22] Speaker 1: I'm not\n[2025-06-01 20:36:11] Speaker 1: going about the function again the first thing is the power the temperature and utilization\n[2025-06-01 20:36:11] Speaker 1: So we found out what is the range of that and if the temperature, the power...\n[2025-06-01 20:36:11] Speaker 1: visualization is in in that range you know we'll say that it has passed if not uh you know we'll say\n[2025-06-01 20:36:11] Speaker 1: that you know uh it has uh you know fade um these are some of the information about the gpus uh what\n[2025-06-01 20:36:11] Speaker 1: shape is that and this type of things till we get to the background computation and compute on gpu\n[2025-06-01 20:36:11] Speaker 1: background computation is that it's a function that it will compute uh you know a simple math mall\n[2025-06-01 20:36:11] Speaker 1: back end so that you know we can measure the for temperature for uh you know for uh utilization\n[2025-06-01 20:36:11] Speaker 1: for the power one thing that i want to mention here is that matmon why we are using in most of these\n[2025-06-01 20:36:11] Speaker 1: we are using MathMol. Why we are using MathMol here? MathMol is a good simulation of fine-tuning\n[2025-06-01 20:36:11] Speaker 1: training llms and these type of things and it's very common for that so that's the reason why we\n[2025-06-01 20:36:11] Speaker 1: I really focus on MatMod as one of the functions that we have used here a lot.\n[2025-06-01 20:36:11] Speaker 1: i can uh the next function is about compute and gpu or compute throughput as i mentioned\n[2025-06-01 20:36:11] Speaker 1: previously and this will measure the performance of the GPU cores under stress.\n[2025-06-01 20:36:22] Speaker 1: Mainly it will\n[2025-06-01 20:36:11] Speaker 1: say that how the gpus can handle the computation and again you can see here that you know this is\n[2025-06-01 20:36:21] Speaker 1: also\n[2025-06-01 20:36:11] Speaker 1: again based on matmul function the next one is the memory bandwidth the memory bandwidth is mainly\n[2025-06-01 20:36:11] Speaker 1: um how the gpu can handle the heavy memory uh you know bound it means that you know how we can move\n[2025-06-01 20:36:11] Speaker 1: the volume of data between the global memory and the compute you need. So here we are using\n[2025-06-01 20:36:11] Speaker 1: uh addition b equal to a plus a\n[2025-06-01 20:36:17] Speaker 1: uh and yeah the next one is about error detection on gpus so uh here\n[2025-06-01 20:36:11] Speaker 1: this is mainly based on the accuracy it means that one time we are using CPU for\n[2025-06-01 20:36:19] Speaker 1: you know\n[2025-06-01 20:36:11] Speaker 1: matmol one time we are using gpus for matmol and you know we compare these numbers and we have some\n[2025-06-01 20:36:11] Speaker 1: threshold for that as you can see here and if that's it's uh not significant we can say that\n[2025-06-01 20:36:11] Speaker 1: the error detection is fine. Next thing is the tensor core utilization. This is the core\n[2025-06-01 20:36:11] Speaker 1: engagement and and it emphasizes on the low precision like the fp8 or fp16 um this is a mainly\n[2025-06-01 20:36:11] Speaker 1: hardware not on the software. After the Tensor Core utilization we did sustained workload and this\n[2025-06-01 20:36:11] Speaker 1: test the long-term compute and stability. So for example here as an example we put 60 seconds.\n[2025-06-01 20:36:11] Speaker 1: and we are running matmod for 60 seconds and we will see that how the gpus are working based on that\n[2025-06-01 20:36:11] Speaker 1: and the last one is mixed precision testing and this will validate both the performance and the\n[2025-06-01 20:36:11] Speaker 1: correctness of the GPUs again when we are using matmult here.\n[2025-06-01 20:36:19] Speaker 1: So with that, you know, we got to the\n[2025-06-01 20:36:11] Speaker 1: end of this part. I will give you an example of how this has worked previously.\n[2025-06-01 20:36:11] Speaker 1: So we had an H100 and you can see the result of the H100 here\n[2025-06-01 20:36:11] Speaker 1: And, um...\n[2025-06-01 20:36:11] Speaker 1: We saw that the H100 is not performing very well.\n[2025-06-01 20:36:15] Speaker 1: I went and saw\n[2025-06-01 20:36:11] Speaker 1: specifically for gpu 4 and you can see that you know in the power the gpu 4 is uh going up very\n[2025-06-01 20:36:11] Speaker 1: you know fast compared to other ones but you know after that the rest of that went sharply up but this\n[2025-06-01 20:36:11] Speaker 1: one didn't go up uh so maybe that was the reason why the h100 was not performing very well then i\n[2025-06-01 20:36:18] Speaker 1: also\n[2025-06-01 20:36:11] Speaker 1: saw the um the temperature you can see that for gpu for the temperature again is sharply going up like\n[2025-06-01 20:36:19] Speaker 1: 80.\n[2025-06-01 20:36:11] Speaker 1: which is above the range that you know we specified and then you know it stays there so\n[2025-06-01 20:36:11] Speaker 1: maybe that was the reason why you know it wasn't able to uh you know do power good and the rest of\n[2025-06-01 20:36:11] Speaker 1: that you know you can see that you know it's not even close to 86 so we found out that you know the\n[2025-06-01 20:36:16] Speaker 1: gpu 4\n[2025-06-01 20:36:11] Speaker 1: is not working very well but one thing that you know here i want to emphasize is that this is inside\n[2025-06-01 20:36:11] Speaker 1: a node so what does it mean it means that you know for example in the h100 and knows we have eight gpu\n[2025-06-01 20:36:11] Speaker 1: so it will give us the result for the hgpus but we have also extended this uh code for\n[2025-06-01 20:36:11] Speaker 1: multi-node and by that so this was this code was given to us by another team from John Shelley's\n[2025-06-01 20:36:11] Speaker 1: theme and it's mainly doing some more tests about nodes between the nodes specifically it\n[2025-06-01 20:36:11] Speaker 1: It will evaluate the RDMA, the different type of errors like error correctness code.\n[2025-06-01 20:36:17] Speaker 1: It will also...\n[2025-06-01 20:36:11] Speaker 1: evaluate the PCIe link and this type of things.\n[2025-06-01 20:36:17] Speaker 1: The way that our structure is working right now...\n[2025-06-01 20:36:11] Speaker 1: is that first of all we run if you have a multi node we run this one\n[2025-06-01 20:36:11] Speaker 1: uh let me show you the result or the log maybe that would help\n[2025-06-01 20:36:11] Speaker 1: First of all, we'll run that one. For example here I used two nodes and you can see\n[2025-06-01 20:36:11] Speaker 1: that you know it will go and run all of these ones one by one and it will get to this end.\n[2025-06-01 20:36:11] Speaker 1: Once this is done, it will go inside each node.\n[2025-06-01 20:36:15] Speaker 1: For example, here we have the GPU 471.\n[2025-06-01 20:36:11] Speaker 1: here everything will be saved as a json file and this multi-node result will be saved as a json\n[2025-06-01 20:36:11] Speaker 1: file and you can see that you know it's here\n[2025-06-01 20:36:11] Speaker 1: Again, for the other one, I think it was another GPU, which was probably the 509.\n[2025-06-01 20:36:11] Speaker 1: and it will again all of the json file will uh you know save all of these results then it will go\n[2025-06-01 20:36:11] Speaker 1: inside the gpu 471 and inside that gpu it will run the health check that you know and we just we were\n[2025-06-01 20:36:11] Speaker 1: just talking about.\n[2025-06-01 20:36:11] Speaker 1: it will go it will do that for gpu 471 again it will do that for the other gpu gpu 507 which have\n[2025-06-01 20:36:11] Speaker 1: which we have here, then it will give us the result and the testing result.\n[2025-06-01 20:36:11] Speaker 1: so you will have um you know a pdf and a json file and a log for each of the gpu 431 and 507 here\n[2025-06-01 20:36:11] Speaker 1: These PDFs will be saved based on the timing and calendar.\n[2025-06-01 20:36:11] Speaker 1: uh with that uh this was the main idea about the health check and you know how we are running that\n[2025-06-01 20:36:11] Speaker 1: And I really appreciate your time.\n[2025-06-01 20:36:14] Speaker 1: And I would be happy if you can, you know, reach out.\n[2025-06-01 20:36:11] Speaker 1: any question about this. Thank you so much.", + "summary": "Key Points:\n\n* The meeting discussed a health check recipe for running heavy workloads on GPUs.\n* The recipe aims to provide customers with the confidence that they have enough resources to run workloads.\n* The recipe includes tests for compute throughput, memory bandwidth, error detection, tensor core utilization, sustain workload, mixed precision, power, temperature, and utilization.\n* The results of the tests are provided in a table format, with the duration of each test and whether it has failed or not.\n* A graph is also provided to make customers sure that the numbers they are getting are in the same kind of range as what the number should be.\n* The JSON file is used to compare the results of the tests with the baseline and to identify which tests have failed.\n* The customer can choose the type, D-type, and direction for the functions they want to use.\n* The tensor shape is an important factor in determining the performance of the functions.\n* The meeting discussed the power, temperature, and utilization of the GPUs and their ranges.\n\nAction Items:\n\n* Share the screen and provide the result of the health check recipe.\n* Provide the PDF and JSON formats of the test results.\n* Explain the tests in detail for those customers who may not understand the numbers.\n* Provide a graph to make customers sure that the numbers they are getting are in the same kind of range as what the number should be.\n* Explain the JSON file and how it works.\n* Discuss the power, temperature, and utilization of the GPUs and their ranges.\n\n---\n\nKey points:\n\n* Background computation is a function that measures temperature, utilization, power, memory bandwidth, error detection, tensor core utilization, sustained workload, and mixed precision testing on GPUs.\n* MathMol is a good simulation of fine-tuning training LLMs and other types of things and is commonly used for this purpose.\n* The next function is about compute and GPU throughput, which measures the performance of GPU cores under stress.\n* Memory bandwidth is mainly about how the GPU can handle heavy memory-bound tasks.\n* Error detection is based on accuracy and compares the performance of CPU and GPU for the same task.\n* Tensor core utilization emphasizes low precision like fp8 or fp16 and is a hardware-based measure.\n* Sustained workload tests the long-term compute and stability of GPUs.\n* Mixed precision testing validates both the performance and correctness of GPUs when using matmult.\n\nAction items:\n\n* None specified in the transcript.\n\n---\n\nKey points:\n\n* PDFs will be saved based on timing and calendar.\n* This was the main idea about the health check.\n* Speaker 1 appreciated the time spent in the meeting.\n* Speaker 1 was happy if any questions about the health check could be reached out.\n\nDecisions:\n\n* None mentioned in the transcript.\n\nAction items:\n\n* Save PDFs based on timing and calendar.\n* None mentioned in the transcript." +} \ No newline at end of file diff --git a/docs/whisper_transcription/Examples/Test2/video1591686795_all_transcripts_20250601_203730.txt b/docs/whisper_transcription/Examples/Test2/video1591686795_all_transcripts_20250601_203730.txt new file mode 100644 index 0000000..83008ef --- /dev/null +++ b/docs/whisper_transcription/Examples/Test2/video1591686795_all_transcripts_20250601_203730.txt @@ -0,0 +1,200 @@ +[2025-06-01 20:36:11] Speaker 1: hello everyone thank you for joining to this session this session we will talk about health +[2025-06-01 20:36:11] Speaker 1: recipe that we provided in seeds group Ammar, Soumya and me etc. we are working on this. +[2025-06-01 20:36:11] Speaker 1: a resume and with that we can start to talk about that so the main idea of the health check +[2025-06-01 20:36:11] Speaker 1: recipe is to make the customer confidence that they have enough resources to run heavy workloads +[2025-06-01 20:36:11] Speaker 1: what i mean by that is that most of the customers when they want to run the llm's training find +[2025-06-01 20:36:29] Speaker 1: you +[2025-06-01 20:36:11] Speaker 1: tuning or any other things they need which assumed to be a heavy workload they need to have a lot of +[2025-06-01 20:36:11] Speaker 1: gpus and we want to make sure that these gpus work well so they can run this recipe before starting +[2025-06-01 20:36:11] Speaker 1: this task and to just make sure that you know everything is working very well i'll go ahead +[2025-06-01 20:36:11] Speaker 1: share my screen and yeah i will start with the result of this recipe i can say that this you know +[2025-06-01 20:36:11] Speaker 1: recipe the health check recipe will provide you two different formats one is the pdf which i'm showing +[2025-06-01 20:36:11] Speaker 1: to you and the other one is json one i will also go through that as well but here you can see uh what +[2025-06-01 20:36:11] Speaker 1: test we are evaluating for the gpus uh we are doing bunch of uh testing like the background +[2025-06-01 20:36:11] Speaker 1: compare computation compute throughput memory bandwidth error detection tensor core utilization +[2025-06-01 20:36:17] Speaker 1: sustain workload +[2025-06-01 20:36:11] Speaker 1: mixed precision, power, temperature and utilization. So these are the main tests that we are +[2025-06-01 20:36:11] Speaker 1: evaluating you can see that in this format once you know you run the health +[2025-06-01 20:36:15] Speaker 1: show +[2025-06-01 20:36:11] Speaker 1: recipe you will get the result you know everything is in a table the duration of each +[2025-06-01 20:36:11] Speaker 1: test, whether it has failed or not. +[2025-06-01 20:36:18] Speaker 1: For example, here for compute throughput, +[2025-06-01 20:36:11] Speaker 1: has failed and i will show you uh in detail that how it works and for those customers who you know +[2025-06-01 20:36:19] Speaker 1: maybe +[2025-06-01 20:36:11] Speaker 1: don't know what are these numbers you know they can go to the end of that we +[2025-06-01 20:36:17] Speaker 1: have also provided a graph +[2025-06-01 20:36:11] Speaker 1: I'm waiting for that. We have also provided a graph. In this graph, we have provided, you know, +[2025-06-01 20:36:11] Speaker 1: So first what we did was that we did an extensive research from NVIDIA website articles to see what are the... +[2025-06-01 20:36:11] Speaker 1: uh you know ideal uh range for for example temperature the compute throughput memory bandwidth +[2025-06-01 20:36:17] Speaker 1: and this type of +[2025-06-01 20:36:11] Speaker 1: things. What is the upper threshold? What is the lower threshold? We added a 10% to each of +[2025-06-01 20:36:11] Speaker 1: this once and uh we got the average of that these average are shown here as a baseline for example +[2025-06-01 20:36:18] Speaker 1: 3000 is +[2025-06-01 20:36:11] Speaker 1: shown here as a baseline and the measured one is the for example uh the yellow one it shows that uh +[2025-06-01 20:36:19] Speaker 1: you know +[2025-06-01 20:36:11] Speaker 1: So once we are using the test compute throughput, the measure was around 651.92, which is around 630. +[2025-06-01 20:36:11] Speaker 1: uh 30 which was the baseline so the main purpose of this graph is to make the customer sure that +[2025-06-01 20:36:11] Speaker 1: numbers that they are getting are in the same kind of range of you know what the +[2025-06-01 20:36:18] Speaker 1: number should be +[2025-06-01 20:36:11] Speaker 1: With that, I can also go to the JSON file. +[2025-06-01 20:36:11] Speaker 1: which we have here. I will just open one of them. +[2025-06-01 20:36:11] Speaker 1: So the way that they know the JSON file. +[2025-06-01 20:36:11] Speaker 1: is working is that you know basically the same numbers are both in the PDF and the JSON file. +[2025-06-01 20:36:11] Speaker 1: The good thing about JSON file here is that, you know, for example, for compute throughput, +[2025-06-01 20:36:11] Speaker 1: it has failed if it go on each gpu one by one um based on the tensor and you know duration +[2025-06-01 20:36:11] Speaker 1: flops bandwidth and it will uh make the comparison with the baseline and if uh for all of them it's not +[2025-06-01 20:36:11] Speaker 1: in the range that you know we specify it will give us the fail. Similarly for the memory band +[2025-06-01 20:36:11] Speaker 1: it you know it has some for each gpu it will say that you know what does what are the information +[2025-06-01 20:36:11] Speaker 1: at the end it will give whether it is ideal or not so this is the way that uh you know the +[2025-06-01 20:36:11] Speaker 1: JSON file is working. With that, I can go inside the code and I can talk a little bit more about +[2025-06-01 20:36:11] Speaker 1: the code what we have provided for the customer so we are we have enabled the customer to choose +[2025-06-01 20:36:11] Speaker 1: what type, what D-type, you know, they can choose. +[2025-06-01 20:36:14] Speaker 1: They can go with FV32, FV16, which is the most common one. +[2025-06-01 20:36:11] Speaker 1: uh fp8 uh we have also enabled them that whether they want to just choose one um uh you know one of +[2025-06-01 20:36:11] Speaker 1: these uh uh functions or you know all of them the default is on all of them also they can choose +[2025-06-01 20:36:11] Speaker 1: to do the direction so we have provided a couple of uh you know uh parser uh so that you know +[2025-06-01 20:36:11] Speaker 1: can choose uh uh you know uh to play with them uh code one important thing is the tensor shape +[2025-06-01 20:36:11] Speaker 1: example for a10 the tensor shape is 8192 this is the default that we have provided for all of the +[2025-06-01 20:36:20] Speaker 1: a10 +[2025-06-01 20:36:11] Speaker 1: a hundred and h hundred um with this i will give you some information that we can we also measure +[2025-06-01 20:36:20] Speaker 1: the highest +[2025-06-01 20:36:11] Speaker 1: uh tensor for example for 800 we can also use 65k for that that will take a long time so that was +[2025-06-01 20:36:11] Speaker 1: reason why we wanted to go to go with something which take less time so for example +[2025-06-01 20:36:11] Speaker 1: the H100 with 8192 it will take around 110 seconds. A100 it will take around 492 or similar like that. +[2025-06-01 20:36:11] Speaker 1: and for 810 it will take around 649 seconds and with that we can start to talk about the functions +[2025-06-01 20:36:22] Speaker 1: I'm not +[2025-06-01 20:36:11] Speaker 1: going about the function again the first thing is the power the temperature and utilization +[2025-06-01 20:36:11] Speaker 1: So we found out what is the range of that and if the temperature, the power... +[2025-06-01 20:36:11] Speaker 1: visualization is in in that range you know we'll say that it has passed if not uh you know we'll say +[2025-06-01 20:36:11] Speaker 1: that you know uh it has uh you know fade um these are some of the information about the gpus uh what +[2025-06-01 20:36:11] Speaker 1: shape is that and this type of things till we get to the background computation and compute on gpu +[2025-06-01 20:36:11] Speaker 1: background computation is that it's a function that it will compute uh you know a simple math mall +[2025-06-01 20:36:11] Speaker 1: back end so that you know we can measure the for temperature for uh you know for uh utilization +[2025-06-01 20:36:11] Speaker 1: for the power one thing that i want to mention here is that matmon why we are using in most of these +[2025-06-01 20:36:11] Speaker 1: we are using MathMol. Why we are using MathMol here? MathMol is a good simulation of fine-tuning +[2025-06-01 20:36:11] Speaker 1: training llms and these type of things and it's very common for that so that's the reason why we +[2025-06-01 20:36:11] Speaker 1: I really focus on MatMod as one of the functions that we have used here a lot. +[2025-06-01 20:36:11] Speaker 1: i can uh the next function is about compute and gpu or compute throughput as i mentioned +[2025-06-01 20:36:11] Speaker 1: previously and this will measure the performance of the GPU cores under stress. +[2025-06-01 20:36:22] Speaker 1: Mainly it will +[2025-06-01 20:36:11] Speaker 1: say that how the gpus can handle the computation and again you can see here that you know this is +[2025-06-01 20:36:21] Speaker 1: also +[2025-06-01 20:36:11] Speaker 1: again based on matmul function the next one is the memory bandwidth the memory bandwidth is mainly +[2025-06-01 20:36:11] Speaker 1: um how the gpu can handle the heavy memory uh you know bound it means that you know how we can move +[2025-06-01 20:36:11] Speaker 1: the volume of data between the global memory and the compute you need. So here we are using +[2025-06-01 20:36:11] Speaker 1: uh addition b equal to a plus a +[2025-06-01 20:36:17] Speaker 1: uh and yeah the next one is about error detection on gpus so uh here +[2025-06-01 20:36:11] Speaker 1: this is mainly based on the accuracy it means that one time we are using CPU for +[2025-06-01 20:36:19] Speaker 1: you know +[2025-06-01 20:36:11] Speaker 1: matmol one time we are using gpus for matmol and you know we compare these numbers and we have some +[2025-06-01 20:36:11] Speaker 1: threshold for that as you can see here and if that's it's uh not significant we can say that +[2025-06-01 20:36:11] Speaker 1: the error detection is fine. Next thing is the tensor core utilization. This is the core +[2025-06-01 20:36:11] Speaker 1: engagement and and it emphasizes on the low precision like the fp8 or fp16 um this is a mainly +[2025-06-01 20:36:11] Speaker 1: hardware not on the software. After the Tensor Core utilization we did sustained workload and this +[2025-06-01 20:36:11] Speaker 1: test the long-term compute and stability. So for example here as an example we put 60 seconds. +[2025-06-01 20:36:11] Speaker 1: and we are running matmod for 60 seconds and we will see that how the gpus are working based on that +[2025-06-01 20:36:11] Speaker 1: and the last one is mixed precision testing and this will validate both the performance and the +[2025-06-01 20:36:11] Speaker 1: correctness of the GPUs again when we are using matmult here. +[2025-06-01 20:36:19] Speaker 1: So with that, you know, we got to the +[2025-06-01 20:36:11] Speaker 1: end of this part. I will give you an example of how this has worked previously. +[2025-06-01 20:36:11] Speaker 1: So we had an H100 and you can see the result of the H100 here +[2025-06-01 20:36:11] Speaker 1: And, um... +[2025-06-01 20:36:11] Speaker 1: We saw that the H100 is not performing very well. +[2025-06-01 20:36:15] Speaker 1: I went and saw +[2025-06-01 20:36:11] Speaker 1: specifically for gpu 4 and you can see that you know in the power the gpu 4 is uh going up very +[2025-06-01 20:36:11] Speaker 1: you know fast compared to other ones but you know after that the rest of that went sharply up but this +[2025-06-01 20:36:11] Speaker 1: one didn't go up uh so maybe that was the reason why the h100 was not performing very well then i +[2025-06-01 20:36:18] Speaker 1: also +[2025-06-01 20:36:11] Speaker 1: saw the um the temperature you can see that for gpu for the temperature again is sharply going up like +[2025-06-01 20:36:19] Speaker 1: 80. +[2025-06-01 20:36:11] Speaker 1: which is above the range that you know we specified and then you know it stays there so +[2025-06-01 20:36:11] Speaker 1: maybe that was the reason why you know it wasn't able to uh you know do power good and the rest of +[2025-06-01 20:36:11] Speaker 1: that you know you can see that you know it's not even close to 86 so we found out that you know the +[2025-06-01 20:36:16] Speaker 1: gpu 4 +[2025-06-01 20:36:11] Speaker 1: is not working very well but one thing that you know here i want to emphasize is that this is inside +[2025-06-01 20:36:11] Speaker 1: a node so what does it mean it means that you know for example in the h100 and knows we have eight gpu +[2025-06-01 20:36:11] Speaker 1: so it will give us the result for the hgpus but we have also extended this uh code for +[2025-06-01 20:36:11] Speaker 1: multi-node and by that so this was this code was given to us by another team from John Shelley's +[2025-06-01 20:36:11] Speaker 1: theme and it's mainly doing some more tests about nodes between the nodes specifically it +[2025-06-01 20:36:11] Speaker 1: It will evaluate the RDMA, the different type of errors like error correctness code. +[2025-06-01 20:36:17] Speaker 1: It will also... +[2025-06-01 20:36:11] Speaker 1: evaluate the PCIe link and this type of things. +[2025-06-01 20:36:17] Speaker 1: The way that our structure is working right now... +[2025-06-01 20:36:11] Speaker 1: is that first of all we run if you have a multi node we run this one +[2025-06-01 20:36:11] Speaker 1: uh let me show you the result or the log maybe that would help +[2025-06-01 20:36:11] Speaker 1: First of all, we'll run that one. For example here I used two nodes and you can see +[2025-06-01 20:36:11] Speaker 1: that you know it will go and run all of these ones one by one and it will get to this end. +[2025-06-01 20:36:11] Speaker 1: Once this is done, it will go inside each node. +[2025-06-01 20:36:15] Speaker 1: For example, here we have the GPU 471. +[2025-06-01 20:36:11] Speaker 1: here everything will be saved as a json file and this multi-node result will be saved as a json +[2025-06-01 20:36:11] Speaker 1: file and you can see that you know it's here +[2025-06-01 20:36:11] Speaker 1: Again, for the other one, I think it was another GPU, which was probably the 509. +[2025-06-01 20:36:11] Speaker 1: and it will again all of the json file will uh you know save all of these results then it will go +[2025-06-01 20:36:11] Speaker 1: inside the gpu 471 and inside that gpu it will run the health check that you know and we just we were +[2025-06-01 20:36:11] Speaker 1: just talking about. +[2025-06-01 20:36:11] Speaker 1: it will go it will do that for gpu 471 again it will do that for the other gpu gpu 507 which have +[2025-06-01 20:36:11] Speaker 1: which we have here, then it will give us the result and the testing result. +[2025-06-01 20:36:11] Speaker 1: so you will have um you know a pdf and a json file and a log for each of the gpu 431 and 507 here +[2025-06-01 20:36:11] Speaker 1: These PDFs will be saved based on the timing and calendar. +[2025-06-01 20:36:11] Speaker 1: uh with that uh this was the main idea about the health check and you know how we are running that +[2025-06-01 20:36:11] Speaker 1: And I really appreciate your time. +[2025-06-01 20:36:14] Speaker 1: And I would be happy if you can, you know, reach out. +[2025-06-01 20:36:11] Speaker 1: any question about this. Thank you so much. + +====== Summary ====== + +Key Points: + +* The meeting discussed a health check recipe for running heavy workloads on GPUs. +* The recipe aims to provide customers with the confidence that they have enough resources to run workloads. +* The recipe includes tests for compute throughput, memory bandwidth, error detection, tensor core utilization, sustain workload, mixed precision, power, temperature, and utilization. +* The results of the tests are provided in a table format, with the duration of each test and whether it has failed or not. +* A graph is also provided to make customers sure that the numbers they are getting are in the same kind of range as what the number should be. +* The JSON file is used to compare the results of the tests with the baseline and to identify which tests have failed. +* The customer can choose the type, D-type, and direction for the functions they want to use. +* The tensor shape is an important factor in determining the performance of the functions. +* The meeting discussed the power, temperature, and utilization of the GPUs and their ranges. + +Action Items: + +* Share the screen and provide the result of the health check recipe. +* Provide the PDF and JSON formats of the test results. +* Explain the tests in detail for those customers who may not understand the numbers. +* Provide a graph to make customers sure that the numbers they are getting are in the same kind of range as what the number should be. +* Explain the JSON file and how it works. +* Discuss the power, temperature, and utilization of the GPUs and their ranges. + +--- + +Key points: + +* Background computation is a function that measures temperature, utilization, power, memory bandwidth, error detection, tensor core utilization, sustained workload, and mixed precision testing on GPUs. +* MathMol is a good simulation of fine-tuning training LLMs and other types of things and is commonly used for this purpose. +* The next function is about compute and GPU throughput, which measures the performance of GPU cores under stress. +* Memory bandwidth is mainly about how the GPU can handle heavy memory-bound tasks. +* Error detection is based on accuracy and compares the performance of CPU and GPU for the same task. +* Tensor core utilization emphasizes low precision like fp8 or fp16 and is a hardware-based measure. +* Sustained workload tests the long-term compute and stability of GPUs. +* Mixed precision testing validates both the performance and correctness of GPUs when using matmult. + +Action items: + +* None specified in the transcript. + +--- + +Key points: + +* PDFs will be saved based on timing and calendar. +* This was the main idea about the health check. +* Speaker 1 appreciated the time spent in the meeting. +* Speaker 1 was happy if any questions about the health check could be reached out. + +Decisions: + +* None mentioned in the transcript. + +Action items: + +* Save PDFs based on timing and calendar. +* None mentioned in the transcript. \ No newline at end of file diff --git a/docs/whisper_transcription/Examples/Test3/audio1788670787.m4a b/docs/whisper_transcription/Examples/Test3/audio1788670787.m4a new file mode 100644 index 0000000..f05ea90 Binary files /dev/null and b/docs/whisper_transcription/Examples/Test3/audio1788670787.m4a differ diff --git a/docs/whisper_transcription/Examples/Test3/audio1788670787_all_transcripts_20250601_191710.json b/docs/whisper_transcription/Examples/Test3/audio1788670787_all_transcripts_20250601_191710.json new file mode 100644 index 0000000..bd9d695 --- /dev/null +++ b/docs/whisper_transcription/Examples/Test3/audio1788670787_all_transcripts_20250601_191710.json @@ -0,0 +1,5568 @@ +{ + "chunks": { + "chunk_000.wav": { + "chunk_id": "001", + "transcript": "[2025-06-01 19:13:25] Speaker 5: All right. Thank you all. Morning, afternoon. So my name is Amar Gowda. I am one of the\n[2025-06-01 19:13:30] Speaker 5: products.", + "segments": [ + { + "text": "All right. Thank you all. Morning, afternoon. So my name is Amar Gowda. I am one of the", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "products.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_001.wav": { + "chunk_id": "002", + "transcript": "[2025-06-01 19:13:25] Speaker 5: managers leading the OCI Lens initiative. So right now it's kind of what we call incubation.", + "segments": [ + { + "text": "managers leading the OCI Lens initiative. So right now it's kind of what we call incubation.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_002.wav": { + "chunk_id": "003", + "transcript": "[2025-06-01 19:13:25] Speaker 5: phases so we are not yet to be a mvp or closer to it so we're kind of getting early feedback", + "segments": [ + { + "text": "phases so we are not yet to be a mvp or closer to it so we're kind of getting early feedback", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_003.wav": { + "chunk_id": "004", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and this is like think of this as a kind of private preview in a way right so a little earlier on", + "segments": [ + { + "text": "and this is like think of this as a kind of private preview in a way right so a little earlier on", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_004.wav": { + "chunk_id": "005", + "transcript": "[2025-06-01 19:13:25] Speaker 5: on the journey.\n[2025-06-01 19:13:26] Speaker 5: But this is something we started looking into,\n[2025-06-01 19:13:31] Speaker 5: which we, I think we spoke with.", + "segments": [ + { + "text": "on the journey.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "But this is something we started looking into,", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 5" + }, + { + "text": "which we, I think we spoke with.", + "timestamp": "2025-06-01 19:13:31", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_005.wav": { + "chunk_id": "006", + "transcript": "[2025-06-01 19:13:25] Speaker 5: you, Cecil, you and the rest of the team on what does it take to operate at scale?", + "segments": [ + { + "text": "you, Cecil, you and the rest of the team on what does it take to operate at scale?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_006.wav": { + "chunk_id": "007", + "transcript": "[2025-06-01 19:13:25] Speaker 5: what are the things that we could improve on, especially the monitoring, the health check.", + "segments": [ + { + "text": "what are the things that we could improve on, especially the monitoring, the health check.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_007.wav": { + "chunk_id": "008", + "transcript": "[2025-06-01 19:13:25] Speaker 5: side of the things, which has, you know, you gave a set of requirements and that led us.", + "segments": [ + { + "text": "side of the things, which has, you know, you gave a set of requirements and that led us.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_008.wav": { + "chunk_id": "009", + "transcript": "[2025-06-01 19:13:25] Speaker 5: to building something to help customers like you.\n[2025-06-01 19:13:30] Speaker 5: Okay, so I have a team here.\n[2025-06-01 19:13:32] Speaker 5: Joletta is...", + "segments": [ + { + "text": "to building something to help customers like you.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Okay, so I have a team here.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + }, + { + "text": "Joletta is...", + "timestamp": "2025-06-01 19:13:32", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_009.wav": { + "chunk_id": "010", + "transcript": "[2025-06-01 19:13:25] Speaker 5: another software engineers somia on the call is our data scientist selection ml engineer\n[2025-06-01 19:13:30] Speaker 5: uh", + "segments": [ + { + "text": "another software engineers somia on the call is our data scientist selection ml engineer", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "uh", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_010.wav": { + "chunk_id": "011", + "transcript": "[2025-06-01 19:13:25] Speaker 5: also have uh hithika who's our golang developer she's ex uh akis team so that's where if you all can", + "segments": [ + { + "text": "also have uh hithika who's our golang developer she's ex uh akis team so that's where if you all can", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_011.wav": { + "chunk_id": "012", + "transcript": "[2025-06-01 19:13:25] Speaker 5: connect a little bit too.\n[2025-06-01 19:13:27] Speaker 5: All right. Today, I'm going to keep it more focused on the demo.", + "segments": [ + { + "text": "connect a little bit too.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "All right. Today, I'm going to keep it more focused on the demo.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_012.wav": { + "chunk_id": "013", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and what this tool is all about.\n[2025-06-01 19:13:27] Speaker 5: And then we'll, you know, answer questions\n[2025-06-01 19:13:29] Speaker 5: and kind of talk through.", + "segments": [ + { + "text": "and what this tool is all about.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "And then we'll, you know, answer questions", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "and kind of talk through.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_013.wav": { + "chunk_id": "014", + "transcript": "[2025-06-01 19:13:25] Speaker 5: But most importantly, we want to hear if the current version has a set of features you're", + "segments": [ + { + "text": "But most importantly, we want to hear if the current version has a set of features you're", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_014.wav": { + "chunk_id": "015", + "transcript": "[2025-06-01 19:13:25] Speaker 5: looking for? Is it missing? Is it you want to see something more? Because we are kind of very", + "segments": [ + { + "text": "looking for? Is it missing? Is it you want to see something more? Because we are kind of very", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_015.wav": { + "chunk_id": "016", + "transcript": "[2025-06-01 19:13:25] Speaker 5: rapidly on a weekly sprint uh releasing the features to this and we are really happy to incorporate", + "segments": [ + { + "text": "rapidly on a weekly sprint uh releasing the features to this and we are really happy to incorporate", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_016.wav": { + "chunk_id": "017", + "transcript": "[2025-06-01 19:13:25] Speaker 5: anything and work on how you can onboard to this by the way\n[2025-06-01 19:13:31] Speaker 5: uh so four areas where we are focused on", + "segments": [ + { + "text": "anything and work on how you can onboard to this by the way", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "uh so four areas where we are focused on", + "timestamp": "2025-06-01 19:13:31", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_017.wav": { + "chunk_id": "018", + "transcript": "[2025-06-01 19:13:25] Speaker 5: this solution is continuous GPU and cluster level monitoring of both NVIDIA GPUs and AMD GPUs.", + "segments": [ + { + "text": "this solution is continuous GPU and cluster level monitoring of both NVIDIA GPUs and AMD GPUs.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_018.wav": { + "chunk_id": "019", + "transcript": "[2025-06-01 19:13:25] Speaker 5: together, right? So this is how we want to kind of platform\n[2025-06-01 19:13:29] Speaker 5: agnostic solution that works for all GPUs.", + "segments": [ + { + "text": "together, right? So this is how we want to kind of platform", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "agnostic solution that works for all GPUs.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_019.wav": { + "chunk_id": "020", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and has all the latest metrics available for you to digest and look at.\n[2025-06-01 19:13:32] Speaker 5: The next thing which we've highlighted...", + "segments": [ + { + "text": "and has all the latest metrics available for you to digest and look at.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "The next thing which we've highlighted...", + "timestamp": "2025-06-01 19:13:32", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_020.wav": { + "chunk_id": "021", + "transcript": "[2025-06-01 19:13:25] Speaker 5: focused on is the health checks piece to this right are my nodes healthy are they performing to\n[2025-06-01 19:13:30] Speaker 5: what they", + "segments": [ + { + "text": "focused on is the health checks piece to this right are my nodes healthy are they performing to", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "what they", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_021.wav": { + "chunk_id": "022", + "transcript": "[2025-06-01 19:13:25] Speaker 5: should be uh active health checks active monitoring is is is a huge investment that we did uh and i'll\n[2025-06-01 19:13:32] Speaker 5: go over some", + "segments": [ + { + "text": "should be uh active health checks active monitoring is is is a huge investment that we did uh and i'll", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "go over some", + "timestamp": "2025-06-01 19:13:32", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_022.wav": { + "chunk_id": "023", + "transcript": "[2025-06-01 19:13:25] Speaker 5: details about what all health checks that we have enabled including the RDMA cluster level.", + "segments": [ + { + "text": "details about what all health checks that we have enabled including the RDMA cluster level.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_023.wav": { + "chunk_id": "024", + "transcript": "[2025-06-01 19:13:25] Speaker 5: checks that we call MPI tests as part of the solution fully baked into.\n[2025-06-01 19:13:30] Speaker 5: We kept it to cloud native.", + "segments": [ + { + "text": "checks that we call MPI tests as part of the solution fully baked into.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "We kept it to cloud native.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_024.wav": { + "chunk_id": "025", + "transcript": "[2025-06-01 19:13:25] Speaker 5: because I heard it very clear that it has to be Grafana, Prometheus, Native, or Otel, anything open source.", + "segments": [ + { + "text": "because I heard it very clear that it has to be Grafana, Prometheus, Native, or Otel, anything open source.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_025.wav": { + "chunk_id": "026", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Cloud Native Alliance so that it's easy to be move between clouds or on-prem and use\n[2025-06-01 19:13:29] Speaker 5: the same existing.", + "segments": [ + { + "text": "Cloud Native Alliance so that it's easy to be move between clouds or on-prem and use", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "the same existing.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_026.wav": { + "chunk_id": "027", + "transcript": "[2025-06-01 19:13:25] Speaker 5: tools you always used for for monitoring another area that we've paid attention and we built is the", + "segments": [ + { + "text": "tools you always used for for monitoring another area that we've paid attention and we built is the", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_027.wav": { + "chunk_id": "028", + "transcript": "[2025-06-01 19:13:25] Speaker 5: team level tracking. A lot of these large size clusters are usually used by multiple teams and", + "segments": [ + { + "text": "team level tracking. A lot of these large size clusters are usually used by multiple teams and", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_028.wav": { + "chunk_id": "029", + "transcript": "[2025-06-01 19:13:25] Speaker 5: everybody wants to know the health of their own subset of systems or how their experiment is\n[2025-06-01 19:13:30] Speaker 5: performing.", + "segments": [ + { + "text": "everybody wants to know the health of their own subset of systems or how their experiment is", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "performing.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_029.wav": { + "chunk_id": "030", + "transcript": "[2025-06-01 19:13:25] Speaker 5: if they have an unhealthy node, all of those combinations.\n[2025-06-01 19:13:29] Speaker 5: And another important thing is about", + "segments": [ + { + "text": "if they have an unhealthy node, all of those combinations.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "And another important thing is about", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_030.wav": { + "chunk_id": "031", + "transcript": "[2025-06-01 19:13:25] Speaker 5: cost tracking right people want to see how much computer resources they've used uh whatever the", + "segments": [ + { + "text": "cost tracking right people want to see how much computer resources they've used uh whatever the", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_031.wav": { + "chunk_id": "032", + "transcript": "[2025-06-01 19:13:25] Speaker 5: the power the total gpu power they have used for running this experiment right those were a few\n[2025-06-01 19:13:29] Speaker 5: things that", + "segments": [ + { + "text": "the power the total gpu power they have used for running this experiment right those were a few", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "things that", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_032.wav": { + "chunk_id": "033", + "transcript": "[2025-06-01 19:13:25] Speaker 5: that we also focused to build on this.\n[2025-06-01 19:13:29] Speaker 5: The current set of features that we have\n[2025-06-01 19:13:32] Speaker 5: and what we have built right now", + "segments": [ + { + "text": "that we also focused to build on this.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "The current set of features that we have", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + }, + { + "text": "and what we have built right now", + "timestamp": "2025-06-01 19:13:32", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_033.wav": { + "chunk_id": "034", + "transcript": "[2025-06-01 19:13:25] Speaker 5: now is first thing tenancy level monitoring you no longer have region barriers right so we any", + "segments": [ + { + "text": "now is first thing tenancy level monitoring you no longer have region barriers right so we any", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_034.wav": { + "chunk_id": "035", + "transcript": "[2025-06-01 19:13:25] Speaker 5: data, any region, OCI region you may be in, everything is monitorable as a single instance\n[2025-06-01 19:13:31] Speaker 5: for us.", + "segments": [ + { + "text": "data, any region, OCI region you may be in, everything is monitorable as a single instance", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "for us.", + "timestamp": "2025-06-01 19:13:31", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_035.wav": { + "chunk_id": "036", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So I'm going to show you a demo.\n[2025-06-01 19:13:27] Speaker 5: We allow you to monitor either single, bare metal, virtual machine", + "segments": [ + { + "text": "So I'm going to show you a demo.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "We allow you to monitor either single, bare metal, virtual machine", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_036.wav": { + "chunk_id": "037", + "transcript": "[2025-06-01 19:13:25] Speaker 5: instances or a full oke cluster or an hpc cluster that if you may be using slurm or a native uh", + "segments": [ + { + "text": "instances or a full oke cluster or an hpc cluster that if you may be using slurm or a native uh", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_037.wav": { + "chunk_id": "038", + "transcript": "[2025-06-01 19:13:25] Speaker 5: called cluster network, compute cluster setup in OCI.\n[2025-06-01 19:13:29] Speaker 5: The third part of the feature that we worked on", + "segments": [ + { + "text": "called cluster network, compute cluster setup in OCI.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "The third part of the feature that we worked on", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_038.wav": { + "chunk_id": "039", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and all of this we're going to see a demo quickly but i'm going to just spend maybe five more minutes", + "segments": [ + { + "text": "and all of this we're going to see a demo quickly but i'm going to just spend maybe five more minutes", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_039.wav": { + "chunk_id": "040", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and then go to the demo.\n[2025-06-01 19:13:26] Speaker 5: Is it team level tracking?\n[2025-06-01 19:13:27] Speaker 5: You can create team level tracking.", + "segments": [ + { + "text": "and then go to the demo.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Is it team level tracking?", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 5" + }, + { + "text": "You can create team level tracking.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_040.wav": { + "chunk_id": "041", + "transcript": "[2025-06-01 19:13:25] Speaker 5: handpicking whichever nodes are part of this experiment.\n[2025-06-01 19:13:29] Speaker 5: We are also working towards...", + "segments": [ + { + "text": "handpicking whichever nodes are part of this experiment.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "We are also working towards...", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_041.wav": { + "chunk_id": "042", + "transcript": "[2025-06-01 19:13:25] Speaker 5: automatically fetching the nodes that are running experiment based on Kubernetes\n[2025-06-01 19:13:29] Speaker 5: how the job system.", + "segments": [ + { + "text": "automatically fetching the nodes that are running experiment based on Kubernetes", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "how the job system.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_042.wav": { + "chunk_id": "043", + "transcript": "[2025-06-01 19:13:25] Speaker 5: scheduled. So that's another area we're working on automatically, dynamically pulling the nodes that", + "segments": [ + { + "text": "scheduled. So that's another area we're working on automatically, dynamically pulling the nodes that", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_043.wav": { + "chunk_id": "044", + "transcript": "[2025-06-01 19:13:25] Speaker 5: part of that experiment rather than you allocating what compute nodes go to.\n[2025-06-01 19:13:31] Speaker 5: This I've already said.", + "segments": [ + { + "text": "part of that experiment rather than you allocating what compute nodes go to.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "This I've already said.", + "timestamp": "2025-06-01 19:13:31", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_044.wav": { + "chunk_id": "045", + "transcript": "[2025-06-01 19:13:25] Speaker 5: already have both of them running and i'm going to show you that uh the difference between what we do", + "segments": [ + { + "text": "already have both of them running and i'm going to show you that uh the difference between what we do", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_045.wav": { + "chunk_id": "046", + "transcript": "[2025-06-01 19:13:25] Speaker 5: with performance monitoring and health check is we go very close to the layers that an ML engineer would", + "segments": [ + { + "text": "with performance monitoring and health check is we go very close to the layers that an ML engineer would", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_046.wav": { + "chunk_id": "047", + "transcript": "[2025-06-01 19:13:25] Speaker 5: operate under, which is PyTorch for us, PyTorch and JAX primarily, right? So we picked up PyTorch base.", + "segments": [ + { + "text": "operate under, which is PyTorch for us, PyTorch and JAX primarily, right? So we picked up PyTorch base.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_047.wav": { + "chunk_id": "048", + "transcript": "[2025-06-01 19:13:25] Speaker 5: matmal and linear regression based performance testing to achieve how many flops did we achieve", + "segments": [ + { + "text": "matmal and linear regression based performance testing to achieve how many flops did we achieve", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_048.wav": { + "chunk_id": "049", + "transcript": "[2025-06-01 19:13:25] Speaker 5: on this compute node how much is each gpu performing and is it is it within the threshold of", + "segments": [ + { + "text": "on this compute node how much is each gpu performing and is it is it within the threshold of", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_049.wav": { + "chunk_id": "050", + "transcript": "[2025-06-01 19:13:25] Speaker 5: the specification from NVIDIA or AMD what we have traditionally seen. So we have a baseline.", + "segments": [ + { + "text": "the specification from NVIDIA or AMD what we have traditionally seen. So we have a baseline.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_050.wav": { + "chunk_id": "051", + "transcript": "[2025-06-01 19:13:25] Speaker 5: that we score the performance to because it's the standard set of precision based testing either FBA.", + "segments": [ + { + "text": "that we score the performance to because it's the standard set of precision based testing either FBA.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_051.wav": { + "chunk_id": "052", + "transcript": "[2025-06-01 19:13:25] Speaker 5: FP16, FP32, it automatically does all of this. And most of this code, by the way, for the health", + "segments": [ + { + "text": "FP16, FP32, it automatically does all of this. And most of this code, by the way, for the health", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_052.wav": { + "chunk_id": "053", + "transcript": "[2025-06-01 19:13:25] Speaker 5: will all be open source so you can see you can change the things if you like or add to it", + "segments": [ + { + "text": "will all be open source so you can see you can change the things if you like or add to it", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_053.wav": { + "chunk_id": "054", + "transcript": "[2025-06-01 19:13:25] Speaker 5: right and please feel free to contribute to it too.\n[2025-06-01 19:13:28] Speaker 5: This you've seen it.\n[2025-06-01 19:13:30] Speaker 5: The approach we have taken.", + "segments": [ + { + "text": "right and please feel free to contribute to it too.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "This you've seen it.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + }, + { + "text": "The approach we have taken.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_054.wav": { + "chunk_id": "055", + "transcript": "[2025-06-01 19:13:25] Speaker 5: here is instead of asking you to poke holes in your existing cluster or networks, we basically push the", + "segments": [ + { + "text": "here is instead of asking you to poke holes in your existing cluster or networks, we basically push the", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_055.wav": { + "chunk_id": "056", + "transcript": "[2025-06-01 19:13:25] Speaker 5: metrics and we push the health check data to the Prometheus and our central control plane.", + "segments": [ + { + "text": "metrics and we push the health check data to the Prometheus and our central control plane.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_056.wav": { + "chunk_id": "057", + "transcript": "[2025-06-01 19:13:25] Speaker 5: It could be looked at as, oh, I need to open this port, that port, no exception.", + "segments": [ + { + "text": "It could be looked at as, oh, I need to open this port, that port, no exception.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_057.wav": { + "chunk_id": "058", + "transcript": "[2025-06-01 19:13:25] Speaker 5: if you're in a vc and in a production environment where everything is locked down egress is usually", + "segments": [ + { + "text": "if you're in a vc and in a production environment where everything is locked down egress is usually", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_058.wav": { + "chunk_id": "059", + "transcript": "[2025-06-01 19:13:25] Speaker 5: easier and you can always sniff the egress but these are all running within your own tenancy so", + "segments": [ + { + "text": "easier and you can always sniff the egress but these are all running within your own tenancy so", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_059.wav": { + "chunk_id": "060", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So the origination of all of the metrics to where it's being sent is all within your network.", + "segments": [ + { + "text": "So the origination of all of the metrics to where it's being sent is all within your network.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_060.wav": { + "chunk_id": "061", + "transcript": "[2025-06-01 19:13:25] Speaker 5: nothing is going into the public. All right. So I'm going to go over this.", + "segments": [ + { + "text": "nothing is going into the public. All right. So I'm going to go over this.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_061.wav": { + "chunk_id": "062", + "transcript": "[2025-06-01 19:13:25] Speaker 5: but there are plenty of metrics, just a lot.\n[2025-06-01 19:13:28] Speaker 5: And it's kind of an eye candy chart on Grafana's to look.", + "segments": [ + { + "text": "but there are plenty of metrics, just a lot.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "And it's kind of an eye candy chart on Grafana's to look.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_062.wav": { + "chunk_id": "063", + "transcript": "[2025-06-01 19:13:25] Speaker 5: at but i think you get the point right you have full nvidia dcgm exporter all metrics", + "segments": [ + { + "text": "at but i think you get the point right you have full nvidia dcgm exporter all metrics", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_063.wav": { + "chunk_id": "064", + "transcript": "[2025-06-01 19:13:25] Speaker 5: You have all AMD SMI metrics.\n[2025-06-01 19:13:27] Speaker 5: You have all of the RDMA metrics that is particularly on our ROC.", + "segments": [ + { + "text": "You have all AMD SMI metrics.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "You have all of the RDMA metrics that is particularly on our ROC.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_064.wav": { + "chunk_id": "065", + "transcript": "[2025-06-01 19:13:25] Speaker 5: implementation that we sniffed the melanocs drivers for to capture that front end next", + "segments": [ + { + "text": "implementation that we sniffed the melanocs drivers for to capture that front end next", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_065.wav": { + "chunk_id": "066", + "transcript": "[2025-06-01 19:13:25] Speaker 5: node health is included, health check, full check is included,\n[2025-06-01 19:13:29] Speaker 5: and the traditional disk IO usage.", + "segments": [ + { + "text": "node health is included, health check, full check is included,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "and the traditional disk IO usage.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_066.wav": { + "chunk_id": "067", + "transcript": "[2025-06-01 19:13:25] Speaker 5: all of those that you usually get with Prometheus already included.\n[2025-06-01 19:13:28] Speaker 5: So all of these is bundled as one.", + "segments": [ + { + "text": "all of those that you usually get with Prometheus already included.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So all of these is bundled as one.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_067.wav": { + "chunk_id": "068", + "transcript": "[2025-06-01 19:13:25] Speaker 5: unpacking.", + "segments": [ + { + "text": "unpacking.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_068.wav": { + "chunk_id": "069", + "transcript": "[2025-06-01 19:13:25] Speaker 5: On the health checks piece, these are all the tests we do today and we'll continue to add more or adjust.", + "segments": [ + { + "text": "On the health checks piece, these are all the tests we do today and we'll continue to add more or adjust.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_069.wav": { + "chunk_id": "070", + "transcript": "[2025-06-01 19:13:25] Speaker 5: based on how performance we see we do extensive benchmarking to see where the threshold should", + "segments": [ + { + "text": "based on how performance we see we do extensive benchmarking to see where the threshold should", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_070.wav": { + "chunk_id": "071", + "transcript": "[2025-06-01 19:13:25] Speaker 5: be or the baseline should be and then we either go up and down based on how we are seeing.", + "segments": [ + { + "text": "be or the baseline should be and then we either go up and down based on how we are seeing.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_071.wav": { + "chunk_id": "072", + "transcript": "[2025-06-01 19:13:25] Speaker 5: example the what should be the ideal mfu on a 140 gb against b200 or mi300x right and then we we lower it", + "segments": [ + { + "text": "example the what should be the ideal mfu on a 140 gb against b200 or mi300x right and then we we lower it", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_072.wav": { + "chunk_id": "073", + "transcript": "[2025-06-01 19:13:25] Speaker 5: if needed right so but we work with our vendors to also see if this we already doing the right way or not", + "segments": [ + { + "text": "if needed right so but we work with our vendors to also see if this we already doing the right way or not", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_073.wav": { + "chunk_id": "074", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So here are all the checks we run.\n[2025-06-01 19:13:27] Speaker 5: And all of these, once they complete, you will see a...", + "segments": [ + { + "text": "So here are all the checks we run.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "And all of these, once they complete, you will see a...", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_074.wav": { + "chunk_id": "075", + "transcript": "[2025-06-01 19:13:25] Speaker 5: metrics flowing through Grafana and you'll also see this.", + "segments": [ + { + "text": "metrics flowing through Grafana and you'll also see this.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_075.wav": { + "chunk_id": "076", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Very simple. This is how the architecture is. I'm not going to spend too much time here either.", + "segments": [ + { + "text": "Very simple. This is how the architecture is. I'm not going to spend too much time here either.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_076.wav": { + "chunk_id": "077", + "transcript": "[2025-06-01 19:13:25] Speaker 5: because we are evolving how this is going to go right now the way we will release it is this will", + "segments": [ + { + "text": "because we are evolving how this is going to go right now the way we will release it is this will", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_077.wav": { + "chunk_id": "078", + "transcript": "[2025-06-01 19:13:25] Speaker 5: deploy as a we're not a kubernetes operator yet but you have to basically deploy a dedicated", + "segments": [ + { + "text": "deploy as a we're not a kubernetes operator yet but you have to basically deploy a dedicated", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_078.wav": { + "chunk_id": "079", + "transcript": "[2025-06-01 19:13:25] Speaker 5: OKE cluster with just CPU nodes, which installs Prometheus Grafana open source and our control.", + "segments": [ + { + "text": "OKE cluster with just CPU nodes, which installs Prometheus Grafana open source and our control.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_079.wav": { + "chunk_id": "080", + "transcript": "[2025-06-01 19:13:25] Speaker 5: plane api as well as portal right those are all the components that come as part of this", + "segments": [ + { + "text": "plane api as well as portal right those are all the components that come as part of this", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_080.wav": { + "chunk_id": "081", + "transcript": "[2025-06-01 19:13:25] Speaker 5: And this is that footprint.\n[2025-06-01 19:13:27] Speaker 5: We run a local Postgres for Profana storage.\n[2025-06-01 19:13:30] Speaker 5: We also use Postgres for...", + "segments": [ + { + "text": "And this is that footprint.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "We run a local Postgres for Profana storage.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "We also use Postgres for...", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_081.wav": { + "chunk_id": "082", + "transcript": "[2025-06-01 19:13:25] Speaker 5: kind of operating these things.\n[2025-06-01 19:13:27] Speaker 5: This is our kind of MVP minus one approach, right?\n[2025-06-01 19:13:32] Speaker 5: And once this...", + "segments": [ + { + "text": "kind of operating these things.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "This is our kind of MVP minus one approach, right?", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "And once this...", + "timestamp": "2025-06-01 19:13:32", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_082.wav": { + "chunk_id": "083", + "transcript": "[2025-06-01 19:13:25] Speaker 5: becomes a service this will start looking differently any questions so far", + "segments": [ + { + "text": "becomes a service this will start looking differently any questions so far", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_083.wav": { + "chunk_id": "084", + "transcript": "[2025-06-01 19:13:25] Speaker 1: I have a few questions.\n[2025-06-01 19:13:26] Speaker 1: Do you prefer we wait till the end?", + "segments": [ + { + "text": "I have a few questions.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "Do you prefer we wait till the end?", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_084.wav": { + "chunk_id": "085", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Yeah, let's take it, Zizal.\n[2025-06-01 19:13:26] Speaker 5: Or if you want to see the demo and then ask those questions, it's up to you.", + "segments": [ + { + "text": "Yeah, let's take it, Zizal.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Or if you want to see the demo and then ask those questions, it's up to you.", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_085.wav": { + "chunk_id": "086", + "transcript": "[2025-06-01 19:13:25] Speaker 1: I guess let's do the demo and then we can come back to questions.\n[2025-06-01 19:13:28] Speaker 1: Maybe that will answer some of it.", + "segments": [ + { + "text": "I guess let's do the demo and then we can come back to questions.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "Maybe that will answer some of it.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_086.wav": { + "chunk_id": "087", + "transcript": "[2025-06-01 19:13:25] Speaker 1: questions perfect yep all right so the the way we we know you can install this is we you will", + "segments": [ + { + "text": "questions perfect yep all right so the the way we we know you can install this is we you will", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_087.wav": { + "chunk_id": "088", + "transcript": "[2025-06-01 19:13:25] Speaker 5: have a GitHub repo where it could go and say deploy this and it sets up a Kubernetes cluster.", + "segments": [ + { + "text": "have a GitHub repo where it could go and say deploy this and it sets up a Kubernetes cluster.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_088.wav": { + "chunk_id": "089", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Kubernetes cluster setup and all of the Grafana Prometeas,\n[2025-06-01 19:13:29] Speaker 5: everything is installed.", + "segments": [ + { + "text": "Kubernetes cluster setup and all of the Grafana Prometeas,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "everything is installed.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_089.wav": { + "chunk_id": "090", + "transcript": "[2025-06-01 19:13:25] Speaker 5: in your primary OSID or your tenancy region,\n[2025-06-01 19:13:29] Speaker 5: you will get access to a portal like this.", + "segments": [ + { + "text": "in your primary OSID or your tenancy region,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "you will get access to a portal like this.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_090.wav": { + "chunk_id": "091", + "transcript": "[2025-06-01 19:13:25] Speaker 5: right we call it corino lens but it is basically the lens so once you are here", + "segments": [ + { + "text": "right we call it corino lens but it is basically the lens so once you are here", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_091.wav": { + "chunk_id": "092", + "transcript": "[2025-06-01 19:13:25] Speaker 5: right it will go it has the ability to do a multi-region query in a single api", + "segments": [ + { + "text": "right it will go it has the ability to do a multi-region query in a single api", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_092.wav": { + "chunk_id": "093", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So everything is API first, you have portal and there is a REST API endpoint you can interact.", + "segments": [ + { + "text": "So everything is API first, you have portal and there is a REST API endpoint you can interact.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_093.wav": { + "chunk_id": "094", + "transcript": "[2025-06-01 19:13:25] Speaker 5: all of that is deployed in your tenancy so you can lock it down if you like to", + "segments": [ + { + "text": "all of that is deployed in your tenancy so you can lock it down if you like to", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_094.wav": { + "chunk_id": "095", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Right now we run on a public domain just for a demo, but everything in here.\n[2025-06-01 19:13:30] Speaker 5: So if you see here.", + "segments": [ + { + "text": "Right now we run on a public domain just for a demo, but everything in here.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So if you see here.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_095.wav": { + "chunk_id": "096", + "transcript": "[2025-06-01 19:13:25] Speaker 5: This is done. If I refresh this page, it's going to come back with", + "segments": [ + { + "text": "This is done. If I refresh this page, it's going to come back with", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_096.wav": { + "chunk_id": "097", + "transcript": "[2025-06-01 19:13:25] Speaker 5: It's doing all the regions that you have subscribed in your OSIT or Oracle tenancy.", + "segments": [ + { + "text": "It's doing all the regions that you have subscribed in your OSIT or Oracle tenancy.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_097.wav": { + "chunk_id": "098", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Then it comes back and tells you, here are all the instances that I can monitor.\n[2025-06-01 19:13:28] Speaker 5: These are basically...", + "segments": [ + { + "text": "Then it comes back and tells you, here are all the instances that I can monitor.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "These are basically...", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_098.wav": { + "chunk_id": "099", + "transcript": "[2025-06-01 19:13:25] Speaker 5: just GPU instances that are running okay if you have new instances running you can also add", + "segments": [ + { + "text": "just GPU instances that are running okay if you have new instances running you can also add", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_099.wav": { + "chunk_id": "100", + "transcript": "[2025-06-01 19:13:25] Speaker 5: what we call kind of plug-in.\n[2025-06-01 19:13:27] Speaker 5: So it's a plug-in model.\n[2025-06-01 19:13:28] Speaker 5: You already say, okay, start monitoring.", + "segments": [ + { + "text": "what we call kind of plug-in.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So it's a plug-in model.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "You already say, okay, start monitoring.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_100.wav": { + "chunk_id": "101", + "transcript": "[2025-06-01 19:13:25] Speaker 5: these to OCI lengths. And so if you look at here, I have Frankfurt and Ashburn machines showing.", + "segments": [ + { + "text": "these to OCI lengths. And so if you look at here, I have Frankfurt and Ashburn machines showing.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_101.wav": { + "chunk_id": "102", + "transcript": "[2025-06-01 19:13:25] Speaker 5: up and it allows me to combine all of this and say they're all part of the same team or same", + "segments": [ + { + "text": "up and it allows me to combine all of this and say they're all part of the same team or same", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_102.wav": { + "chunk_id": "103", + "transcript": "[2025-06-01 19:13:25] Speaker 5: experiment or same way i want to monitor and i can create what's called monitoring right", + "segments": [ + { + "text": "experiment or same way i want to monitor and i can create what's called monitoring right", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_103.wav": { + "chunk_id": "104", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So before I go there, right, how can you start monitoring this?\n[2025-06-01 19:13:29] Speaker 5: If you have an existing instance...", + "segments": [ + { + "text": "So before I go there, right, how can you start monitoring this?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "If you have an existing instance...", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_104.wav": { + "chunk_id": "105", + "transcript": "[2025-06-01 19:13:25] Speaker 5: running or a new instance being provisioned you have to just include this script for now right", + "segments": [ + { + "text": "running or a new instance being provisioned you have to just include this script for now right", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_105.wav": { + "chunk_id": "106", + "transcript": "[2025-06-01 19:13:25] Speaker 5: we will we will change the way it is we're going to make it a native oci plugin", + "segments": [ + { + "text": "we will we will change the way it is we're going to make it a native oci plugin", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_106.wav": { + "chunk_id": "107", + "transcript": "[2025-06-01 19:13:25] Speaker 5: once we upstream all of this to our plugin framework.\n[2025-06-01 19:13:27] Speaker 5: But for now, it's basically a tarz file.", + "segments": [ + { + "text": "once we upstream all of this to our plugin framework.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "But for now, it's basically a tarz file.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_107.wav": { + "chunk_id": "108", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and it's a Golang-based code primarily.\n[2025-06-01 19:13:28] Speaker 5: The health checks are all Python-based.", + "segments": [ + { + "text": "and it's a Golang-based code primarily.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "The health checks are all Python-based.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_108.wav": { + "chunk_id": "109", + "transcript": "[2025-06-01 19:13:25] Speaker 5: And this is basically end of the day, just all these metrics are composed and pushed.", + "segments": [ + { + "text": "And this is basically end of the day, just all these metrics are composed and pushed.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_109.wav": { + "chunk_id": "110", + "transcript": "[2025-06-01 19:13:25] Speaker 5: through Prometheus Push gateway,\n[2025-06-01 19:13:27] Speaker 5: which is getting scraped by Push.\n[2025-06-01 19:13:28] Speaker 5: So architecture.", + "segments": [ + { + "text": "through Prometheus Push gateway,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "which is getting scraped by Push.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "So architecture.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_110.wav": { + "chunk_id": "111", + "transcript": "[2025-06-01 19:13:25] Speaker 5: this is nothing very different or rocket science here right so it's very simple", + "segments": [ + { + "text": "this is nothing very different or rocket science here right so it's very simple", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_111.wav": { + "chunk_id": "112", + "transcript": "[2025-06-01 19:13:25] Speaker 5: straightforward, but what it runs and how it runs is what the value we are adding.", + "segments": [ + { + "text": "straightforward, but what it runs and how it runs is what the value we are adding.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_112.wav": { + "chunk_id": "113", + "transcript": "[2025-06-01 19:13:25] Speaker 5: This is like a cloud in its script as well.\n[2025-06-01 19:13:27] Speaker 5: You can add it as part of a new provisioning.", + "segments": [ + { + "text": "This is like a cloud in its script as well.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "You can add it as part of a new provisioning.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_113.wav": { + "chunk_id": "114", + "transcript": "[2025-06-01 19:13:25] Speaker 5: instance automatically every new instance that comes up has monitoring enabled and everything is", + "segments": [ + { + "text": "instance automatically every new instance that comes up has monitoring enabled and everything is", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_114.wav": { + "chunk_id": "115", + "transcript": "[2025-06-01 19:13:25] Speaker 5: running at the host level as a system service it's not operating at the kubernetes layers or at the", + "segments": [ + { + "text": "running at the host level as a system service it's not operating at the kubernetes layers or at the", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_115.wav": { + "chunk_id": "116", + "transcript": "[2025-06-01 19:13:25] Speaker 5: the Slurm application layers.\n[2025-06-01 19:13:27] Speaker 5: It's running directly on the bare metal host.\n[2025-06-01 19:13:30] Speaker 5: And we felt that was.", + "segments": [ + { + "text": "the Slurm application layers.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "It's running directly on the bare metal host.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "And we felt that was.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_116.wav": { + "chunk_id": "117", + "transcript": "[2025-06-01 19:13:25] Speaker 5: the ideal way and the right place for us to remove specific implementations to Kubernetes.", + "segments": [ + { + "text": "the ideal way and the right place for us to remove specific implementations to Kubernetes.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_117.wav": { + "chunk_id": "118", + "transcript": "[2025-06-01 19:13:25] Speaker 5: or Slurm or others.\n[2025-06-01 19:13:28] Speaker 5: So we can also scan Kubernetes clusters you have running\n[2025-06-01 19:13:31] Speaker 5: or you have RDMA.", + "segments": [ + { + "text": "or Slurm or others.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So we can also scan Kubernetes clusters you have running", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + }, + { + "text": "or you have RDMA.", + "timestamp": "2025-06-01 19:13:31", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_118.wav": { + "chunk_id": "119", + "transcript": "[2025-06-01 19:13:25] Speaker 5: cluster networks or what we call compute clusters and allow you to also monitor that.", + "segments": [ + { + "text": "cluster networks or what we call compute clusters and allow you to also monitor that.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_119.wav": { + "chunk_id": "120", + "transcript": "[2025-06-01 19:13:25] Speaker 5: And when you check the whole Kubernetes cluster, all the GPU nodes under it are automatically\n[2025-06-01 19:13:30] Speaker 5: scanned.", + "segments": [ + { + "text": "And when you check the whole Kubernetes cluster, all the GPU nodes under it are automatically", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "scanned.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_120.wav": { + "chunk_id": "121", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and add it to the monitoring for us.\n[2025-06-01 19:13:27] Speaker 5: That's the experience.\n[2025-06-01 19:13:30] Speaker 5: Okay.\n[2025-06-01 19:13:30] Speaker 5: So I'll quickly show you how...", + "segments": [ + { + "text": "and add it to the monitoring for us.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "That's the experience.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "Okay.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + }, + { + "text": "So I'll quickly show you how...", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_121.wav": { + "chunk_id": "122", + "transcript": "[2025-06-01 19:13:25] Speaker 5: you know this this is basically a bare minimum um portal uh access we have everything is api", + "segments": [ + { + "text": "you know this this is basically a bare minimum um portal uh access we have everything is api", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_122.wav": { + "chunk_id": "123", + "transcript": "[2025-06-01 19:13:25] Speaker 5: first so you get exactly what you're doing here you could you could do that uh through", + "segments": [ + { + "text": "first so you get exactly what you're doing here you could you could do that uh through", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_123.wav": { + "chunk_id": "124", + "transcript": "[2025-06-01 19:13:25] Speaker 5: REST API endpoints. So once this loads up, because it's doing like a close to 10 region query,", + "segments": [ + { + "text": "REST API endpoints. So once this loads up, because it's doing like a close to 10 region query,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_124.wav": { + "chunk_id": "125", + "transcript": "[2025-06-01 19:13:25] Speaker 5: takes roughly six seconds but six to ten seconds we will we'll make it better with some cash", + "segments": [ + { + "text": "takes roughly six seconds but six to ten seconds we will we'll make it better with some cash", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_125.wav": { + "chunk_id": "126", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So, yeah, no, we have close to 30, 30 plus machines, right, Joleta?\n[2025-06-01 19:13:33] Speaker 5: I think it's performing.", + "segments": [ + { + "text": "So, yeah, no, we have close to 30, 30 plus machines, right, Joleta?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "I think it's performing.", + "timestamp": "2025-06-01 19:13:33", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_126.wav": { + "chunk_id": "127", + "transcript": "[2025-06-01 19:13:25] Speaker 5: okay but you know joletta is working hard to see how she can do parallel um parallelization", + "segments": [ + { + "text": "okay but you know joletta is working hard to see how she can do parallel um parallelization", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_127.wav": { + "chunk_id": "128", + "transcript": "[2025-06-01 19:13:25] Speaker 5: of some of these runs and get back.\n[2025-06-01 19:13:27] Speaker 5: But we got them better.\n[2025-06-01 19:13:29] Speaker 5: We started from 30 seconds.", + "segments": [ + { + "text": "of some of these runs and get back.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "But we got them better.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "We started from 30 seconds.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_128.wav": { + "chunk_id": "129", + "transcript": "[2025-06-01 19:13:25] Speaker 5: by now at six seconds, but we'll keep pushing.\n[2025-06-01 19:13:29] Speaker 5: All right. So it allows me to create a...", + "segments": [ + { + "text": "by now at six seconds, but we'll keep pushing.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "All right. So it allows me to create a...", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_129.wav": { + "chunk_id": "130", + "transcript": "[2025-06-01 19:13:25] Speaker 5: monitoring ring, it's like just an arbitrary way for you to\n[2025-06-01 19:13:28] Speaker 5: combine these resources and say", + "segments": [ + { + "text": "monitoring ring, it's like just an arbitrary way for you to", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "combine these resources and say", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_130.wav": { + "chunk_id": "131", + "transcript": "[2025-06-01 19:13:25] Speaker 5: I want a dedicated dashboard for this,\n[2025-06-01 19:13:27] Speaker 5: these set of machines, right?\n[2025-06-01 19:13:29] Speaker 5: That's what you.", + "segments": [ + { + "text": "I want a dedicated dashboard for this,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "these set of machines, right?", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "That's what you.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_131.wav": { + "chunk_id": "132", + "transcript": "[2025-06-01 19:13:25] Speaker 5: allows you to create a ring and if you go to monitoring rings here and you have OK,\n[2025-06-01 19:13:30] Speaker 5: this is.", + "segments": [ + { + "text": "allows you to create a ring and if you go to monitoring rings here and you have OK,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "this is.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_132.wav": { + "chunk_id": "133", + "transcript": "[2025-06-01 19:13:25] Speaker 5: for ML training, this is for the team, this is for cost center, whatever, however you", + "segments": [ + { + "text": "for ML training, this is for the team, this is for cost center, whatever, however you", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_133.wav": { + "chunk_id": "134", + "transcript": "[2025-06-01 19:13:25] Speaker 5: want to use this for kind of bundling all the things. Right? And the thing about", + "segments": [ + { + "text": "want to use this for kind of bundling all the things. Right? And the thing about", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_134.wav": { + "chunk_id": "135", + "transcript": "[2025-06-01 19:13:25] Speaker 5: this is you can always come back and add more instances or remove the instances if they go", + "segments": [ + { + "text": "this is you can always come back and add more instances or remove the instances if they go", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_135.wav": { + "chunk_id": "136", + "transcript": "[2025-06-01 19:13:25] Speaker 5: offline or you have to turn back in whatever right so so every every ring you create monitor", + "segments": [ + { + "text": "offline or you have to turn back in whatever right so so every every ring you create monitor", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_136.wav": { + "chunk_id": "137", + "transcript": "[2025-06-01 19:13:25] Speaker 5: ring you create comes with a dedicated Grafana board which includes all the hosts that are", + "segments": [ + { + "text": "ring you create comes with a dedicated Grafana board which includes all the hosts that are", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_137.wav": { + "chunk_id": "138", + "transcript": "[2025-06-01 19:13:25] Speaker 5: part of this and I'm going to click you through the Grafana.\n[2025-06-01 19:13:31] Speaker 5: So what you see here at the first part of", + "segments": [ + { + "text": "part of this and I'm going to click you through the Grafana.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So what you see here at the first part of", + "timestamp": "2025-06-01 19:13:31", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_138.wav": { + "chunk_id": "139", + "transcript": "[2025-06-01 19:13:25] Speaker 5: this is the health summary of all the compute nodes that are part of this, right? So we added more nodes.", + "segments": [ + { + "text": "this is the health summary of all the compute nodes that are part of this, right? So we added more nodes.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_139.wav": { + "chunk_id": "140", + "transcript": "[2025-06-01 19:13:25] Speaker 5: which you're in this part just to show you demo, right?\n[2025-06-01 19:13:27] Speaker 5: So if you see the first board here is this is the.", + "segments": [ + { + "text": "which you're in this part just to show you demo, right?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So if you see the first board here is this is the.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_140.wav": { + "chunk_id": "141", + "transcript": "[2025-06-01 19:13:25] Speaker 5: actual health check of all the performance related checks that we did on the host when you activate", + "segments": [ + { + "text": "actual health check of all the performance related checks that we did on the host when you activate", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_141.wav": { + "chunk_id": "142", + "transcript": "[2025-06-01 19:13:25] Speaker 5: the plugin we will also allow you to run these on demand so if you look at the categories here", + "segments": [ + { + "text": "the plugin we will also allow you to run these on demand so if you look at the categories here", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_142.wav": { + "chunk_id": "143", + "transcript": "[2025-06-01 19:13:25] Speaker 5: compute throughput, temperature check, all these check, check, check,\n[2025-06-01 19:13:28] Speaker 5: model MFU based stuff.", + "segments": [ + { + "text": "compute throughput, temperature check, all these check, check, check,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "model MFU based stuff.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_143.wav": { + "chunk_id": "144", + "transcript": "[2025-06-01 19:13:25] Speaker 5: All of those are readily available for you and we will very soon have a link that will exactly take", + "segments": [ + { + "text": "All of those are readily available for you and we will very soon have a link that will exactly take", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_144.wav": { + "chunk_id": "145", + "transcript": "[2025-06-01 19:13:25] Speaker 5: you to a JSON that is very much deeper on all the tests. This is basically the JSON that the health check.", + "segments": [ + { + "text": "you to a JSON that is very much deeper on all the tests. This is basically the JSON that the health check.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_145.wav": { + "chunk_id": "146", + "transcript": "[2025-06-01 19:13:25] Speaker 5: script create like what's the bandwidth achieved the tflops achieved for each of the node which", + "segments": [ + { + "text": "script create like what's the bandwidth achieved the tflops achieved for each of the node which", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_146.wav": { + "chunk_id": "147", + "transcript": "[2025-06-01 19:13:25] Speaker 5: GPU ran this for example right this is the GPU ID of an H100 we ran this is actually the JSON document", + "segments": [ + { + "text": "GPU ran this for example right this is the GPU ID of an H100 we ran this is actually the JSON document", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_147.wav": { + "chunk_id": "148", + "transcript": "[2025-06-01 19:13:25] Speaker 5: you'll have access to this as well as the logs piece uh but in a grafana it looks", + "segments": [ + { + "text": "you'll have access to this as well as the logs piece uh but in a grafana it looks", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_148.wav": { + "chunk_id": "149", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Now, it's an easier way for you to digest all of this.\n[2025-06-01 19:13:27] Speaker 5: Let me put it up.", + "segments": [ + { + "text": "Now, it's an easier way for you to digest all of this.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Let me put it up.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_149.wav": { + "chunk_id": "150", + "transcript": "[2025-06-01 19:13:25] Speaker 5: end of the day it's the same data but everything is a very detailed data that you may", + "segments": [ + { + "text": "end of the day it's the same data but everything is a very detailed data that you may", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_150.wav": { + "chunk_id": "151", + "transcript": "[2025-06-01 19:13:25] Speaker 5: be looking for um than just a summary like this okay both are available okay so we will add more", + "segments": [ + { + "text": "be looking for um than just a summary like this okay both are available okay so we will add more", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_151.wav": { + "chunk_id": "152", + "transcript": "[2025-06-01 19:13:25] Speaker 5: filters when it allows you to you know go by region and check the gpu types if you have mi300x", + "segments": [ + { + "text": "filters when it allows you to you know go by region and check the gpu types if you have mi300x", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_152.wav": { + "chunk_id": "153", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So Osaka does not have, maybe I think other machines in Chicago have MI300X.", + "segments": [ + { + "text": "So Osaka does not have, maybe I think other machines in Chicago have MI300X.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_153.wav": { + "chunk_id": "154", + "transcript": "[2025-06-01 19:13:25] Speaker 5: see here yes you can see my 300x instance um so yeah these are all of both uh my 300x", + "segments": [ + { + "text": "see here yes you can see my 300x instance um so yeah these are all of both uh my 300x", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_154.wav": { + "chunk_id": "155", + "transcript": "[2025-06-01 19:13:25] Speaker 5: as well as GPU from NVidia are fully monitored.\n[2025-06-01 19:13:29] Speaker 5: And we have health checks running for both.", + "segments": [ + { + "text": "as well as GPU from NVidia are fully monitored.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "And we have health checks running for both.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_155.wav": { + "chunk_id": "156", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Okay, so all of this is the data that you may be interested in.", + "segments": [ + { + "text": "Okay, so all of this is the data that you may be interested in.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_156.wav": { + "chunk_id": "157", + "transcript": "[2025-06-01 19:13:25] Speaker 5: that allows you to kind of baseline and benchmark all of this. Okay. So all of these metrics are", + "segments": [ + { + "text": "that allows you to kind of baseline and benchmark all of this. Okay. So all of these metrics are", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_157.wav": { + "chunk_id": "158", + "transcript": "[2025-06-01 19:13:25] Speaker 5: natively available in Prometheus and if I have to show you so we push with OCI", + "segments": [ + { + "text": "natively available in Prometheus and if I have to show you so we push with OCI", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_158.wav": { + "chunk_id": "159", + "transcript": "[2025-06-01 19:13:25] Speaker 5: lens labels to this and this will allow you to extend it any further you like.", + "segments": [ + { + "text": "lens labels to this and this will allow you to extend it any further you like.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_159.wav": { + "chunk_id": "160", + "transcript": "[2025-06-01 19:13:25] Speaker 5: custom boards if you like, or you want to merge left join, right join some datasets here to get to more.", + "segments": [ + { + "text": "custom boards if you like, or you want to merge left join, right join some datasets here to get to more.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_160.wav": { + "chunk_id": "161", + "transcript": "[2025-06-01 19:13:25] Speaker 5: details everything is fully available that's just native krafanov right this is the lcl lens", + "segments": [ + { + "text": "details everything is fully available that's just native krafanov right this is the lcl lens", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_161.wav": { + "chunk_id": "162", + "transcript": "[2025-06-01 19:13:25] Speaker 5: health check stuff too. There is quite a bit of\n[2025-06-01 19:13:31] Speaker 5: health. Yeah, this is the health summary.", + "segments": [ + { + "text": "health check stuff too. There is quite a bit of", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "health. Yeah, this is the health summary.", + "timestamp": "2025-06-01 19:13:31", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_162.wav": { + "chunk_id": "163", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So all of these metrics, the only thing that we are adding,\n[2025-06-01 19:13:27] Speaker 5: this comes out as OCI lens.", + "segments": [ + { + "text": "So all of these metrics, the only thing that we are adding,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "this comes out as OCI lens.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_163.wav": { + "chunk_id": "164", + "transcript": "[2025-06-01 19:13:25] Speaker 5: anything that comes directly from the vendor,\n[2025-06-01 19:13:26] Speaker 5: like AMD or NVIDIA come as is, right?\n[2025-06-01 19:13:29] Speaker 5: DCGM append it.", + "segments": [ + { + "text": "anything that comes directly from the vendor,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "like AMD or NVIDIA come as is, right?", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 5" + }, + { + "text": "DCGM append it.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_164.wav": { + "chunk_id": "165", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So one additional thing we also did is if you know, let me go back here.", + "segments": [ + { + "text": "So one additional thing we also did is if you know, let me go back here.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_165.wav": { + "chunk_id": "166", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So this is the overall health board.\n[2025-06-01 19:13:28] Speaker 5: And there's another board that we bring is this is.", + "segments": [ + { + "text": "So this is the overall health board.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "And there's another board that we bring is this is.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_166.wav": { + "chunk_id": "167", + "transcript": "[2025-06-01 19:13:25] Speaker 5: per host data right this is the actual gpu rdma metrics uh you know you know node health and all", + "segments": [ + { + "text": "per host data right this is the actual gpu rdma metrics uh you know you know node health and all", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_167.wav": { + "chunk_id": "168", + "transcript": "[2025-06-01 19:13:25] Speaker 5: that. Additional things that we are trying to add is if you come here and look at this table.", + "segments": [ + { + "text": "that. Additional things that we are trying to add is if you come here and look at this table.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_168.wav": { + "chunk_id": "169", + "transcript": "[2025-06-01 19:13:25] Speaker 5: We will improve the UI a little bit,\n[2025-06-01 19:13:27] Speaker 5: but there is something called host metadata survey.", + "segments": [ + { + "text": "We will improve the UI a little bit,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "but there is something called host metadata survey.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_169.wav": { + "chunk_id": "170", + "transcript": "[2025-06-01 19:13:25] Speaker 5: It's basically a local service endpoint that we hit\n[2025-06-01 19:13:28] Speaker 5: and we fetch the serial number because if the-", + "segments": [ + { + "text": "It's basically a local service endpoint that we hit", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "and we fetch the serial number because if the-", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_170.wav": { + "chunk_id": "171", + "transcript": "[2025-06-01 19:13:25] Speaker 5: host is underperforming and you have to turn back it in for repair.\n[2025-06-01 19:13:29] Speaker 5: You know, some data that we have.", + "segments": [ + { + "text": "host is underperforming and you have to turn back it in for repair.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "You know, some data that we have.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_171.wav": { + "chunk_id": "172", + "transcript": "[2025-06-01 19:13:25] Speaker 5: going to publish here is going to be super helpful.\n[2025-06-01 19:13:27] Speaker 5: And we are looking at how to put an agent.", + "segments": [ + { + "text": "going to publish here is going to be super helpful.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "And we are looking at how to put an agent.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_172.wav": { + "chunk_id": "173", + "transcript": "[2025-06-01 19:13:25] Speaker 5: AI on top of this, right? Let's\n[2025-06-01 19:13:26] Speaker 5: shortcut that, right? So we\n[2025-06-01 19:13:28] Speaker 5: will instead of throwing more", + "segments": [ + { + "text": "AI on top of this, right? Let's", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "shortcut that, right? So we", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 5" + }, + { + "text": "will instead of throwing more", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_173.wav": { + "chunk_id": "174", + "transcript": "[2025-06-01 19:13:25] Speaker 5: more boards, we will kind of curate an experience where if you see a underperforming node or a bad,", + "segments": [ + { + "text": "more boards, we will kind of curate an experience where if you see a underperforming node or a bad,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_174.wav": { + "chunk_id": "175", + "transcript": "[2025-06-01 19:13:25] Speaker 5: node. We may automate some of that to see if we could automatically raise a ticket or turn the", + "segments": [ + { + "text": "node. We may automate some of that to see if we could automatically raise a ticket or turn the", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_175.wav": { + "chunk_id": "176", + "transcript": "[2025-06-01 19:13:25] Speaker 5: machine in or invoke folks in the support team to help you guys out.\n[2025-06-01 19:13:30] Speaker 5: So that's the.", + "segments": [ + { + "text": "machine in or invoke folks in the support team to help you guys out.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So that's the.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_176.wav": { + "chunk_id": "177", + "transcript": "[2025-06-01 19:13:25] Speaker 5: future roadmap. So yeah, everything here, we're going to add more stuff like what's the", + "segments": [ + { + "text": "future roadmap. So yeah, everything here, we're going to add more stuff like what's the", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_177.wav": { + "chunk_id": "178", + "transcript": "[2025-06-01 19:13:25] Speaker 5: the huda driver you're running what's the kernel versions you're running um what's the OS", + "segments": [ + { + "text": "the huda driver you're running what's the kernel versions you're running um what's the OS", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_178.wav": { + "chunk_id": "179", + "transcript": "[2025-06-01 19:13:25] Speaker 5: attached to, what's the Rockham driver version you're running, all of those data will be fully", + "segments": [ + { + "text": "attached to, what's the Rockham driver version you're running, all of those data will be fully", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_179.wav": { + "chunk_id": "180", + "transcript": "[2025-06-01 19:13:25] Speaker 5: available everything related to metadata will be published here and you can continue to", + "segments": [ + { + "text": "available everything related to metadata will be published here and you can continue to", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_180.wav": { + "chunk_id": "181", + "transcript": "[2025-06-01 19:13:25] Speaker 5: use that instead of you SSHing into the machine,\n[2025-06-01 19:13:28] Speaker 5: running some commands and all of that.", + "segments": [ + { + "text": "use that instead of you SSHing into the machine,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "running some commands and all of that.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_181.wav": { + "chunk_id": "182", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So, readily available.\n[2025-06-01 19:13:27] Speaker 5: Everything else you see here is something you may have been using already.", + "segments": [ + { + "text": "So, readily available.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Everything else you see here is something you may have been using already.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_182.wav": { + "chunk_id": "183", + "transcript": "[2025-06-01 19:13:25] Speaker 5: which is power, usage, temperature, utilization,\n[2025-06-01 19:13:28] Speaker 5: all that stuff.\n[2025-06-01 19:13:30] Speaker 5: Okay.\n[2025-06-01 19:13:31] Speaker 5: So in an actual, this-", + "segments": [ + { + "text": "which is power, usage, temperature, utilization,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "all that stuff.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + }, + { + "text": "Okay.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + }, + { + "text": "So in an actual, this-", + "timestamp": "2025-06-01 19:13:31", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_183.wav": { + "chunk_id": "184", + "transcript": "[2025-06-01 19:13:25] Speaker 5: this is this is what it is right so at least our first iteration of um can we get a monitor", + "segments": [ + { + "text": "this is this is what it is right so at least our first iteration of um can we get a monitor", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_184.wav": { + "chunk_id": "185", + "transcript": "[2025-06-01 19:13:25] Speaker 5: going which is native and health checks going which is what lci published health checks and", + "segments": [ + { + "text": "going which is native and health checks going which is what lci published health checks and", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_185.wav": { + "chunk_id": "186", + "transcript": "[2025-06-01 19:13:25] Speaker 5: just always vendor specified. So I'm gonna stop here and see if you folks have questions.", + "segments": [ + { + "text": "just always vendor specified. So I'm gonna stop here and see if you folks have questions.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_186.wav": { + "chunk_id": "187", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and also talk about some feedback if you have for us.\n[2025-06-01 19:13:32] Speaker 1: Thanks for the demo.\n[2025-06-01 19:13:33] Speaker 1: I have a few...", + "segments": [ + { + "text": "and also talk about some feedback if you have for us.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Thanks for the demo.", + "timestamp": "2025-06-01 19:13:32", + "speaker": "Speaker 1" + }, + { + "text": "I have a few...", + "timestamp": "2025-06-01 19:13:33", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_187.wav": { + "chunk_id": "188", + "transcript": "[2025-06-01 19:13:25] Speaker 1: questions um so i think you mentioned in the first slide one of the things being like um you know", + "segments": [ + { + "text": "questions um so i think you mentioned in the first slide one of the things being like um you know", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_188.wav": { + "chunk_id": "189", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Kubernetes native, cloud native,\n[2025-06-01 19:13:26] Speaker 1: which was one of the things we discussed previously\n[2025-06-01 19:13:28] Speaker 1: as being one of the...", + "segments": [ + { + "text": "Kubernetes native, cloud native,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "which was one of the things we discussed previously", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 1" + }, + { + "text": "as being one of the...", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_189.wav": { + "chunk_id": "190", + "transcript": "[2025-06-01 19:13:25] Speaker 1: key things really important for us um what other levels of integrations with oke are you planning", + "segments": [ + { + "text": "key things really important for us um what other levels of integrations with oke are you planning", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_190.wav": { + "chunk_id": "191", + "transcript": "[2025-06-01 19:13:25] Speaker 1: at all, if any.\n[2025-06-01 19:13:26] Speaker 1: Things that top of mind would be super helpful\n[2025-06-01 19:13:28] Speaker 1: for us would be being able to", + "segments": [ + { + "text": "at all, if any.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "Things that top of mind would be super helpful", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 1" + }, + { + "text": "for us would be being able to", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_191.wav": { + "chunk_id": "192", + "transcript": "[2025-06-01 19:13:25] Speaker 1: surface some of these unhealthy node conditions to OKE or to the actual Kubernetes object.", + "segments": [ + { + "text": "surface some of these unhealthy node conditions to OKE or to the actual Kubernetes object.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_192.wav": { + "chunk_id": "193", + "transcript": "[2025-06-01 19:13:25] Speaker 1: so that we can use that in our tooling.\n[2025-06-01 19:13:27] Speaker 1: And then, you know, when we're launching, submitting jobs.", + "segments": [ + { + "text": "so that we can use that in our tooling.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "And then, you know, when we're launching, submitting jobs.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_193.wav": { + "chunk_id": "194", + "transcript": "[2025-06-01 19:13:25] Speaker 1: being able to detect that a node is unhealthy.\n[2025-06-01 19:13:28] Speaker 1: The other thing I think is like the monitoring.", + "segments": [ + { + "text": "being able to detect that a node is unhealthy.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "The other thing I think is like the monitoring.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_194.wav": { + "chunk_id": "195", + "transcript": "[2025-06-01 19:13:25] Speaker 1: the rings that you mentioned, I don't think that would be super helpful for us because we're not", + "segments": [ + { + "text": "the rings that you mentioned, I don't think that would be super helpful for us because we're not", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_195.wav": { + "chunk_id": "196", + "transcript": "[2025-06-01 19:13:25] Speaker 1: assigning nodes to people or teams.\n[2025-06-01 19:13:28] Speaker 1: Instead, we're extremely dynamic.\n[2025-06-01 19:13:31] Speaker 1: So we have...", + "segments": [ + { + "text": "assigning nodes to people or teams.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "Instead, we're extremely dynamic.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 1" + }, + { + "text": "So we have...", + "timestamp": "2025-06-01 19:13:31", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_196.wav": { + "chunk_id": "197", + "transcript": "[2025-06-01 19:13:25] Speaker 1: you know, super cluster of thousands of GPUs.\n[2025-06-01 19:13:27] Speaker 1: And those nodes get used.\n[2025-06-01 19:13:30] Speaker 1: We use Q, which is a native.", + "segments": [ + { + "text": "you know, super cluster of thousands of GPUs.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "And those nodes get used.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 1" + }, + { + "text": "We use Q, which is a native.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_197.wav": { + "chunk_id": "198", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Kubernetes project to schedule workloads.\n[2025-06-01 19:13:30] Speaker 1: So at any given time, we want to know, like, this", + "segments": [ + { + "text": "Kubernetes project to schedule workloads.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "So at any given time, we want to know, like, this", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_198.wav": { + "chunk_id": "199", + "transcript": "[2025-06-01 19:13:25] Speaker 1: job is running with, you know, 128 GPUs, are any of the nodes unhealthy in this job? And so that job", + "segments": [ + { + "text": "job is running with, you know, 128 GPUs, are any of the nodes unhealthy in this job? And so that job", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_199.wav": { + "chunk_id": "200", + "transcript": "[2025-06-01 19:13:25] Speaker 1: might be using a different set of nodes than another job would be for the same team the next", + "segments": [ + { + "text": "might be using a different set of nodes than another job would be for the same team the next", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_200.wav": { + "chunk_id": "201", + "transcript": "[2025-06-01 19:13:25] Speaker 1: day. So something like that would be helpful. Okay, wonderful. I think for the answer for the first", + "segments": [ + { + "text": "day. So something like that would be helpful. Okay, wonderful. I think for the answer for the first", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_201.wav": { + "chunk_id": "202", + "transcript": "[2025-06-01 19:13:25] Speaker 5: one is uh we are working with okay team i think okay team also has added some features lately where", + "segments": [ + { + "text": "one is uh we are working with okay team i think okay team also has added some features lately where", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_202.wav": { + "chunk_id": "203", + "transcript": "[2025-06-01 19:13:25] Speaker 5: they can detect an unhealthy gpu node and start tagging them as unallocatable uh for some some", + "segments": [ + { + "text": "they can detect an unhealthy gpu node and start tagging them as unallocatable uh for some some", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_203.wav": { + "chunk_id": "204", + "transcript": "[2025-06-01 19:13:25] Speaker 5: the checks that they ran.\n[2025-06-01 19:13:27] Speaker 5: So we are actually working with them\n[2025-06-01 19:13:29] Speaker 5: to see if anything...", + "segments": [ + { + "text": "the checks that they ran.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So we are actually working with them", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "to see if anything...", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_204.wav": { + "chunk_id": "205", + "transcript": "[2025-06-01 19:13:25] Speaker 5: we find in our health check uh that uh that says that you know this node one of the gpus in the", + "segments": [ + { + "text": "we find in our health check uh that uh that says that you know this node one of the gpus in the", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_205.wav": { + "chunk_id": "206", + "transcript": "[2025-06-01 19:13:25] Speaker 5: node is continuous to underperform and communicate back on unschedulable.", + "segments": [ + { + "text": "node is continuous to underperform and communicate back on unschedulable.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_206.wav": { + "chunk_id": "207", + "transcript": "[2025-06-01 19:13:25] Speaker 5: we will have that integration point in the future for sure.\n[2025-06-01 19:13:29] Speaker 5: Right. So, and the metrics from.", + "segments": [ + { + "text": "we will have that integration point in the future for sure.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Right. So, and the metrics from.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_207.wav": { + "chunk_id": "208", + "transcript": "[2025-06-01 19:13:25] Speaker 5: OKE directly. So if there are part level, position volume level and lower", + "segments": [ + { + "text": "OKE directly. So if there are part level, position volume level and lower", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_208.wav": { + "chunk_id": "209", + "transcript": "[2025-06-01 19:13:25] Speaker 5: of good healthy prometheus metrics that come we can directly pipe all of those into this lens with", + "segments": [ + { + "text": "of good healthy prometheus metrics that come we can directly pipe all of those into this lens with", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_209.wav": { + "chunk_id": "210", + "transcript": "[2025-06-01 19:13:25] Speaker 5: added set of things from our side to kind of consolidate and give you that one view that means", + "segments": [ + { + "text": "added set of things from our side to kind of consolidate and give you that one view that means", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_210.wav": { + "chunk_id": "211", + "transcript": "[2025-06-01 19:13:25] Speaker 5: be needed. So that definitely is the plan, but we are looking into, okay, now a customer has to run.", + "segments": [ + { + "text": "be needed. So that definitely is the plan, but we are looking into, okay, now a customer has to run.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_211.wav": { + "chunk_id": "212", + "transcript": "[2025-06-01 19:13:25] Speaker 5: a Prometheus instance in a Kubernetes cluster and that complication is what we're trying to do.", + "segments": [ + { + "text": "a Prometheus instance in a Kubernetes cluster and that complication is what we're trying to do.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_212.wav": { + "chunk_id": "213", + "transcript": "[2025-06-01 19:13:25] Speaker 5: tackle the next feature set.\n[2025-06-01 19:13:29] Speaker 1: Okay, makes sense.\n[2025-06-01 19:13:30] Speaker 1: Yeah, just to reemphasize.", + "segments": [ + { + "text": "tackle the next feature set.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Okay, makes sense.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 1" + }, + { + "text": "Yeah, just to reemphasize.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_213.wav": { + "chunk_id": "214", + "transcript": "[2025-06-01 19:13:25] Speaker 1: this is like probably the most important thing for us like really being able to", + "segments": [ + { + "text": "this is like probably the most important thing for us like really being able to", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_214.wav": { + "chunk_id": "215", + "transcript": "[2025-06-01 19:13:25] Speaker 1: because we want to be able to do this automatically, right?\n[2025-06-01 19:13:28] Speaker 1: If a GPU breaks overnight, we don't want...", + "segments": [ + { + "text": "because we want to be able to do this automatically, right?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "If a GPU breaks overnight, we don't want...", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_215.wav": { + "chunk_id": "216", + "transcript": "[2025-06-01 19:13:25] Speaker 1: someone to have to, you know, go do something, go look at a Grafana dashboard we want.", + "segments": [ + { + "text": "someone to have to, you know, go do something, go look at a Grafana dashboard we want.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_216.wav": { + "chunk_id": "217", + "transcript": "[2025-06-01 19:13:25] Speaker 1: our jobs to automatically react to these things.\n[2025-06-01 19:13:28] Speaker 5: Yeah. So I think that that brings another second.", + "segments": [ + { + "text": "our jobs to automatically react to these things.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "Yeah. So I think that that brings another second.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_217.wav": { + "chunk_id": "218", + "transcript": "[2025-06-01 19:13:25] Speaker 5: question or even feedback you had for us is like the static way of creating monitoring ranks is", + "segments": [ + { + "text": "question or even feedback you had for us is like the static way of creating monitoring ranks is", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_218.wav": { + "chunk_id": "219", + "transcript": "[2025-06-01 19:13:25] Speaker 5: not ideal we fully hear you uh i think because of the dynamic if you're using q where you know", + "segments": [ + { + "text": "not ideal we fully hear you uh i think because of the dynamic if you're using q where you know", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_219.wav": { + "chunk_id": "220", + "transcript": "[2025-06-01 19:13:25] Speaker 5: you get it booted out based on what team and priority and cohort and all of those stuff are.", + "segments": [ + { + "text": "you get it booted out based on what team and priority and cohort and all of those stuff are.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_220.wav": { + "chunk_id": "221", + "transcript": "[2025-06-01 19:13:25] Speaker 5: We will integrate with OKE and the way we will look at is where the job is running or where", + "segments": [ + { + "text": "We will integrate with OKE and the way we will look at is where the job is running or where", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_221.wav": { + "chunk_id": "222", + "transcript": "[2025-06-01 19:13:25] Speaker 5: the deployment is running at a point of time and try to dynamically what we call add more tags to", + "segments": [ + { + "text": "the deployment is running at a point of time and try to dynamically what we call add more tags to", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_222.wav": { + "chunk_id": "223", + "transcript": "[2025-06-01 19:13:25] Speaker 5: metadata that's coming in to exactly relate to what experiment who ran it when when did it", + "segments": [ + { + "text": "metadata that's coming in to exactly relate to what experiment who ran it when when did it", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_223.wav": { + "chunk_id": "224", + "transcript": "[2025-06-01 19:13:25] Speaker 5: complete timestamp it and try to pull those in.\n[2025-06-01 19:13:28] Speaker 5: So it's a thing that we are working.", + "segments": [ + { + "text": "complete timestamp it and try to pull those in.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So it's a thing that we are working.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_224.wav": { + "chunk_id": "225", + "transcript": "[2025-06-01 19:13:25] Speaker 5: towards, but we started with more non-Kubernetes-based experience, and then we would quick switch.", + "segments": [ + { + "text": "towards, but we started with more non-Kubernetes-based experience, and then we would quick switch.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_225.wav": { + "chunk_id": "226", + "transcript": "[2025-06-01 19:13:25] Speaker 5: move into the dynamic workload where we pull in the nodes and even the GPUs, right?\n[2025-06-01 19:13:30] Speaker 5: Not full nodes.", + "segments": [ + { + "text": "move into the dynamic workload where we pull in the nodes and even the GPUs, right?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Not full nodes.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_226.wav": { + "chunk_id": "227", + "transcript": "[2025-06-01 19:13:25] Speaker 5: be running every experiment so we will try to say four out of that no GPUs were running for this", + "segments": [ + { + "text": "be running every experiment so we will try to say four out of that no GPUs were running for this", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_227.wav": { + "chunk_id": "228", + "transcript": "[2025-06-01 19:13:25] Speaker 5: experiment and that's what is aggregated into that board.\n[2025-06-01 19:13:28] Speaker 5: We will, that's a good point.", + "segments": [ + { + "text": "experiment and that's what is aggregated into that board.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "We will, that's a good point.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_228.wav": { + "chunk_id": "229", + "transcript": "[2025-06-01 19:13:25] Speaker 5: feedback and I think we will try to prioritize that.\n[2025-06-01 19:13:27] Speaker 5: Yeah, I think you're not very far from.", + "segments": [ + { + "text": "feedback and I think we will try to prioritize that.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Yeah, I think you're not very far from.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_229.wav": { + "chunk_id": "230", + "transcript": "[2025-06-01 19:13:25] Speaker 1: that honestly even without having to do like tagging of instances and trying to keep track of", + "segments": [ + { + "text": "that honestly even without having to do like tagging of instances and trying to keep track of", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_230.wav": { + "chunk_id": "231", + "transcript": "[2025-06-01 19:13:25] Speaker 1: what's running what. I think you could do some queries in your dashboard that just...", + "segments": [ + { + "text": "what's running what. I think you could do some queries in your dashboard that just...", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_231.wav": { + "chunk_id": "232", + "transcript": "[2025-06-01 19:13:25] Speaker 1: shows like your health metrics and then also use Kubernetes metrics and see like for", + "segments": [ + { + "text": "shows like your health metrics and then also use Kubernetes metrics and see like for", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_232.wav": { + "chunk_id": "233", + "transcript": "[2025-06-01 19:13:25] Speaker 1: any given job or deployment, like you said, show all the nodes of that job and show if", + "segments": [ + { + "text": "any given job or deployment, like you said, show all the nodes of that job and show if", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_233.wav": { + "chunk_id": "234", + "transcript": "[2025-06-01 19:13:25] Speaker 1: of them are unhealthy so i think it's just a matter of like tying everything together", + "segments": [ + { + "text": "of them are unhealthy so i think it's just a matter of like tying everything together", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_234.wav": { + "chunk_id": "235", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Yeah, I fully agree. Actually, we started like that, by the way, and I think Joleta and...", + "segments": [ + { + "text": "Yeah, I fully agree. Actually, we started like that, by the way, and I think Joleta and...", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_235.wav": { + "chunk_id": "236", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and most of my team knows that we started with a full Kubernetes-based experience, but we were nudged.", + "segments": [ + { + "text": "and most of my team knows that we started with a full Kubernetes-based experience, but we were nudged.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_236.wav": { + "chunk_id": "237", + "transcript": "[2025-06-01 19:13:25] Speaker 5: on like well i think there are a lot of people who want a full monitoring and this is where we", + "segments": [ + { + "text": "on like well i think there are a lot of people who want a full monitoring and this is where we", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_237.wav": { + "chunk_id": "238", + "transcript": "[2025-06-01 19:13:25] Speaker 5: I think we have the both. We like to just pull it in like you said.", + "segments": [ + { + "text": "I think we have the both. We like to just pull it in like you said.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_238.wav": { + "chunk_id": "239", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and start relating where, where, what is running\n[2025-06-01 19:13:28] Speaker 5: and start kind of creating that.", + "segments": [ + { + "text": "and start relating where, where, what is running", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "and start kind of creating that.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_239.wav": { + "chunk_id": "240", + "transcript": "[2025-06-01 19:13:25] Speaker 5: key value pair mapping here.", + "segments": [ + { + "text": "key value pair mapping here.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_240.wav": { + "chunk_id": "241", + "transcript": "[2025-06-01 19:13:25] Speaker 1: That's another question I'll have and then I'll pass it to like other.", + "segments": [ + { + "text": "That's another question I'll have and then I'll pass it to like other.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_241.wav": { + "chunk_id": "242", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Luis, Ace, if you have any questions, is you mentioned health checks. Can you talk?", + "segments": [ + { + "text": "Luis, Ace, if you have any questions, is you mentioned health checks. Can you talk?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_242.wav": { + "chunk_id": "243", + "transcript": "[2025-06-01 19:13:25] Speaker 1: a bit more about um how those would run like are you running anything that requires the workload", + "segments": [ + { + "text": "a bit more about um how those would run like are you running anything that requires the workload", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_243.wav": { + "chunk_id": "244", + "transcript": "[2025-06-01 19:13:25] Speaker 1: to be idle and if so is it or the gpu note to be idle and if so is it going to be doing", + "segments": [ + { + "text": "to be idle and if so is it or the gpu note to be idle and if so is it going to be doing", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_244.wav": { + "chunk_id": "245", + "transcript": "[2025-06-01 19:13:25] Speaker 1: something like when nodes go idle to check the gpus or are these like running passively in the background", + "segments": [ + { + "text": "something like when nodes go idle to check the gpus or are these like running passively in the background", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_245.wav": { + "chunk_id": "246", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So I'll ask Soumya to chime in as well on she's an ML engineer worked on the script.", + "segments": [ + { + "text": "So I'll ask Soumya to chime in as well on she's an ML engineer worked on the script.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_246.wav": { + "chunk_id": "247", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So we run this, we expect the GPUs to be idle and no workloads to be running.", + "segments": [ + { + "text": "So we run this, we expect the GPUs to be idle and no workloads to be running.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_247.wav": { + "chunk_id": "248", + "transcript": "[2025-06-01 19:13:25] Speaker 5: this is where the PyTorch is loading into the tensor so it's loading into the accelerator.", + "segments": [ + { + "text": "this is where the PyTorch is loading into the tensor so it's loading into the accelerator.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_248.wav": { + "chunk_id": "249", + "transcript": "[2025-06-01 19:13:25] Speaker 5: running all these checks and coming back.\n[2025-06-01 19:13:27] Speaker 5: So we expect no workloads to be running when this.", + "segments": [ + { + "text": "running all these checks and coming back.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So we expect no workloads to be running when this.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_249.wav": { + "chunk_id": "250", + "transcript": "[2025-06-01 19:13:25] Speaker 5: this is running. We run it when you say you need to run it.\n[2025-06-01 19:13:28] Speaker 5: Right now we have not automated it.", + "segments": [ + { + "text": "this is running. We run it when you say you need to run it.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Right now we have not automated it.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_250.wav": { + "chunk_id": "251", + "transcript": "[2025-06-01 19:13:25] Speaker 5: And this is where we need feedback.\n[2025-06-01 19:13:26] Speaker 5: Like, do you want to run at midnight, 12 o'clock every day?", + "segments": [ + { + "text": "And this is where we need feedback.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Like, do you want to run at midnight, 12 o'clock every day?", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_251.wav": { + "chunk_id": "252", + "transcript": "[2025-06-01 19:13:25] Speaker 5: right you need that kind of no definitely not", + "segments": [ + { + "text": "right you need that kind of no definitely not", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_252.wav": { + "chunk_id": "253", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Yeah, if I'm an ML engineer, I'm like, okay, I know when my pipeline is at a pause where I say call.", + "segments": [ + { + "text": "Yeah, if I'm an ML engineer, I'm like, okay, I know when my pipeline is at a pause where I say call.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_253.wav": { + "chunk_id": "254", + "transcript": "[2025-06-01 19:13:25] Speaker 5: of my gpus are flushed out and idle i want to run the health checks for everything and make sure", + "segments": [ + { + "text": "of my gpus are flushed out and idle i want to run the health checks for everything and make sure", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_254.wav": { + "chunk_id": "255", + "transcript": "[2025-06-01 19:13:25] Speaker 5: of that is good before we go to the next phase, right? Is that an experience you would look for?", + "segments": [ + { + "text": "of that is good before we go to the next phase, right? Is that an experience you would look for?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_255.wav": { + "chunk_id": "256", + "transcript": "[2025-06-01 19:13:25] Speaker 5: What we are thinking is we will give you a on-demand way for you to invoke health checkscripts.", + "segments": [ + { + "text": "What we are thinking is we will give you a on-demand way for you to invoke health checkscripts.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_256.wav": { + "chunk_id": "257", + "transcript": "[2025-06-01 19:13:25] Speaker 5: through a REST API.\n[2025-06-01 19:13:26] Speaker 5: You have to just say\n[2025-06-01 19:13:28] Speaker 5: when you want to run it\n[2025-06-01 19:13:30] Speaker 5: and you on-demand run it.", + "segments": [ + { + "text": "through a REST API.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "You have to just say", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 5" + }, + { + "text": "when you want to run it", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + }, + { + "text": "and you on-demand run it.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_257.wav": { + "chunk_id": "258", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Because we don't know when your GPUs are idle, then they're not.", + "segments": [ + { + "text": "Because we don't know when your GPUs are idle, then they're not.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_258.wav": { + "chunk_id": "259", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Interesting. I was thinking like you might be able to figure that out like from a program.", + "segments": [ + { + "text": "Interesting. I was thinking like you might be able to figure that out like from a program.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_259.wav": { + "chunk_id": "260", + "transcript": "[2025-06-01 19:13:25] Speaker 1: programmatic standpoint, you might be able to get that data and whenever they are idle.", + "segments": [ + { + "text": "programmatic standpoint, you might be able to get that data and whenever they are idle.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_260.wav": { + "chunk_id": "261", + "transcript": "[2025-06-01 19:13:25] Speaker 1: kick off like a workload that runs the Huff check.\n[2025-06-01 19:13:30] Speaker 1: So the intuition behind creating", + "segments": [ + { + "text": "kick off like a workload that runs the Huff check.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "So the intuition behind creating", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_261.wav": { + "chunk_id": "262", + "transcript": "[2025-06-01 19:13:25] Speaker 3: this health check recipe is imagine a machine learning engineer or like you have a team that's", + "segments": [ + { + "text": "this health check recipe is imagine a machine learning engineer or like you have a team that's", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_262.wav": { + "chunk_id": "263", + "transcript": "[2025-06-01 19:13:25] Speaker 3: going to do a multi-node training or like multi-node fine tuning, right?\n[2025-06-01 19:13:29] Speaker 3: Before you go into doing...", + "segments": [ + { + "text": "going to do a multi-node training or like multi-node fine tuning, right?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + }, + { + "text": "Before you go into doing...", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_263.wav": { + "chunk_id": "264", + "transcript": "[2025-06-01 19:13:25] Speaker 3: a large-scale training operation or like any kind of like workload that's going to take too much", + "segments": [ + { + "text": "a large-scale training operation or like any kind of like workload that's going to take too much", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_264.wav": { + "chunk_id": "265", + "transcript": "[2025-06-01 19:13:25] Speaker 3: demand of the GPU itself.\n[2025-06-01 19:13:27] Speaker 3: You'd want to run this health check before that to understand that", + "segments": [ + { + "text": "demand of the GPU itself.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + }, + { + "text": "You'd want to run this health check before that to understand that", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_265.wav": { + "chunk_id": "266", + "transcript": "[2025-06-01 19:13:25] Speaker 3: health of your infrastructure right for example the way this health check is designed is it's based", + "segments": [ + { + "text": "health of your infrastructure right for example the way this health check is designed is it's based", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_266.wav": { + "chunk_id": "267", + "transcript": "[2025-06-01 19:13:25] Speaker 3: on just 10 0 matrix multiplication right so depending upon the matrix size you probably be", + "segments": [ + { + "text": "on just 10 0 matrix multiplication right so depending upon the matrix size you probably be", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_267.wav": { + "chunk_id": "268", + "transcript": "[2025-06-01 19:13:25] Speaker 3: loading like 818 like imagine like a matrix size of like 8192 by 818", + "segments": [ + { + "text": "loading like 818 like imagine like a matrix size of like 8192 by 818", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_268.wav": { + "chunk_id": "269", + "transcript": "[2025-06-01 19:13:25] Speaker 3: too right you're going to be loading all these matrix into the gpu memory and we are going to", + "segments": [ + { + "text": "too right you're going to be loading all these matrix into the gpu memory and we are going to", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_269.wav": { + "chunk_id": "270", + "transcript": "[2025-06-01 19:13:25] Speaker 3: pressurize and see how much your machine is able to take the computation throughput or like throat", + "segments": [ + { + "text": "pressurize and see how much your machine is able to take the computation throughput or like throat", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_270.wav": { + "chunk_id": "271", + "transcript": "[2025-06-01 19:13:25] Speaker 3: the power and things like that. So ideally you would want to do it when your machine is idle and to", + "segments": [ + { + "text": "the power and things like that. So ideally you would want to do it when your machine is idle and to", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_271.wav": { + "chunk_id": "272", + "transcript": "[2025-06-01 19:13:25] Speaker 3: address your point yeah we could actually schedule and understand when the jobs are not running", + "segments": [ + { + "text": "address your point yeah we could actually schedule and understand when the jobs are not running", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_272.wav": { + "chunk_id": "273", + "transcript": "[2025-06-01 19:13:25] Speaker 3: and we could just run the health check at that point.\n[2025-06-01 19:13:27] Speaker 3: And we can detect when the machines are...", + "segments": [ + { + "text": "and we could just run the health check at that point.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + }, + { + "text": "And we can detect when the machines are...", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_273.wav": { + "chunk_id": "274", + "transcript": "[2025-06-01 19:13:25] Speaker 3: Yeah, I think it'd be interesting to have the option to schedule it on demand, like as a...", + "segments": [ + { + "text": "Yeah, I think it'd be interesting to have the option to schedule it on demand, like as a...", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 3" + } + ], + "language": "en" + }, + "chunk_274.wav": { + "chunk_id": "275", + "transcript": "[2025-06-01 19:13:25] Speaker 1: we hook like to the jobs like you said um but i don't i need to think more about this", + "segments": [ + { + "text": "we hook like to the jobs like you said um but i don't i need to think more about this", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_275.wav": { + "chunk_id": "276", + "transcript": "[2025-06-01 19:13:25] Speaker 1: But I think, you know, my intuition is why not both, you know, like having a periodic check.", + "segments": [ + { + "text": "But I think, you know, my intuition is why not both, you know, like having a periodic check.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_276.wav": { + "chunk_id": "277", + "transcript": "[2025-06-01 19:13:25] Speaker 1: that best effort runs when the node is sitting there doing nothing.\n[2025-06-01 19:13:29] Speaker 1: So that if a node does go unhealthy,", + "segments": [ + { + "text": "that best effort runs when the node is sitting there doing nothing.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "So that if a node does go unhealthy,", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_277.wav": { + "chunk_id": "278", + "transcript": "[2025-06-01 19:13:25] Speaker 1: we can find out about it early and not wait for a job to get scheduled there to find out.", + "segments": [ + { + "text": "we can find out about it early and not wait for a job to get scheduled there to find out.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_278.wav": { + "chunk_id": "279", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Yeah, no, we'll take it as a good feedback.\n[2025-06-01 19:13:28] Speaker 5: We have still, because we have to think.", + "segments": [ + { + "text": "Yeah, no, we'll take it as a good feedback.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "We have still, because we have to think.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_279.wav": { + "chunk_id": "280", + "transcript": "[2025-06-01 19:13:25] Speaker 5: through a little bit on scheduling because it takes roughly five minutes for this head check", + "segments": [ + { + "text": "through a little bit on scheduling because it takes roughly five minutes for this head check", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_280.wav": { + "chunk_id": "281", + "transcript": "[2025-06-01 19:13:25] Speaker 5: to finish and in mix of this if your queue ends up scheduling another job uh they'll they'll", + "segments": [ + { + "text": "to finish and in mix of this if your queue ends up scheduling another job uh they'll they'll", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_281.wav": { + "chunk_id": "282", + "transcript": "[2025-06-01 19:13:25] Speaker 5: it can't find an ideal GPU. So they go on pending. So we have to just think through this.", + "segments": [ + { + "text": "it can't find an ideal GPU. So they go on pending. So we have to just think through this.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_282.wav": { + "chunk_id": "283", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Yeah, no, this would definitely, yes, this would definitely have to be an...", + "segments": [ + { + "text": "Yeah, no, this would definitely, yes, this would definitely have to be an...", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_283.wav": { + "chunk_id": "284", + "transcript": "[2025-06-01 19:13:25] Speaker 1: interruptible workload so that our workloads can always schedule the priority.", + "segments": [ + { + "text": "interruptible workload so that our workloads can always schedule the priority.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_284.wav": { + "chunk_id": "285", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Exactly right you cannot boot this workload because this is running at the system", + "segments": [ + { + "text": "Exactly right you cannot boot this workload because this is running at the system", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_285.wav": { + "chunk_id": "286", + "transcript": "[2025-06-01 19:13:25] Speaker 5: level, right? And Kubernetes layers are at a much higher layer too.", + "segments": [ + { + "text": "level, right? And Kubernetes layers are at a much higher layer too.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_286.wav": { + "chunk_id": "287", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and we need to think of this a little bit but i think this is really good feedback um thank you", + "segments": [ + { + "text": "and we need to think of this a little bit but i think this is really good feedback um thank you", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_287.wav": { + "chunk_id": "288", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Yeah, no, thank you.\n[2025-06-01 19:13:26] Speaker 1: Louise Ace, you have anything?", + "segments": [ + { + "text": "Yeah, no, thank you.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "Louise Ace, you have anything?", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_288.wav": { + "chunk_id": "289", + "transcript": "[2025-06-01 19:13:25] Speaker 4: Mostly just echoing your thoughts. I mean, when we talk about node lifecycle,", + "segments": [ + { + "text": "Mostly just echoing your thoughts. I mean, when we talk about node lifecycle,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 4" + } + ], + "language": "en" + }, + "chunk_289.wav": { + "chunk_id": "290", + "transcript": "[2025-06-01 19:13:25] Speaker 4: It's very much like an ongoing thing.\n[2025-06-01 19:13:26] Speaker 4: It's not sort of a one off thing.\n[2025-06-01 19:13:27] Speaker 4: So just, yeah.", + "segments": [ + { + "text": "It's very much like an ongoing thing.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 4" + }, + { + "text": "It's not sort of a one off thing.", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 4" + }, + { + "text": "So just, yeah.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 4" + } + ], + "language": "en" + }, + "chunk_290.wav": { + "chunk_id": "291", + "transcript": "[2025-06-01 19:13:25] Speaker 4: like having both elements and thinking about how nodes like proceed through.", + "segments": [ + { + "text": "like having both elements and thinking about how nodes like proceed through.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 4" + } + ], + "language": "en" + }, + "chunk_291.wav": { + "chunk_id": "292", + "transcript": "[2025-06-01 19:13:25] Speaker 4: I don't want to say this process exactly, but like, I don't know, mostly just echoing Cecile's", + "segments": [ + { + "text": "I don't want to say this process exactly, but like, I don't know, mostly just echoing Cecile's", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 4" + } + ], + "language": "en" + }, + "chunk_292.wav": { + "chunk_id": "293", + "transcript": "[2025-06-01 19:13:25] Speaker 4: thoughts. Okay. Yeah. Thank you. So what, what we are trying to also avoid is before you shut this", + "segments": [ + { + "text": "thoughts. Okay. Yeah. Thank you. So what, what we are trying to also avoid is before you shut this", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 4" + } + ], + "language": "en" + }, + "chunk_293.wav": { + "chunk_id": "294", + "transcript": "[2025-06-01 19:13:25] Speaker 5: machine and turn it back into OCI to repair. We want to see what is the issue, right?", + "segments": [ + { + "text": "machine and turn it back into OCI to repair. We want to see what is the issue, right?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_294.wav": { + "chunk_id": "295", + "transcript": "[2025-06-01 19:13:25] Speaker 5: deeper into what where is the concern is it a single gpu always coming back with a lower", + "segments": [ + { + "text": "deeper into what where is the concern is it a single gpu always coming back with a lower", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_295.wav": { + "chunk_id": "296", + "transcript": "[2025-06-01 19:13:25] Speaker 5: performance compared to the rest seven of them in the same node or is it consistent uh no", + "segments": [ + { + "text": "performance compared to the rest seven of them in the same node or is it consistent uh no", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_296.wav": { + "chunk_id": "297", + "transcript": "[2025-06-01 19:13:25] Speaker 5: RDMA nick flapping issues, errors that show up.\n[2025-06-01 19:13:28] Speaker 5: And this is a start.\n[2025-06-01 19:13:30] Speaker 5: And where we want to go.", + "segments": [ + { + "text": "RDMA nick flapping issues, errors that show up.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "And this is a start.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + }, + { + "text": "And where we want to go.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_297.wav": { + "chunk_id": "298", + "transcript": "[2025-06-01 19:13:25] Speaker 5: is if we start seeing a pattern of things which are very consistent with a lot of other customers", + "segments": [ + { + "text": "is if we start seeing a pattern of things which are very consistent with a lot of other customers", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_298.wav": { + "chunk_id": "299", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and a lot of type of GPUs,\n[2025-06-01 19:13:26] Speaker 5: we will come back with recommendations\n[2025-06-01 19:13:28] Speaker 5: through an agentic AI type of app.", + "segments": [ + { + "text": "and a lot of type of GPUs,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "we will come back with recommendations", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 5" + }, + { + "text": "through an agentic AI type of app.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_299.wav": { + "chunk_id": "300", + "transcript": "[2025-06-01 19:13:25] Speaker 5: flow where probably just a reboot may fix the things or it could be specific to a driver.", + "segments": [ + { + "text": "flow where probably just a reboot may fix the things or it could be specific to a driver.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_300.wav": { + "chunk_id": "301", + "transcript": "[2025-06-01 19:13:25] Speaker 5: the GPU driver you have, which has seen this incompatibility with the Mellanox drivers we have.", + "segments": [ + { + "text": "the GPU driver you have, which has seen this incompatibility with the Mellanox drivers we have.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_301.wav": { + "chunk_id": "302", + "transcript": "[2025-06-01 19:13:25] Speaker 5: example. So we are trying to learn from using the data as well as all the issues customers.", + "segments": [ + { + "text": "example. So we are trying to learn from using the data as well as all the issues customers.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_302.wav": { + "chunk_id": "303", + "transcript": "[2025-06-01 19:13:25] Speaker 5: give us to start recommending, right?\n[2025-06-01 19:13:27] Speaker 5: So this is just a start for us to start collecting,\n[2025-06-01 19:13:29] Speaker 5: but next approach for us is", + "segments": [ + { + "text": "give us to start recommending, right?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So this is just a start for us to start collecting,", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "but next approach for us is", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_303.wav": { + "chunk_id": "304", + "transcript": "[2025-06-01 19:13:25] Speaker 5: remove the noise and tell if there is really a issue with the host or if this is a transient issue.", + "segments": [ + { + "text": "remove the noise and tell if there is really a issue with the host or if this is a transient issue.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_304.wav": { + "chunk_id": "305", + "transcript": "[2025-06-01 19:13:25] Speaker 5: or not an issue with something related to an experiment that has been set up.", + "segments": [ + { + "text": "or not an issue with something related to an experiment that has been set up.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_305.wav": { + "chunk_id": "306", + "transcript": "[2025-06-01 19:13:25] Speaker 5: workload is being scheduled on this or the topology that it's been deployed with.\n[2025-06-01 19:13:29] Speaker 5: Any of those questions?", + "segments": [ + { + "text": "workload is being scheduled on this or the topology that it's been deployed with.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Any of those questions?", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_306.wav": { + "chunk_id": "307", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Right, so that's where we want to get to.", + "segments": [ + { + "text": "Right, so that's where we want to get to.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_307.wav": { + "chunk_id": "308", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Awesome. So here's the log as well.\n[2025-06-01 19:13:28] Speaker 5: This is this we'll build up more and more.", + "segments": [ + { + "text": "Awesome. So here's the log as well.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "This is this we'll build up more and more.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_308.wav": { + "chunk_id": "309", + "transcript": "[2025-06-01 19:13:25] Speaker 5: stuff and it's all accessible either through portal or an api where you don't have to assess", + "segments": [ + { + "text": "stuff and it's all accessible either through portal or an api where you don't have to assess", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_309.wav": { + "chunk_id": "310", + "transcript": "[2025-06-01 19:13:25] Speaker 5: an instance. Because the scripts are going to be standardized that what you see against", + "segments": [ + { + "text": "an instance. Because the scripts are going to be standardized that what you see against", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_310.wav": { + "chunk_id": "311", + "transcript": "[2025-06-01 19:13:25] Speaker 5: what OCI support sees everything will be consistent and maybe in the future once we have support team", + "segments": [ + { + "text": "what OCI support sees everything will be consistent and maybe in the future once we have support team", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_311.wav": { + "chunk_id": "312", + "transcript": "[2025-06-01 19:13:25] Speaker 5: running full diagnosis they know where to narrow down the issue to like exact GPU or the NICs or", + "segments": [ + { + "text": "running full diagnosis they know where to narrow down the issue to like exact GPU or the NICs or", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_312.wav": { + "chunk_id": "313", + "transcript": "[2025-06-01 19:13:25] Speaker 5: anything instead of running this full you know six hours uh diagnosis on the host right", + "segments": [ + { + "text": "anything instead of running this full you know six hours uh diagnosis on the host right", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_313.wav": { + "chunk_id": "314", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So we're just trying to narrow down that time it takes to fix these machines too.\n[2025-06-01 19:13:30] Speaker 5: So that will help.", + "segments": [ + { + "text": "So we're just trying to narrow down that time it takes to fix these machines too.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So that will help.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_314.wav": { + "chunk_id": "315", + "transcript": "[2025-06-01 19:13:25] Speaker 1: When you say consistent, is the idea that you give us a set of scripts and we run these?", + "segments": [ + { + "text": "When you say consistent, is the idea that you give us a set of scripts and we run these?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_315.wav": { + "chunk_id": "316", + "transcript": "[2025-06-01 19:13:25] Speaker 1: or they see the results of our health checks.", + "segments": [ + { + "text": "or they see the results of our health checks.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_316.wav": { + "chunk_id": "317", + "transcript": "[2025-06-01 19:13:25] Speaker 1: So right now with the way it is integrated, the OCI support does not see the results, but you could just take...", + "segments": [ + { + "text": "So right now with the way it is integrated, the OCI support does not see the results, but you could just take...", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_317.wav": { + "chunk_id": "318", + "transcript": "[2025-06-01 19:13:25] Speaker 5: these results copy paste in your support ticket if you need to OCI support will have access", + "segments": [ + { + "text": "these results copy paste in your support ticket if you need to OCI support will have access", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_318.wav": { + "chunk_id": "319", + "transcript": "[2025-06-01 19:13:25] Speaker 5: the same health check tools and scripts that you are running.\n[2025-06-01 19:13:29] Speaker 5: So it's a consistent tool.", + "segments": [ + { + "text": "the same health check tools and scripts that you are running.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So it's a consistent tool.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_319.wav": { + "chunk_id": "320", + "transcript": "[2025-06-01 19:13:25] Speaker 5: that you both are running,\n[2025-06-01 19:13:26] Speaker 5: where what you have seen as the results,\n[2025-06-01 19:13:29] Speaker 5: for example, the score,", + "segments": [ + { + "text": "that you both are running,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "where what you have seen as the results,", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 5" + }, + { + "text": "for example, the score,", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_320.wav": { + "chunk_id": "321", + "transcript": "[2025-06-01 19:13:25] Speaker 5: So here's the flops, for example.\n[2025-06-01 19:13:28] Speaker 5: This is the duration, right?\n[2025-06-01 19:13:30] Speaker 5: What you are seeing...", + "segments": [ + { + "text": "So here's the flops, for example.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "This is the duration, right?", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + }, + { + "text": "What you are seeing...", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_321.wav": { + "chunk_id": "322", + "transcript": "[2025-06-01 19:13:25] Speaker 5: when OCI support is trying to run this on their own.\n[2025-06-01 19:13:28] Speaker 5: There is a lot of consistency.", + "segments": [ + { + "text": "when OCI support is trying to run this on their own.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "There is a lot of consistency.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_322.wav": { + "chunk_id": "323", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Right. And if you see this GPU consistently coming back with like, okay, it takes 31 seconds.", + "segments": [ + { + "text": "Right. And if you see this GPU consistently coming back with like, okay, it takes 31 seconds.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_323.wav": { + "chunk_id": "324", + "transcript": "[2025-06-01 19:13:25] Speaker 5: It's easy to narrow down saying that GQ7 is something.", + "segments": [ + { + "text": "It's easy to narrow down saying that GQ7 is something.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_324.wav": { + "chunk_id": "325", + "transcript": "[2025-06-01 19:13:25] Speaker 5: that we need to request.", + "segments": [ + { + "text": "that we need to request.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_325.wav": { + "chunk_id": "326", + "transcript": "[2025-06-01 19:13:25] Speaker 1: I see. Yeah, I think it would be even better if we can get to a stage where they...", + "segments": [ + { + "text": "I see. Yeah, I think it would be even better if we can get to a stage where they...", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_326.wav": { + "chunk_id": "327", + "transcript": "[2025-06-01 19:13:25] Speaker 1: like have access to historical results and they can just see things that are\n[2025-06-01 19:13:28] Speaker 1: ZeeBrand.", + "segments": [ + { + "text": "like have access to historical results and they can just see things that are", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "ZeeBrand.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_327.wav": { + "chunk_id": "328", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Yeah.", + "segments": [ + { + "text": "Yeah.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_328.wav": { + "chunk_id": "329", + "transcript": "[2025-06-01 19:13:25] Speaker 1: instead of having to reproduce and copy pasting things.", + "segments": [ + { + "text": "instead of having to reproduce and copy pasting things.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_329.wav": { + "chunk_id": "330", + "transcript": "[2025-06-01 19:13:25] Speaker 1: i fully agree i think that's our goal but we are not a oci service yet", + "segments": [ + { + "text": "i fully agree i think that's our goal but we are not a oci service yet", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_330.wav": { + "chunk_id": "331", + "transcript": "[2025-06-01 19:13:25] Speaker 5: right and uh because it's all running the customer's tenancy uh there's obviously", + "segments": [ + { + "text": "right and uh because it's all running the customer's tenancy uh there's obviously", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_331.wav": { + "chunk_id": "332", + "transcript": "[2025-06-01 19:13:25] Speaker 5: little bit of things about what we can collect, what we cannot. And we had to go through those.", + "segments": [ + { + "text": "little bit of things about what we can collect, what we cannot. And we had to go through those.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_332.wav": { + "chunk_id": "333", + "transcript": "[2025-06-01 19:13:25] Speaker 5: product approval phases.\n[2025-06-01 19:13:28] Speaker 5: And then we will probably\n[2025-06-01 19:13:29] Speaker 5: support and have access to this.", + "segments": [ + { + "text": "product approval phases.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "And then we will probably", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + }, + { + "text": "support and have access to this.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_333.wav": { + "chunk_id": "334", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Okay, makes sense. One other question for Prometheus. You mentioned at some point in the", + "segments": [ + { + "text": "Okay, makes sense. One other question for Prometheus. You mentioned at some point in the", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_334.wav": { + "chunk_id": "335", + "transcript": "[2025-06-01 19:13:25] Speaker 1: this slide something about setting up a CPU cluster or gate cluster to run Prometheus.", + "segments": [ + { + "text": "this slide something about setting up a CPU cluster or gate cluster to run Prometheus.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_335.wav": { + "chunk_id": "336", + "transcript": "[2025-06-01 19:13:25] Speaker 1: can we reuse our existing primukes instead?", + "segments": [ + { + "text": "can we reuse our existing primukes instead?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_336.wav": { + "chunk_id": "337", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Yes, yes, absolutely. You can use your existing. You need to just enable push gateway and add.", + "segments": [ + { + "text": "Yes, yes, absolutely. You can use your existing. You need to just enable push gateway and add.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_337.wav": { + "chunk_id": "338", + "transcript": "[2025-06-01 19:13:25] Speaker 5: push gateway configurations because that's how all of the scripts push their results and metrics.", + "segments": [ + { + "text": "push gateway configurations because that's how all of the scripts push their results and metrics.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_338.wav": { + "chunk_id": "339", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Okay, can you give a bit more detail about how this works?", + "segments": [ + { + "text": "Okay, can you give a bit more detail about how this works?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_339.wav": { + "chunk_id": "340", + "transcript": "[2025-06-01 19:13:25] Speaker 1: scripts push metrics or how does that work", + "segments": [ + { + "text": "scripts push metrics or how does that work", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_340.wav": { + "chunk_id": "341", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Yeah, Hrithika, can you elaborate on this?", + "segments": [ + { + "text": "Yeah, Hrithika, can you elaborate on this?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_342.wav": { + "chunk_id": "343", + "transcript": "[2025-06-01 19:13:25] Speaker 2: Hello? Can you hear me?", + "segments": [ + { + "text": "Hello? Can you hear me?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 2" + } + ], + "language": "en" + }, + "chunk_341.wav": { + "chunk_id": "342", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Is VTK wrong?", + "segments": [ + { + "text": "Is VTK wrong?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_343.wav": { + "chunk_id": "344", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Yeah.", + "segments": [ + { + "text": "Yeah.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_344.wav": { + "chunk_id": "345", + "transcript": "[2025-06-01 19:13:25] Speaker 2: All right. Could you give that question once again?", + "segments": [ + { + "text": "All right. Could you give that question once again?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 2" + } + ], + "language": "en" + }, + "chunk_345.wav": { + "chunk_id": "346", + "transcript": "[2025-06-01 19:13:25] Speaker 1: It was just having a bit more detail about how the metrics get emitted.", + "segments": [ + { + "text": "It was just having a bit more detail about how the metrics get emitted.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_346.wav": { + "chunk_id": "347", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Because I understand there are scripts and then I'm just trying to figure out like how we could.", + "segments": [ + { + "text": "Because I understand there are scripts and then I'm just trying to figure out like how we could.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_347.wav": { + "chunk_id": "348", + "transcript": "[2025-06-01 19:13:25] Speaker 1: scrape our own metrics.", + "segments": [ + { + "text": "scrape our own metrics.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_348.wav": { + "chunk_id": "349", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Oh yeah, so we basically build the node exporter, DCGM, AMD exporter, all of that depending on the GPU.", + "segments": [ + { + "text": "Oh yeah, so we basically build the node exporter, DCGM, AMD exporter, all of that depending on the GPU.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_349.wav": { + "chunk_id": "350", + "transcript": "[2025-06-01 19:13:25] Speaker 2: it is and bring those up and there are also some metrics which are OCI specific which are", + "segments": [ + { + "text": "it is and bring those up and there are also some metrics which are OCI specific which are", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 2" + } + ], + "language": "en" + }, + "chunk_350.wav": { + "chunk_id": "351", + "transcript": "[2025-06-01 19:13:25] Speaker 2: return in a Go plugin and that plugin\n[2025-06-01 19:13:29] Speaker 2: fits metrics to the push gateway at a regular interval.", + "segments": [ + { + "text": "return in a Go plugin and that plugin", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 2" + }, + { + "text": "fits metrics to the push gateway at a regular interval.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 2" + } + ], + "language": "en" + }, + "chunk_351.wav": { + "chunk_id": "352", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Yeah, all of the scripts here are...\n[2025-06-01 19:13:27] Speaker 5: Does that make sense?", + "segments": [ + { + "text": "Yeah, all of the scripts here are...", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Does that make sense?", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_352.wav": { + "chunk_id": "353", + "transcript": "[2025-06-01 19:13:25] Speaker 5: push gateway if you see it here and", + "segments": [ + { + "text": "push gateway if you see it here and", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_353.wav": { + "chunk_id": "354", + "transcript": "[2025-06-01 19:13:25] Speaker 5: And these are scraped by Prometheus\n[2025-06-01 19:13:27] Speaker 5: based on Prometheus script config.\n[2025-06-01 19:13:29] Speaker 5: And-", + "segments": [ + { + "text": "And these are scraped by Prometheus", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "based on Prometheus script config.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "And-", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_354.wav": { + "chunk_id": "355", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Push gateway is part of a service that is default in Prometheus server install.\n[2025-06-01 19:13:30] Speaker 5: And it's the same.", + "segments": [ + { + "text": "Push gateway is part of a service that is default in Prometheus server install.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "And it's the same.", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_355.wav": { + "chunk_id": "356", + "transcript": "[2025-06-01 19:13:25] Speaker 5: mechanism.\n[2025-06-01 19:13:27] Speaker 5: Gotcha.\n[2025-06-01 19:13:28] Speaker 1: Okay.\n[2025-06-01 19:13:28] Speaker 1: I'll look into this a bit more.\n[2025-06-01 19:13:29] Speaker 1: I haven't used push gateway before.", + "segments": [ + { + "text": "mechanism.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Gotcha.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "Okay.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 1" + }, + { + "text": "I'll look into this a bit more.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 1" + }, + { + "text": "I haven't used push gateway before.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_356.wav": { + "chunk_id": "357", + "transcript": "[2025-06-01 19:13:25] Speaker 1: So that's probably where my gap comes from.\n[2025-06-01 19:13:27] Speaker 1: Okay.\n[2025-06-01 19:13:27] Speaker 5: Yeah, I think it's part of the setup.", + "segments": [ + { + "text": "So that's probably where my gap comes from.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "Okay.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 1" + }, + { + "text": "Yeah, I think it's part of the setup.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_357.wav": { + "chunk_id": "358", + "transcript": "[2025-06-01 19:13:25] Speaker 5: And if you go, if you run Prometheus through operator,\n[2025-06-01 19:13:28] Speaker 5: Push Gateway is usually installed as a service.", + "segments": [ + { + "text": "And if you go, if you run Prometheus through operator,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Push Gateway is usually installed as a service.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_358.wav": { + "chunk_id": "359", + "transcript": "[2025-06-01 19:13:25] Speaker 5: It should have its own service endpoint,\n[2025-06-01 19:13:28] Speaker 5: and you have to just enable a few things.", + "segments": [ + { + "text": "It should have its own service endpoint,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "and you have to just enable a few things.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_359.wav": { + "chunk_id": "360", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and you're good to go. Yeah. So, this is for instead of you going and reaching out.", + "segments": [ + { + "text": "and you're good to go. Yeah. So, this is for instead of you going and reaching out.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_360.wav": { + "chunk_id": "361", + "transcript": "[2025-06-01 19:13:25] Speaker 5: going and reaching out and pulling the metrics out which is where you need firewall exception", + "segments": [ + { + "text": "going and reaching out and pulling the metrics out which is where you need firewall exception", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_361.wav": { + "chunk_id": "362", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and opening the ports and all of that.\n[2025-06-01 19:13:27] Speaker 5: Here, any metrics you are generating, including DCGM,", + "segments": [ + { + "text": "and opening the ports and all of that.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Here, any metrics you are generating, including DCGM,", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_362.wav": { + "chunk_id": "363", + "transcript": "[2025-06-01 19:13:25] Speaker 5: all of those are pool metrics, right?\n[2025-06-01 19:13:27] Speaker 5: We push it directly to the push page.", + "segments": [ + { + "text": "all of those are pool metrics, right?", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "We push it directly to the push page.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_363.wav": { + "chunk_id": "364", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and your Prometheus is scraping push gateway basically.\n[2025-06-01 19:13:29] Speaker 5: It's like an intermediate data store.", + "segments": [ + { + "text": "and your Prometheus is scraping push gateway basically.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "It's like an intermediate data store.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_364.wav": { + "chunk_id": "365", + "transcript": "[2025-06-01 19:13:25] Speaker 5: All right. Any further questions? And we are also eager if you have any feedback.", + "segments": [ + { + "text": "All right. Any further questions? And we are also eager if you have any feedback.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_365.wav": { + "chunk_id": "366", + "transcript": "[2025-06-01 19:13:25] Speaker 5: can think about we are thinking in next roughly two weeks you can you can have access to all of", + "segments": [ + { + "text": "can think about we are thinking in next roughly two weeks you can you can have access to all of", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_366.wav": { + "chunk_id": "367", + "transcript": "[2025-06-01 19:13:25] Speaker 5: the health check scripts.\n[2025-06-01 19:13:27] Speaker 5: We'll get access to your repo\n[2025-06-01 19:13:29] Speaker 5: once you share your GitHub IDs.", + "segments": [ + { + "text": "the health check scripts.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "We'll get access to your repo", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "once you share your GitHub IDs.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_367.wav": { + "chunk_id": "368", + "transcript": "[2025-06-01 19:13:25] Speaker 5: can poke around and really we are here to listen and and build something that that makes uh", + "segments": [ + { + "text": "can poke around and really we are here to listen and and build something that that makes uh", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_368.wav": { + "chunk_id": "369", + "transcript": "[2025-06-01 19:13:25] Speaker 5: life easy right so we are here to listen and you know continue to iterate as fast as we", + "segments": [ + { + "text": "life easy right so we are here to listen and you know continue to iterate as fast as we", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_369.wav": { + "chunk_id": "370", + "transcript": "[2025-06-01 19:13:25] Speaker 5: can to give you what you're looking for.\n[2025-06-01 19:13:27] Speaker 5: So.\n[2025-06-01 19:13:28] Speaker 1: Thanks, Samar.\n[2025-06-01 19:13:29] Speaker 1: I think overall this is, you know.", + "segments": [ + { + "text": "can to give you what you're looking for.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "Thanks, Samar.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 1" + }, + { + "text": "I think overall this is, you know.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_370.wav": { + "chunk_id": "371", + "transcript": "[2025-06-01 19:13:25] Speaker 1: something that we've been asking for. So really, really happy it's, it's getting there.", + "segments": [ + { + "text": "something that we've been asking for. So really, really happy it's, it's getting there.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_371.wav": { + "chunk_id": "372", + "transcript": "[2025-06-01 19:13:25] Speaker 1: and you all are focusing your efforts there.\n[2025-06-01 19:13:26] Speaker 1: It's definitely heading the right direction.", + "segments": [ + { + "text": "and you all are focusing your efforts there.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "It's definitely heading the right direction.", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_372.wav": { + "chunk_id": "373", + "transcript": "[2025-06-01 19:13:25] Speaker 1: I think looking at what we have today that we built ourselves a best over the past year,\n[2025-06-01 19:13:29] Speaker 1: it's definitely...", + "segments": [ + { + "text": "I think looking at what we have today that we built ourselves a best over the past year,", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "it's definitely...", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_373.wav": { + "chunk_id": "374", + "transcript": "[2025-06-01 19:13:25] Speaker 1: not there yet. I think what we have right now is a bit more advanced just because we've built", + "segments": [ + { + "text": "not there yet. I think what we have right now is a bit more advanced just because we've built", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_374.wav": { + "chunk_id": "375", + "transcript": "[2025-06-01 19:13:25] Speaker 1: our own set of scripts and like we've iterated on them and we have all that automation.", + "segments": [ + { + "text": "our own set of scripts and like we've iterated on them and we have all that automation.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_375.wav": { + "chunk_id": "376", + "transcript": "[2025-06-01 19:13:25] Speaker 1: and integration with Kubernetes already.\n[2025-06-01 19:13:27] Speaker 1: But I like where this is going and yeah, excited too.", + "segments": [ + { + "text": "and integration with Kubernetes already.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "But I like where this is going and yeah, excited too.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_376.wav": { + "chunk_id": "377", + "transcript": "[2025-06-01 19:13:25] Speaker 1: work with you all too.\n[2025-06-01 19:13:26] Speaker 1: It's great.\n[2025-06-01 19:13:27] Speaker 5: Okay.\n[2025-06-01 19:13:28] Speaker 5: Yeah.\n[2025-06-01 19:13:28] Speaker 5: We'll try to,\n[2025-06-01 19:13:30] Speaker 5: right now,\n[2025-06-01 19:13:31] Speaker 5: we are not overly", + "segments": [ + { + "text": "work with you all too.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "It's great.", + "timestamp": "2025-06-01 19:13:26", + "speaker": "Speaker 1" + }, + { + "text": "Okay.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + }, + { + "text": "Yeah.", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + }, + { + "text": "We'll try to,", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + }, + { + "text": "right now,", + "timestamp": "2025-06-01 19:13:30", + "speaker": "Speaker 5" + }, + { + "text": "we are not overly", + "timestamp": "2025-06-01 19:13:31", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_377.wav": { + "chunk_id": "378", + "transcript": "[2025-06-01 19:13:25] Speaker 5: Kubernetes heavy features and if we will probably in the next iteration or a milestone", + "segments": [ + { + "text": "Kubernetes heavy features and if we will probably in the next iteration or a milestone", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_378.wav": { + "chunk_id": "379", + "transcript": "[2025-06-01 19:13:25] Speaker 5: add all of those capabilities to bring Kubernetes metrics and that experience together.", + "segments": [ + { + "text": "add all of those capabilities to bring Kubernetes metrics and that experience together.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_379.wav": { + "chunk_id": "380", + "transcript": "[2025-06-01 19:13:25] Speaker 5: And hopefully, Sisal, I'll post that.\n[2025-06-01 19:13:27] Speaker 5: You would be open to trying it out and giving us the feedback.", + "segments": [ + { + "text": "And hopefully, Sisal, I'll post that.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "You would be open to trying it out and giving us the feedback.", + "timestamp": "2025-06-01 19:13:27", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_380.wav": { + "chunk_id": "381", + "transcript": "[2025-06-01 19:13:25] Speaker 5: and iterating like that.\n[2025-06-01 19:13:29] Speaker 1: Sounds good.\n[2025-06-01 19:13:32] Speaker 5: Thanks for making the time to give us a demo.", + "segments": [ + { + "text": "and iterating like that.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "Sounds good.", + "timestamp": "2025-06-01 19:13:29", + "speaker": "Speaker 1" + }, + { + "text": "Thanks for making the time to give us a demo.", + "timestamp": "2025-06-01 19:13:32", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_381.wav": { + "chunk_id": "382", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Appreciate it.\n[2025-06-01 19:13:25] Speaker 5: Thanks for spending your time with us as well.\n[2025-06-01 19:13:28] Speaker 5: So anyone else, any questions?", + "segments": [ + { + "text": "Appreciate it.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + }, + { + "text": "Thanks for spending your time with us as well.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 5" + }, + { + "text": "So anyone else, any questions?", + "timestamp": "2025-06-01 19:13:28", + "speaker": "Speaker 5" + } + ], + "language": "en" + }, + "chunk_382.wav": { + "chunk_id": "383", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Thank you.", + "segments": [ + { + "text": "Thank you.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_383.wav": { + "chunk_id": "384", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Thank you.", + "segments": [ + { + "text": "Thank you.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_384.wav": { + "chunk_id": "385", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Thank you.", + "segments": [ + { + "text": "Thank you.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + }, + "chunk_385.wav": { + "chunk_id": "386", + "transcript": "[2025-06-01 19:13:25] Speaker 1: Thank you.", + "segments": [ + { + "text": "Thank you.", + "timestamp": "2025-06-01 19:13:25", + "speaker": "Speaker 1" + } + ], + "language": "en" + } + }, + "transcript": "[2025-06-01 19:13:25] Speaker 5: All right. Thank you all. Morning, afternoon. So my name is Amar Gowda. I am one of the\n[2025-06-01 19:13:30] Speaker 5: products.\n[2025-06-01 19:13:25] Speaker 5: managers leading the OCI Lens initiative. So right now it's kind of what we call incubation.\n[2025-06-01 19:13:25] Speaker 5: phases so we are not yet to be a mvp or closer to it so we're kind of getting early feedback\n[2025-06-01 19:13:25] Speaker 5: and this is like think of this as a kind of private preview in a way right so a little earlier on\n[2025-06-01 19:13:25] Speaker 5: on the journey.\n[2025-06-01 19:13:26] Speaker 5: But this is something we started looking into,\n[2025-06-01 19:13:31] Speaker 5: which we, I think we spoke with.\n[2025-06-01 19:13:25] Speaker 5: you, Cecil, you and the rest of the team on what does it take to operate at scale?\n[2025-06-01 19:13:25] Speaker 5: what are the things that we could improve on, especially the monitoring, the health check.\n[2025-06-01 19:13:25] Speaker 5: side of the things, which has, you know, you gave a set of requirements and that led us.\n[2025-06-01 19:13:25] Speaker 5: to building something to help customers like you.\n[2025-06-01 19:13:30] Speaker 5: Okay, so I have a team here.\n[2025-06-01 19:13:32] Speaker 5: Joletta is...\n[2025-06-01 19:13:25] Speaker 5: another software engineers somia on the call is our data scientist selection ml engineer\n[2025-06-01 19:13:30] Speaker 5: uh\n[2025-06-01 19:13:25] Speaker 5: also have uh hithika who's our golang developer she's ex uh akis team so that's where if you all can\n[2025-06-01 19:13:25] Speaker 5: connect a little bit too.\n[2025-06-01 19:13:27] Speaker 5: All right. Today, I'm going to keep it more focused on the demo.\n[2025-06-01 19:13:25] Speaker 5: and what this tool is all about.\n[2025-06-01 19:13:27] Speaker 5: And then we'll, you know, answer questions\n[2025-06-01 19:13:29] Speaker 5: and kind of talk through.\n[2025-06-01 19:13:25] Speaker 5: But most importantly, we want to hear if the current version has a set of features you're\n[2025-06-01 19:13:25] Speaker 5: looking for? Is it missing? Is it you want to see something more? Because we are kind of very\n[2025-06-01 19:13:25] Speaker 5: rapidly on a weekly sprint uh releasing the features to this and we are really happy to incorporate\n[2025-06-01 19:13:25] Speaker 5: anything and work on how you can onboard to this by the way\n[2025-06-01 19:13:31] Speaker 5: uh so four areas where we are focused on\n[2025-06-01 19:13:25] Speaker 5: this solution is continuous GPU and cluster level monitoring of both NVIDIA GPUs and AMD GPUs.\n[2025-06-01 19:13:25] Speaker 5: together, right? So this is how we want to kind of platform\n[2025-06-01 19:13:29] Speaker 5: agnostic solution that works for all GPUs.\n[2025-06-01 19:13:25] Speaker 5: and has all the latest metrics available for you to digest and look at.\n[2025-06-01 19:13:32] Speaker 5: The next thing which we've highlighted...\n[2025-06-01 19:13:25] Speaker 5: focused on is the health checks piece to this right are my nodes healthy are they performing to\n[2025-06-01 19:13:30] Speaker 5: what they\n[2025-06-01 19:13:25] Speaker 5: should be uh active health checks active monitoring is is is a huge investment that we did uh and i'll\n[2025-06-01 19:13:32] Speaker 5: go over some\n[2025-06-01 19:13:25] Speaker 5: details about what all health checks that we have enabled including the RDMA cluster level.\n[2025-06-01 19:13:25] Speaker 5: checks that we call MPI tests as part of the solution fully baked into.\n[2025-06-01 19:13:30] Speaker 5: We kept it to cloud native.\n[2025-06-01 19:13:25] Speaker 5: because I heard it very clear that it has to be Grafana, Prometheus, Native, or Otel, anything open source.\n[2025-06-01 19:13:25] Speaker 5: Cloud Native Alliance so that it's easy to be move between clouds or on-prem and use\n[2025-06-01 19:13:29] Speaker 5: the same existing.\n[2025-06-01 19:13:25] Speaker 5: tools you always used for for monitoring another area that we've paid attention and we built is the\n[2025-06-01 19:13:25] Speaker 5: team level tracking. A lot of these large size clusters are usually used by multiple teams and\n[2025-06-01 19:13:25] Speaker 5: everybody wants to know the health of their own subset of systems or how their experiment is\n[2025-06-01 19:13:30] Speaker 5: performing.\n[2025-06-01 19:13:25] Speaker 5: if they have an unhealthy node, all of those combinations.\n[2025-06-01 19:13:29] Speaker 5: And another important thing is about\n[2025-06-01 19:13:25] Speaker 5: cost tracking right people want to see how much computer resources they've used uh whatever the\n[2025-06-01 19:13:25] Speaker 5: the power the total gpu power they have used for running this experiment right those were a few\n[2025-06-01 19:13:29] Speaker 5: things that\n[2025-06-01 19:13:25] Speaker 5: that we also focused to build on this.\n[2025-06-01 19:13:29] Speaker 5: The current set of features that we have\n[2025-06-01 19:13:32] Speaker 5: and what we have built right now\n[2025-06-01 19:13:25] Speaker 5: now is first thing tenancy level monitoring you no longer have region barriers right so we any\n[2025-06-01 19:13:25] Speaker 5: data, any region, OCI region you may be in, everything is monitorable as a single instance\n[2025-06-01 19:13:31] Speaker 5: for us.\n[2025-06-01 19:13:25] Speaker 5: So I'm going to show you a demo.\n[2025-06-01 19:13:27] Speaker 5: We allow you to monitor either single, bare metal, virtual machine\n[2025-06-01 19:13:25] Speaker 5: instances or a full oke cluster or an hpc cluster that if you may be using slurm or a native uh\n[2025-06-01 19:13:25] Speaker 5: called cluster network, compute cluster setup in OCI.\n[2025-06-01 19:13:29] Speaker 5: The third part of the feature that we worked on\n[2025-06-01 19:13:25] Speaker 5: and all of this we're going to see a demo quickly but i'm going to just spend maybe five more minutes\n[2025-06-01 19:13:25] Speaker 5: and then go to the demo.\n[2025-06-01 19:13:26] Speaker 5: Is it team level tracking?\n[2025-06-01 19:13:27] Speaker 5: You can create team level tracking.\n[2025-06-01 19:13:25] Speaker 5: handpicking whichever nodes are part of this experiment.\n[2025-06-01 19:13:29] Speaker 5: We are also working towards...\n[2025-06-01 19:13:25] Speaker 5: automatically fetching the nodes that are running experiment based on Kubernetes\n[2025-06-01 19:13:29] Speaker 5: how the job system.\n[2025-06-01 19:13:25] Speaker 5: scheduled. So that's another area we're working on automatically, dynamically pulling the nodes that\n[2025-06-01 19:13:25] Speaker 5: part of that experiment rather than you allocating what compute nodes go to.\n[2025-06-01 19:13:31] Speaker 5: This I've already said.\n[2025-06-01 19:13:25] Speaker 5: already have both of them running and i'm going to show you that uh the difference between what we do\n[2025-06-01 19:13:25] Speaker 5: with performance monitoring and health check is we go very close to the layers that an ML engineer would\n[2025-06-01 19:13:25] Speaker 5: operate under, which is PyTorch for us, PyTorch and JAX primarily, right? So we picked up PyTorch base.\n[2025-06-01 19:13:25] Speaker 5: matmal and linear regression based performance testing to achieve how many flops did we achieve\n[2025-06-01 19:13:25] Speaker 5: on this compute node how much is each gpu performing and is it is it within the threshold of\n[2025-06-01 19:13:25] Speaker 5: the specification from NVIDIA or AMD what we have traditionally seen. So we have a baseline.\n[2025-06-01 19:13:25] Speaker 5: that we score the performance to because it's the standard set of precision based testing either FBA.\n[2025-06-01 19:13:25] Speaker 5: FP16, FP32, it automatically does all of this. And most of this code, by the way, for the health\n[2025-06-01 19:13:25] Speaker 5: will all be open source so you can see you can change the things if you like or add to it\n[2025-06-01 19:13:25] Speaker 5: right and please feel free to contribute to it too.\n[2025-06-01 19:13:28] Speaker 5: This you've seen it.\n[2025-06-01 19:13:30] Speaker 5: The approach we have taken.\n[2025-06-01 19:13:25] Speaker 5: here is instead of asking you to poke holes in your existing cluster or networks, we basically push the\n[2025-06-01 19:13:25] Speaker 5: metrics and we push the health check data to the Prometheus and our central control plane.\n[2025-06-01 19:13:25] Speaker 5: It could be looked at as, oh, I need to open this port, that port, no exception.\n[2025-06-01 19:13:25] Speaker 5: if you're in a vc and in a production environment where everything is locked down egress is usually\n[2025-06-01 19:13:25] Speaker 5: easier and you can always sniff the egress but these are all running within your own tenancy so\n[2025-06-01 19:13:25] Speaker 5: So the origination of all of the metrics to where it's being sent is all within your network.\n[2025-06-01 19:13:25] Speaker 5: nothing is going into the public. All right. So I'm going to go over this.\n[2025-06-01 19:13:25] Speaker 5: but there are plenty of metrics, just a lot.\n[2025-06-01 19:13:28] Speaker 5: And it's kind of an eye candy chart on Grafana's to look.\n[2025-06-01 19:13:25] Speaker 5: at but i think you get the point right you have full nvidia dcgm exporter all metrics\n[2025-06-01 19:13:25] Speaker 5: You have all AMD SMI metrics.\n[2025-06-01 19:13:27] Speaker 5: You have all of the RDMA metrics that is particularly on our ROC.\n[2025-06-01 19:13:25] Speaker 5: implementation that we sniffed the melanocs drivers for to capture that front end next\n[2025-06-01 19:13:25] Speaker 5: node health is included, health check, full check is included,\n[2025-06-01 19:13:29] Speaker 5: and the traditional disk IO usage.\n[2025-06-01 19:13:25] Speaker 5: all of those that you usually get with Prometheus already included.\n[2025-06-01 19:13:28] Speaker 5: So all of these is bundled as one.\n[2025-06-01 19:13:25] Speaker 5: unpacking.\n[2025-06-01 19:13:25] Speaker 5: On the health checks piece, these are all the tests we do today and we'll continue to add more or adjust.\n[2025-06-01 19:13:25] Speaker 5: based on how performance we see we do extensive benchmarking to see where the threshold should\n[2025-06-01 19:13:25] Speaker 5: be or the baseline should be and then we either go up and down based on how we are seeing.\n[2025-06-01 19:13:25] Speaker 5: example the what should be the ideal mfu on a 140 gb against b200 or mi300x right and then we we lower it\n[2025-06-01 19:13:25] Speaker 5: if needed right so but we work with our vendors to also see if this we already doing the right way or not\n[2025-06-01 19:13:25] Speaker 5: So here are all the checks we run.\n[2025-06-01 19:13:27] Speaker 5: And all of these, once they complete, you will see a...\n[2025-06-01 19:13:25] Speaker 5: metrics flowing through Grafana and you'll also see this.\n[2025-06-01 19:13:25] Speaker 5: Very simple. This is how the architecture is. I'm not going to spend too much time here either.\n[2025-06-01 19:13:25] Speaker 5: because we are evolving how this is going to go right now the way we will release it is this will\n[2025-06-01 19:13:25] Speaker 5: deploy as a we're not a kubernetes operator yet but you have to basically deploy a dedicated\n[2025-06-01 19:13:25] Speaker 5: OKE cluster with just CPU nodes, which installs Prometheus Grafana open source and our control.\n[2025-06-01 19:13:25] Speaker 5: plane api as well as portal right those are all the components that come as part of this\n[2025-06-01 19:13:25] Speaker 5: And this is that footprint.\n[2025-06-01 19:13:27] Speaker 5: We run a local Postgres for Profana storage.\n[2025-06-01 19:13:30] Speaker 5: We also use Postgres for...\n[2025-06-01 19:13:25] Speaker 5: kind of operating these things.\n[2025-06-01 19:13:27] Speaker 5: This is our kind of MVP minus one approach, right?\n[2025-06-01 19:13:32] Speaker 5: And once this...\n[2025-06-01 19:13:25] Speaker 5: becomes a service this will start looking differently any questions so far\n[2025-06-01 19:13:25] Speaker 1: I have a few questions.\n[2025-06-01 19:13:26] Speaker 1: Do you prefer we wait till the end?\n[2025-06-01 19:13:25] Speaker 5: Yeah, let's take it, Zizal.\n[2025-06-01 19:13:26] Speaker 5: Or if you want to see the demo and then ask those questions, it's up to you.\n[2025-06-01 19:13:25] Speaker 1: I guess let's do the demo and then we can come back to questions.\n[2025-06-01 19:13:28] Speaker 1: Maybe that will answer some of it.\n[2025-06-01 19:13:25] Speaker 1: questions perfect yep all right so the the way we we know you can install this is we you will\n[2025-06-01 19:13:25] Speaker 5: have a GitHub repo where it could go and say deploy this and it sets up a Kubernetes cluster.\n[2025-06-01 19:13:25] Speaker 5: Kubernetes cluster setup and all of the Grafana Prometeas,\n[2025-06-01 19:13:29] Speaker 5: everything is installed.\n[2025-06-01 19:13:25] Speaker 5: in your primary OSID or your tenancy region,\n[2025-06-01 19:13:29] Speaker 5: you will get access to a portal like this.\n[2025-06-01 19:13:25] Speaker 5: right we call it corino lens but it is basically the lens so once you are here\n[2025-06-01 19:13:25] Speaker 5: right it will go it has the ability to do a multi-region query in a single api\n[2025-06-01 19:13:25] Speaker 5: So everything is API first, you have portal and there is a REST API endpoint you can interact.\n[2025-06-01 19:13:25] Speaker 5: all of that is deployed in your tenancy so you can lock it down if you like to\n[2025-06-01 19:13:25] Speaker 5: Right now we run on a public domain just for a demo, but everything in here.\n[2025-06-01 19:13:30] Speaker 5: So if you see here.\n[2025-06-01 19:13:25] Speaker 5: This is done. If I refresh this page, it's going to come back with\n[2025-06-01 19:13:25] Speaker 5: It's doing all the regions that you have subscribed in your OSIT or Oracle tenancy.\n[2025-06-01 19:13:25] Speaker 5: Then it comes back and tells you, here are all the instances that I can monitor.\n[2025-06-01 19:13:28] Speaker 5: These are basically...\n[2025-06-01 19:13:25] Speaker 5: just GPU instances that are running okay if you have new instances running you can also add\n[2025-06-01 19:13:25] Speaker 5: what we call kind of plug-in.\n[2025-06-01 19:13:27] Speaker 5: So it's a plug-in model.\n[2025-06-01 19:13:28] Speaker 5: You already say, okay, start monitoring.\n[2025-06-01 19:13:25] Speaker 5: these to OCI lengths. And so if you look at here, I have Frankfurt and Ashburn machines showing.\n[2025-06-01 19:13:25] Speaker 5: up and it allows me to combine all of this and say they're all part of the same team or same\n[2025-06-01 19:13:25] Speaker 5: experiment or same way i want to monitor and i can create what's called monitoring right\n[2025-06-01 19:13:25] Speaker 5: So before I go there, right, how can you start monitoring this?\n[2025-06-01 19:13:29] Speaker 5: If you have an existing instance...\n[2025-06-01 19:13:25] Speaker 5: running or a new instance being provisioned you have to just include this script for now right\n[2025-06-01 19:13:25] Speaker 5: we will we will change the way it is we're going to make it a native oci plugin\n[2025-06-01 19:13:25] Speaker 5: once we upstream all of this to our plugin framework.\n[2025-06-01 19:13:27] Speaker 5: But for now, it's basically a tarz file.\n[2025-06-01 19:13:25] Speaker 5: and it's a Golang-based code primarily.\n[2025-06-01 19:13:28] Speaker 5: The health checks are all Python-based.\n[2025-06-01 19:13:25] Speaker 5: And this is basically end of the day, just all these metrics are composed and pushed.\n[2025-06-01 19:13:25] Speaker 5: through Prometheus Push gateway,\n[2025-06-01 19:13:27] Speaker 5: which is getting scraped by Push.\n[2025-06-01 19:13:28] Speaker 5: So architecture.\n[2025-06-01 19:13:25] Speaker 5: this is nothing very different or rocket science here right so it's very simple\n[2025-06-01 19:13:25] Speaker 5: straightforward, but what it runs and how it runs is what the value we are adding.\n[2025-06-01 19:13:25] Speaker 5: This is like a cloud in its script as well.\n[2025-06-01 19:13:27] Speaker 5: You can add it as part of a new provisioning.\n[2025-06-01 19:13:25] Speaker 5: instance automatically every new instance that comes up has monitoring enabled and everything is\n[2025-06-01 19:13:25] Speaker 5: running at the host level as a system service it's not operating at the kubernetes layers or at the\n[2025-06-01 19:13:25] Speaker 5: the Slurm application layers.\n[2025-06-01 19:13:27] Speaker 5: It's running directly on the bare metal host.\n[2025-06-01 19:13:30] Speaker 5: And we felt that was.\n[2025-06-01 19:13:25] Speaker 5: the ideal way and the right place for us to remove specific implementations to Kubernetes.\n[2025-06-01 19:13:25] Speaker 5: or Slurm or others.\n[2025-06-01 19:13:28] Speaker 5: So we can also scan Kubernetes clusters you have running\n[2025-06-01 19:13:31] Speaker 5: or you have RDMA.\n[2025-06-01 19:13:25] Speaker 5: cluster networks or what we call compute clusters and allow you to also monitor that.\n[2025-06-01 19:13:25] Speaker 5: And when you check the whole Kubernetes cluster, all the GPU nodes under it are automatically\n[2025-06-01 19:13:30] Speaker 5: scanned.\n[2025-06-01 19:13:25] Speaker 5: and add it to the monitoring for us.\n[2025-06-01 19:13:27] Speaker 5: That's the experience.\n[2025-06-01 19:13:30] Speaker 5: Okay.\n[2025-06-01 19:13:30] Speaker 5: So I'll quickly show you how...\n[2025-06-01 19:13:25] Speaker 5: you know this this is basically a bare minimum um portal uh access we have everything is api\n[2025-06-01 19:13:25] Speaker 5: first so you get exactly what you're doing here you could you could do that uh through\n[2025-06-01 19:13:25] Speaker 5: REST API endpoints. So once this loads up, because it's doing like a close to 10 region query,\n[2025-06-01 19:13:25] Speaker 5: takes roughly six seconds but six to ten seconds we will we'll make it better with some cash\n[2025-06-01 19:13:25] Speaker 5: So, yeah, no, we have close to 30, 30 plus machines, right, Joleta?\n[2025-06-01 19:13:33] Speaker 5: I think it's performing.\n[2025-06-01 19:13:25] Speaker 5: okay but you know joletta is working hard to see how she can do parallel um parallelization\n[2025-06-01 19:13:25] Speaker 5: of some of these runs and get back.\n[2025-06-01 19:13:27] Speaker 5: But we got them better.\n[2025-06-01 19:13:29] Speaker 5: We started from 30 seconds.\n[2025-06-01 19:13:25] Speaker 5: by now at six seconds, but we'll keep pushing.\n[2025-06-01 19:13:29] Speaker 5: All right. So it allows me to create a...\n[2025-06-01 19:13:25] Speaker 5: monitoring ring, it's like just an arbitrary way for you to\n[2025-06-01 19:13:28] Speaker 5: combine these resources and say\n[2025-06-01 19:13:25] Speaker 5: I want a dedicated dashboard for this,\n[2025-06-01 19:13:27] Speaker 5: these set of machines, right?\n[2025-06-01 19:13:29] Speaker 5: That's what you.\n[2025-06-01 19:13:25] Speaker 5: allows you to create a ring and if you go to monitoring rings here and you have OK,\n[2025-06-01 19:13:30] Speaker 5: this is.\n[2025-06-01 19:13:25] Speaker 5: for ML training, this is for the team, this is for cost center, whatever, however you\n[2025-06-01 19:13:25] Speaker 5: want to use this for kind of bundling all the things. Right? And the thing about\n[2025-06-01 19:13:25] Speaker 5: this is you can always come back and add more instances or remove the instances if they go\n[2025-06-01 19:13:25] Speaker 5: offline or you have to turn back in whatever right so so every every ring you create monitor\n[2025-06-01 19:13:25] Speaker 5: ring you create comes with a dedicated Grafana board which includes all the hosts that are\n[2025-06-01 19:13:25] Speaker 5: part of this and I'm going to click you through the Grafana.\n[2025-06-01 19:13:31] Speaker 5: So what you see here at the first part of\n[2025-06-01 19:13:25] Speaker 5: this is the health summary of all the compute nodes that are part of this, right? So we added more nodes.\n[2025-06-01 19:13:25] Speaker 5: which you're in this part just to show you demo, right?\n[2025-06-01 19:13:27] Speaker 5: So if you see the first board here is this is the.\n[2025-06-01 19:13:25] Speaker 5: actual health check of all the performance related checks that we did on the host when you activate\n[2025-06-01 19:13:25] Speaker 5: the plugin we will also allow you to run these on demand so if you look at the categories here\n[2025-06-01 19:13:25] Speaker 5: compute throughput, temperature check, all these check, check, check,\n[2025-06-01 19:13:28] Speaker 5: model MFU based stuff.\n[2025-06-01 19:13:25] Speaker 5: All of those are readily available for you and we will very soon have a link that will exactly take\n[2025-06-01 19:13:25] Speaker 5: you to a JSON that is very much deeper on all the tests. This is basically the JSON that the health check.\n[2025-06-01 19:13:25] Speaker 5: script create like what's the bandwidth achieved the tflops achieved for each of the node which\n[2025-06-01 19:13:25] Speaker 5: GPU ran this for example right this is the GPU ID of an H100 we ran this is actually the JSON document\n[2025-06-01 19:13:25] Speaker 5: you'll have access to this as well as the logs piece uh but in a grafana it looks\n[2025-06-01 19:13:25] Speaker 5: Now, it's an easier way for you to digest all of this.\n[2025-06-01 19:13:27] Speaker 5: Let me put it up.\n[2025-06-01 19:13:25] Speaker 5: end of the day it's the same data but everything is a very detailed data that you may\n[2025-06-01 19:13:25] Speaker 5: be looking for um than just a summary like this okay both are available okay so we will add more\n[2025-06-01 19:13:25] Speaker 5: filters when it allows you to you know go by region and check the gpu types if you have mi300x\n[2025-06-01 19:13:25] Speaker 5: So Osaka does not have, maybe I think other machines in Chicago have MI300X.\n[2025-06-01 19:13:25] Speaker 5: see here yes you can see my 300x instance um so yeah these are all of both uh my 300x\n[2025-06-01 19:13:25] Speaker 5: as well as GPU from NVidia are fully monitored.\n[2025-06-01 19:13:29] Speaker 5: And we have health checks running for both.\n[2025-06-01 19:13:25] Speaker 5: Okay, so all of this is the data that you may be interested in.\n[2025-06-01 19:13:25] Speaker 5: that allows you to kind of baseline and benchmark all of this. Okay. So all of these metrics are\n[2025-06-01 19:13:25] Speaker 5: natively available in Prometheus and if I have to show you so we push with OCI\n[2025-06-01 19:13:25] Speaker 5: lens labels to this and this will allow you to extend it any further you like.\n[2025-06-01 19:13:25] Speaker 5: custom boards if you like, or you want to merge left join, right join some datasets here to get to more.\n[2025-06-01 19:13:25] Speaker 5: details everything is fully available that's just native krafanov right this is the lcl lens\n[2025-06-01 19:13:25] Speaker 5: health check stuff too. There is quite a bit of\n[2025-06-01 19:13:31] Speaker 5: health. Yeah, this is the health summary.\n[2025-06-01 19:13:25] Speaker 5: So all of these metrics, the only thing that we are adding,\n[2025-06-01 19:13:27] Speaker 5: this comes out as OCI lens.\n[2025-06-01 19:13:25] Speaker 5: anything that comes directly from the vendor,\n[2025-06-01 19:13:26] Speaker 5: like AMD or NVIDIA come as is, right?\n[2025-06-01 19:13:29] Speaker 5: DCGM append it.\n[2025-06-01 19:13:25] Speaker 5: So one additional thing we also did is if you know, let me go back here.\n[2025-06-01 19:13:25] Speaker 5: So this is the overall health board.\n[2025-06-01 19:13:28] Speaker 5: And there's another board that we bring is this is.\n[2025-06-01 19:13:25] Speaker 5: per host data right this is the actual gpu rdma metrics uh you know you know node health and all\n[2025-06-01 19:13:25] Speaker 5: that. Additional things that we are trying to add is if you come here and look at this table.\n[2025-06-01 19:13:25] Speaker 5: We will improve the UI a little bit,\n[2025-06-01 19:13:27] Speaker 5: but there is something called host metadata survey.\n[2025-06-01 19:13:25] Speaker 5: It's basically a local service endpoint that we hit\n[2025-06-01 19:13:28] Speaker 5: and we fetch the serial number because if the-\n[2025-06-01 19:13:25] Speaker 5: host is underperforming and you have to turn back it in for repair.\n[2025-06-01 19:13:29] Speaker 5: You know, some data that we have.\n[2025-06-01 19:13:25] Speaker 5: going to publish here is going to be super helpful.\n[2025-06-01 19:13:27] Speaker 5: And we are looking at how to put an agent.\n[2025-06-01 19:13:25] Speaker 5: AI on top of this, right? Let's\n[2025-06-01 19:13:26] Speaker 5: shortcut that, right? So we\n[2025-06-01 19:13:28] Speaker 5: will instead of throwing more\n[2025-06-01 19:13:25] Speaker 5: more boards, we will kind of curate an experience where if you see a underperforming node or a bad,\n[2025-06-01 19:13:25] Speaker 5: node. We may automate some of that to see if we could automatically raise a ticket or turn the\n[2025-06-01 19:13:25] Speaker 5: machine in or invoke folks in the support team to help you guys out.\n[2025-06-01 19:13:30] Speaker 5: So that's the.\n[2025-06-01 19:13:25] Speaker 5: future roadmap. So yeah, everything here, we're going to add more stuff like what's the\n[2025-06-01 19:13:25] Speaker 5: the huda driver you're running what's the kernel versions you're running um what's the OS\n[2025-06-01 19:13:25] Speaker 5: attached to, what's the Rockham driver version you're running, all of those data will be fully\n[2025-06-01 19:13:25] Speaker 5: available everything related to metadata will be published here and you can continue to\n[2025-06-01 19:13:25] Speaker 5: use that instead of you SSHing into the machine,\n[2025-06-01 19:13:28] Speaker 5: running some commands and all of that.\n[2025-06-01 19:13:25] Speaker 5: So, readily available.\n[2025-06-01 19:13:27] Speaker 5: Everything else you see here is something you may have been using already.\n[2025-06-01 19:13:25] Speaker 5: which is power, usage, temperature, utilization,\n[2025-06-01 19:13:28] Speaker 5: all that stuff.\n[2025-06-01 19:13:30] Speaker 5: Okay.\n[2025-06-01 19:13:31] Speaker 5: So in an actual, this-\n[2025-06-01 19:13:25] Speaker 5: this is this is what it is right so at least our first iteration of um can we get a monitor\n[2025-06-01 19:13:25] Speaker 5: going which is native and health checks going which is what lci published health checks and\n[2025-06-01 19:13:25] Speaker 5: just always vendor specified. So I'm gonna stop here and see if you folks have questions.\n[2025-06-01 19:13:25] Speaker 5: and also talk about some feedback if you have for us.\n[2025-06-01 19:13:32] Speaker 1: Thanks for the demo.\n[2025-06-01 19:13:33] Speaker 1: I have a few...\n[2025-06-01 19:13:25] Speaker 1: questions um so i think you mentioned in the first slide one of the things being like um you know\n[2025-06-01 19:13:25] Speaker 1: Kubernetes native, cloud native,\n[2025-06-01 19:13:26] Speaker 1: which was one of the things we discussed previously\n[2025-06-01 19:13:28] Speaker 1: as being one of the...\n[2025-06-01 19:13:25] Speaker 1: key things really important for us um what other levels of integrations with oke are you planning\n[2025-06-01 19:13:25] Speaker 1: at all, if any.\n[2025-06-01 19:13:26] Speaker 1: Things that top of mind would be super helpful\n[2025-06-01 19:13:28] Speaker 1: for us would be being able to\n[2025-06-01 19:13:25] Speaker 1: surface some of these unhealthy node conditions to OKE or to the actual Kubernetes object.\n[2025-06-01 19:13:25] Speaker 1: so that we can use that in our tooling.\n[2025-06-01 19:13:27] Speaker 1: And then, you know, when we're launching, submitting jobs.\n[2025-06-01 19:13:25] Speaker 1: being able to detect that a node is unhealthy.\n[2025-06-01 19:13:28] Speaker 1: The other thing I think is like the monitoring.\n[2025-06-01 19:13:25] Speaker 1: the rings that you mentioned, I don't think that would be super helpful for us because we're not\n[2025-06-01 19:13:25] Speaker 1: assigning nodes to people or teams.\n[2025-06-01 19:13:28] Speaker 1: Instead, we're extremely dynamic.\n[2025-06-01 19:13:31] Speaker 1: So we have...\n[2025-06-01 19:13:25] Speaker 1: you know, super cluster of thousands of GPUs.\n[2025-06-01 19:13:27] Speaker 1: And those nodes get used.\n[2025-06-01 19:13:30] Speaker 1: We use Q, which is a native.\n[2025-06-01 19:13:25] Speaker 1: Kubernetes project to schedule workloads.\n[2025-06-01 19:13:30] Speaker 1: So at any given time, we want to know, like, this\n[2025-06-01 19:13:25] Speaker 1: job is running with, you know, 128 GPUs, are any of the nodes unhealthy in this job? And so that job\n[2025-06-01 19:13:25] Speaker 1: might be using a different set of nodes than another job would be for the same team the next\n[2025-06-01 19:13:25] Speaker 1: day. So something like that would be helpful. Okay, wonderful. I think for the answer for the first\n[2025-06-01 19:13:25] Speaker 5: one is uh we are working with okay team i think okay team also has added some features lately where\n[2025-06-01 19:13:25] Speaker 5: they can detect an unhealthy gpu node and start tagging them as unallocatable uh for some some\n[2025-06-01 19:13:25] Speaker 5: the checks that they ran.\n[2025-06-01 19:13:27] Speaker 5: So we are actually working with them\n[2025-06-01 19:13:29] Speaker 5: to see if anything...\n[2025-06-01 19:13:25] Speaker 5: we find in our health check uh that uh that says that you know this node one of the gpus in the\n[2025-06-01 19:13:25] Speaker 5: node is continuous to underperform and communicate back on unschedulable.\n[2025-06-01 19:13:25] Speaker 5: we will have that integration point in the future for sure.\n[2025-06-01 19:13:29] Speaker 5: Right. So, and the metrics from.\n[2025-06-01 19:13:25] Speaker 5: OKE directly. So if there are part level, position volume level and lower\n[2025-06-01 19:13:25] Speaker 5: of good healthy prometheus metrics that come we can directly pipe all of those into this lens with\n[2025-06-01 19:13:25] Speaker 5: added set of things from our side to kind of consolidate and give you that one view that means\n[2025-06-01 19:13:25] Speaker 5: be needed. So that definitely is the plan, but we are looking into, okay, now a customer has to run.\n[2025-06-01 19:13:25] Speaker 5: a Prometheus instance in a Kubernetes cluster and that complication is what we're trying to do.\n[2025-06-01 19:13:25] Speaker 5: tackle the next feature set.\n[2025-06-01 19:13:29] Speaker 1: Okay, makes sense.\n[2025-06-01 19:13:30] Speaker 1: Yeah, just to reemphasize.\n[2025-06-01 19:13:25] Speaker 1: this is like probably the most important thing for us like really being able to\n[2025-06-01 19:13:25] Speaker 1: because we want to be able to do this automatically, right?\n[2025-06-01 19:13:28] Speaker 1: If a GPU breaks overnight, we don't want...\n[2025-06-01 19:13:25] Speaker 1: someone to have to, you know, go do something, go look at a Grafana dashboard we want.\n[2025-06-01 19:13:25] Speaker 1: our jobs to automatically react to these things.\n[2025-06-01 19:13:28] Speaker 5: Yeah. So I think that that brings another second.\n[2025-06-01 19:13:25] Speaker 5: question or even feedback you had for us is like the static way of creating monitoring ranks is\n[2025-06-01 19:13:25] Speaker 5: not ideal we fully hear you uh i think because of the dynamic if you're using q where you know\n[2025-06-01 19:13:25] Speaker 5: you get it booted out based on what team and priority and cohort and all of those stuff are.\n[2025-06-01 19:13:25] Speaker 5: We will integrate with OKE and the way we will look at is where the job is running or where\n[2025-06-01 19:13:25] Speaker 5: the deployment is running at a point of time and try to dynamically what we call add more tags to\n[2025-06-01 19:13:25] Speaker 5: metadata that's coming in to exactly relate to what experiment who ran it when when did it\n[2025-06-01 19:13:25] Speaker 5: complete timestamp it and try to pull those in.\n[2025-06-01 19:13:28] Speaker 5: So it's a thing that we are working.\n[2025-06-01 19:13:25] Speaker 5: towards, but we started with more non-Kubernetes-based experience, and then we would quick switch.\n[2025-06-01 19:13:25] Speaker 5: move into the dynamic workload where we pull in the nodes and even the GPUs, right?\n[2025-06-01 19:13:30] Speaker 5: Not full nodes.\n[2025-06-01 19:13:25] Speaker 5: be running every experiment so we will try to say four out of that no GPUs were running for this\n[2025-06-01 19:13:25] Speaker 5: experiment and that's what is aggregated into that board.\n[2025-06-01 19:13:28] Speaker 5: We will, that's a good point.\n[2025-06-01 19:13:25] Speaker 5: feedback and I think we will try to prioritize that.\n[2025-06-01 19:13:27] Speaker 5: Yeah, I think you're not very far from.\n[2025-06-01 19:13:25] Speaker 1: that honestly even without having to do like tagging of instances and trying to keep track of\n[2025-06-01 19:13:25] Speaker 1: what's running what. I think you could do some queries in your dashboard that just...\n[2025-06-01 19:13:25] Speaker 1: shows like your health metrics and then also use Kubernetes metrics and see like for\n[2025-06-01 19:13:25] Speaker 1: any given job or deployment, like you said, show all the nodes of that job and show if\n[2025-06-01 19:13:25] Speaker 1: of them are unhealthy so i think it's just a matter of like tying everything together\n[2025-06-01 19:13:25] Speaker 5: Yeah, I fully agree. Actually, we started like that, by the way, and I think Joleta and...\n[2025-06-01 19:13:25] Speaker 5: and most of my team knows that we started with a full Kubernetes-based experience, but we were nudged.\n[2025-06-01 19:13:25] Speaker 5: on like well i think there are a lot of people who want a full monitoring and this is where we\n[2025-06-01 19:13:25] Speaker 5: I think we have the both. We like to just pull it in like you said.\n[2025-06-01 19:13:25] Speaker 5: and start relating where, where, what is running\n[2025-06-01 19:13:28] Speaker 5: and start kind of creating that.\n[2025-06-01 19:13:25] Speaker 5: key value pair mapping here.\n[2025-06-01 19:13:25] Speaker 1: That's another question I'll have and then I'll pass it to like other.\n[2025-06-01 19:13:25] Speaker 1: Luis, Ace, if you have any questions, is you mentioned health checks. Can you talk?\n[2025-06-01 19:13:25] Speaker 1: a bit more about um how those would run like are you running anything that requires the workload\n[2025-06-01 19:13:25] Speaker 1: to be idle and if so is it or the gpu note to be idle and if so is it going to be doing\n[2025-06-01 19:13:25] Speaker 1: something like when nodes go idle to check the gpus or are these like running passively in the background\n[2025-06-01 19:13:25] Speaker 5: So I'll ask Soumya to chime in as well on she's an ML engineer worked on the script.\n[2025-06-01 19:13:25] Speaker 5: So we run this, we expect the GPUs to be idle and no workloads to be running.\n[2025-06-01 19:13:25] Speaker 5: this is where the PyTorch is loading into the tensor so it's loading into the accelerator.\n[2025-06-01 19:13:25] Speaker 5: running all these checks and coming back.\n[2025-06-01 19:13:27] Speaker 5: So we expect no workloads to be running when this.\n[2025-06-01 19:13:25] Speaker 5: this is running. We run it when you say you need to run it.\n[2025-06-01 19:13:28] Speaker 5: Right now we have not automated it.\n[2025-06-01 19:13:25] Speaker 5: And this is where we need feedback.\n[2025-06-01 19:13:26] Speaker 5: Like, do you want to run at midnight, 12 o'clock every day?\n[2025-06-01 19:13:25] Speaker 5: right you need that kind of no definitely not\n[2025-06-01 19:13:25] Speaker 1: Yeah, if I'm an ML engineer, I'm like, okay, I know when my pipeline is at a pause where I say call.\n[2025-06-01 19:13:25] Speaker 5: of my gpus are flushed out and idle i want to run the health checks for everything and make sure\n[2025-06-01 19:13:25] Speaker 5: of that is good before we go to the next phase, right? Is that an experience you would look for?\n[2025-06-01 19:13:25] Speaker 5: What we are thinking is we will give you a on-demand way for you to invoke health checkscripts.\n[2025-06-01 19:13:25] Speaker 5: through a REST API.\n[2025-06-01 19:13:26] Speaker 5: You have to just say\n[2025-06-01 19:13:28] Speaker 5: when you want to run it\n[2025-06-01 19:13:30] Speaker 5: and you on-demand run it.\n[2025-06-01 19:13:25] Speaker 5: Because we don't know when your GPUs are idle, then they're not.\n[2025-06-01 19:13:25] Speaker 1: Interesting. I was thinking like you might be able to figure that out like from a program.\n[2025-06-01 19:13:25] Speaker 1: programmatic standpoint, you might be able to get that data and whenever they are idle.\n[2025-06-01 19:13:25] Speaker 1: kick off like a workload that runs the Huff check.\n[2025-06-01 19:13:30] Speaker 1: So the intuition behind creating\n[2025-06-01 19:13:25] Speaker 3: this health check recipe is imagine a machine learning engineer or like you have a team that's\n[2025-06-01 19:13:25] Speaker 3: going to do a multi-node training or like multi-node fine tuning, right?\n[2025-06-01 19:13:29] Speaker 3: Before you go into doing...\n[2025-06-01 19:13:25] Speaker 3: a large-scale training operation or like any kind of like workload that's going to take too much\n[2025-06-01 19:13:25] Speaker 3: demand of the GPU itself.\n[2025-06-01 19:13:27] Speaker 3: You'd want to run this health check before that to understand that\n[2025-06-01 19:13:25] Speaker 3: health of your infrastructure right for example the way this health check is designed is it's based\n[2025-06-01 19:13:25] Speaker 3: on just 10 0 matrix multiplication right so depending upon the matrix size you probably be\n[2025-06-01 19:13:25] Speaker 3: loading like 818 like imagine like a matrix size of like 8192 by 818\n[2025-06-01 19:13:25] Speaker 3: too right you're going to be loading all these matrix into the gpu memory and we are going to\n[2025-06-01 19:13:25] Speaker 3: pressurize and see how much your machine is able to take the computation throughput or like throat\n[2025-06-01 19:13:25] Speaker 3: the power and things like that. So ideally you would want to do it when your machine is idle and to\n[2025-06-01 19:13:25] Speaker 3: address your point yeah we could actually schedule and understand when the jobs are not running\n[2025-06-01 19:13:25] Speaker 3: and we could just run the health check at that point.\n[2025-06-01 19:13:27] Speaker 3: And we can detect when the machines are...\n[2025-06-01 19:13:25] Speaker 3: Yeah, I think it'd be interesting to have the option to schedule it on demand, like as a...\n[2025-06-01 19:13:25] Speaker 1: we hook like to the jobs like you said um but i don't i need to think more about this\n[2025-06-01 19:13:25] Speaker 1: But I think, you know, my intuition is why not both, you know, like having a periodic check.\n[2025-06-01 19:13:25] Speaker 1: that best effort runs when the node is sitting there doing nothing.\n[2025-06-01 19:13:29] Speaker 1: So that if a node does go unhealthy,\n[2025-06-01 19:13:25] Speaker 1: we can find out about it early and not wait for a job to get scheduled there to find out.\n[2025-06-01 19:13:25] Speaker 5: Yeah, no, we'll take it as a good feedback.\n[2025-06-01 19:13:28] Speaker 5: We have still, because we have to think.\n[2025-06-01 19:13:25] Speaker 5: through a little bit on scheduling because it takes roughly five minutes for this head check\n[2025-06-01 19:13:25] Speaker 5: to finish and in mix of this if your queue ends up scheduling another job uh they'll they'll\n[2025-06-01 19:13:25] Speaker 5: it can't find an ideal GPU. So they go on pending. So we have to just think through this.\n[2025-06-01 19:13:25] Speaker 5: Yeah, no, this would definitely, yes, this would definitely have to be an...\n[2025-06-01 19:13:25] Speaker 1: interruptible workload so that our workloads can always schedule the priority.\n[2025-06-01 19:13:25] Speaker 5: Exactly right you cannot boot this workload because this is running at the system\n[2025-06-01 19:13:25] Speaker 5: level, right? And Kubernetes layers are at a much higher layer too.\n[2025-06-01 19:13:25] Speaker 5: and we need to think of this a little bit but i think this is really good feedback um thank you\n[2025-06-01 19:13:25] Speaker 1: Yeah, no, thank you.\n[2025-06-01 19:13:26] Speaker 1: Louise Ace, you have anything?\n[2025-06-01 19:13:25] Speaker 4: Mostly just echoing your thoughts. I mean, when we talk about node lifecycle,\n[2025-06-01 19:13:25] Speaker 4: It's very much like an ongoing thing.\n[2025-06-01 19:13:26] Speaker 4: It's not sort of a one off thing.\n[2025-06-01 19:13:27] Speaker 4: So just, yeah.\n[2025-06-01 19:13:25] Speaker 4: like having both elements and thinking about how nodes like proceed through.\n[2025-06-01 19:13:25] Speaker 4: I don't want to say this process exactly, but like, I don't know, mostly just echoing Cecile's\n[2025-06-01 19:13:25] Speaker 4: thoughts. Okay. Yeah. Thank you. So what, what we are trying to also avoid is before you shut this\n[2025-06-01 19:13:25] Speaker 5: machine and turn it back into OCI to repair. We want to see what is the issue, right?\n[2025-06-01 19:13:25] Speaker 5: deeper into what where is the concern is it a single gpu always coming back with a lower\n[2025-06-01 19:13:25] Speaker 5: performance compared to the rest seven of them in the same node or is it consistent uh no\n[2025-06-01 19:13:25] Speaker 5: RDMA nick flapping issues, errors that show up.\n[2025-06-01 19:13:28] Speaker 5: And this is a start.\n[2025-06-01 19:13:30] Speaker 5: And where we want to go.\n[2025-06-01 19:13:25] Speaker 5: is if we start seeing a pattern of things which are very consistent with a lot of other customers\n[2025-06-01 19:13:25] Speaker 5: and a lot of type of GPUs,\n[2025-06-01 19:13:26] Speaker 5: we will come back with recommendations\n[2025-06-01 19:13:28] Speaker 5: through an agentic AI type of app.\n[2025-06-01 19:13:25] Speaker 5: flow where probably just a reboot may fix the things or it could be specific to a driver.\n[2025-06-01 19:13:25] Speaker 5: the GPU driver you have, which has seen this incompatibility with the Mellanox drivers we have.\n[2025-06-01 19:13:25] Speaker 5: example. So we are trying to learn from using the data as well as all the issues customers.\n[2025-06-01 19:13:25] Speaker 5: give us to start recommending, right?\n[2025-06-01 19:13:27] Speaker 5: So this is just a start for us to start collecting,\n[2025-06-01 19:13:29] Speaker 5: but next approach for us is\n[2025-06-01 19:13:25] Speaker 5: remove the noise and tell if there is really a issue with the host or if this is a transient issue.\n[2025-06-01 19:13:25] Speaker 5: or not an issue with something related to an experiment that has been set up.\n[2025-06-01 19:13:25] Speaker 5: workload is being scheduled on this or the topology that it's been deployed with.\n[2025-06-01 19:13:29] Speaker 5: Any of those questions?\n[2025-06-01 19:13:25] Speaker 5: Right, so that's where we want to get to.\n[2025-06-01 19:13:25] Speaker 5: Awesome. So here's the log as well.\n[2025-06-01 19:13:28] Speaker 5: This is this we'll build up more and more.\n[2025-06-01 19:13:25] Speaker 5: stuff and it's all accessible either through portal or an api where you don't have to assess\n[2025-06-01 19:13:25] Speaker 5: an instance. Because the scripts are going to be standardized that what you see against\n[2025-06-01 19:13:25] Speaker 5: what OCI support sees everything will be consistent and maybe in the future once we have support team\n[2025-06-01 19:13:25] Speaker 5: running full diagnosis they know where to narrow down the issue to like exact GPU or the NICs or\n[2025-06-01 19:13:25] Speaker 5: anything instead of running this full you know six hours uh diagnosis on the host right\n[2025-06-01 19:13:25] Speaker 5: So we're just trying to narrow down that time it takes to fix these machines too.\n[2025-06-01 19:13:30] Speaker 5: So that will help.\n[2025-06-01 19:13:25] Speaker 1: When you say consistent, is the idea that you give us a set of scripts and we run these?\n[2025-06-01 19:13:25] Speaker 1: or they see the results of our health checks.\n[2025-06-01 19:13:25] Speaker 1: So right now with the way it is integrated, the OCI support does not see the results, but you could just take...\n[2025-06-01 19:13:25] Speaker 5: these results copy paste in your support ticket if you need to OCI support will have access\n[2025-06-01 19:13:25] Speaker 5: the same health check tools and scripts that you are running.\n[2025-06-01 19:13:29] Speaker 5: So it's a consistent tool.\n[2025-06-01 19:13:25] Speaker 5: that you both are running,\n[2025-06-01 19:13:26] Speaker 5: where what you have seen as the results,\n[2025-06-01 19:13:29] Speaker 5: for example, the score,\n[2025-06-01 19:13:25] Speaker 5: So here's the flops, for example.\n[2025-06-01 19:13:28] Speaker 5: This is the duration, right?\n[2025-06-01 19:13:30] Speaker 5: What you are seeing...\n[2025-06-01 19:13:25] Speaker 5: when OCI support is trying to run this on their own.\n[2025-06-01 19:13:28] Speaker 5: There is a lot of consistency.\n[2025-06-01 19:13:25] Speaker 5: Right. And if you see this GPU consistently coming back with like, okay, it takes 31 seconds.\n[2025-06-01 19:13:25] Speaker 5: It's easy to narrow down saying that GQ7 is something.\n[2025-06-01 19:13:25] Speaker 5: that we need to request.\n[2025-06-01 19:13:25] Speaker 1: I see. Yeah, I think it would be even better if we can get to a stage where they...\n[2025-06-01 19:13:25] Speaker 1: like have access to historical results and they can just see things that are\n[2025-06-01 19:13:28] Speaker 1: ZeeBrand.\n[2025-06-01 19:13:25] Speaker 1: Yeah.\n[2025-06-01 19:13:25] Speaker 1: instead of having to reproduce and copy pasting things.\n[2025-06-01 19:13:25] Speaker 1: i fully agree i think that's our goal but we are not a oci service yet\n[2025-06-01 19:13:25] Speaker 5: right and uh because it's all running the customer's tenancy uh there's obviously\n[2025-06-01 19:13:25] Speaker 5: little bit of things about what we can collect, what we cannot. And we had to go through those.\n[2025-06-01 19:13:25] Speaker 5: product approval phases.\n[2025-06-01 19:13:28] Speaker 5: And then we will probably\n[2025-06-01 19:13:29] Speaker 5: support and have access to this.\n[2025-06-01 19:13:25] Speaker 1: Okay, makes sense. One other question for Prometheus. You mentioned at some point in the\n[2025-06-01 19:13:25] Speaker 1: this slide something about setting up a CPU cluster or gate cluster to run Prometheus.\n[2025-06-01 19:13:25] Speaker 1: can we reuse our existing primukes instead?\n[2025-06-01 19:13:25] Speaker 5: Yes, yes, absolutely. You can use your existing. You need to just enable push gateway and add.\n[2025-06-01 19:13:25] Speaker 5: push gateway configurations because that's how all of the scripts push their results and metrics.\n[2025-06-01 19:13:25] Speaker 5: Okay, can you give a bit more detail about how this works?\n[2025-06-01 19:13:25] Speaker 1: scripts push metrics or how does that work\n[2025-06-01 19:13:25] Speaker 5: Yeah, Hrithika, can you elaborate on this?\n[2025-06-01 19:13:25] Speaker 2: Hello? Can you hear me?\n[2025-06-01 19:13:25] Speaker 5: Is VTK wrong?\n[2025-06-01 19:13:25] Speaker 5: Yeah.\n[2025-06-01 19:13:25] Speaker 2: All right. Could you give that question once again?\n[2025-06-01 19:13:25] Speaker 1: It was just having a bit more detail about how the metrics get emitted.\n[2025-06-01 19:13:25] Speaker 1: Because I understand there are scripts and then I'm just trying to figure out like how we could.\n[2025-06-01 19:13:25] Speaker 1: scrape our own metrics.\n[2025-06-01 19:13:25] Speaker 1: Oh yeah, so we basically build the node exporter, DCGM, AMD exporter, all of that depending on the GPU.\n[2025-06-01 19:13:25] Speaker 2: it is and bring those up and there are also some metrics which are OCI specific which are\n[2025-06-01 19:13:25] Speaker 2: return in a Go plugin and that plugin\n[2025-06-01 19:13:29] Speaker 2: fits metrics to the push gateway at a regular interval.\n[2025-06-01 19:13:25] Speaker 5: Yeah, all of the scripts here are...\n[2025-06-01 19:13:27] Speaker 5: Does that make sense?\n[2025-06-01 19:13:25] Speaker 5: push gateway if you see it here and\n[2025-06-01 19:13:25] Speaker 5: And these are scraped by Prometheus\n[2025-06-01 19:13:27] Speaker 5: based on Prometheus script config.\n[2025-06-01 19:13:29] Speaker 5: And-\n[2025-06-01 19:13:25] Speaker 5: Push gateway is part of a service that is default in Prometheus server install.\n[2025-06-01 19:13:30] Speaker 5: And it's the same.\n[2025-06-01 19:13:25] Speaker 5: mechanism.\n[2025-06-01 19:13:27] Speaker 5: Gotcha.\n[2025-06-01 19:13:28] Speaker 1: Okay.\n[2025-06-01 19:13:28] Speaker 1: I'll look into this a bit more.\n[2025-06-01 19:13:29] Speaker 1: I haven't used push gateway before.\n[2025-06-01 19:13:25] Speaker 1: So that's probably where my gap comes from.\n[2025-06-01 19:13:27] Speaker 1: Okay.\n[2025-06-01 19:13:27] Speaker 5: Yeah, I think it's part of the setup.\n[2025-06-01 19:13:25] Speaker 5: And if you go, if you run Prometheus through operator,\n[2025-06-01 19:13:28] Speaker 5: Push Gateway is usually installed as a service.\n[2025-06-01 19:13:25] Speaker 5: It should have its own service endpoint,\n[2025-06-01 19:13:28] Speaker 5: and you have to just enable a few things.\n[2025-06-01 19:13:25] Speaker 5: and you're good to go. Yeah. So, this is for instead of you going and reaching out.\n[2025-06-01 19:13:25] Speaker 5: going and reaching out and pulling the metrics out which is where you need firewall exception\n[2025-06-01 19:13:25] Speaker 5: and opening the ports and all of that.\n[2025-06-01 19:13:27] Speaker 5: Here, any metrics you are generating, including DCGM,\n[2025-06-01 19:13:25] Speaker 5: all of those are pool metrics, right?\n[2025-06-01 19:13:27] Speaker 5: We push it directly to the push page.\n[2025-06-01 19:13:25] Speaker 5: and your Prometheus is scraping push gateway basically.\n[2025-06-01 19:13:29] Speaker 5: It's like an intermediate data store.\n[2025-06-01 19:13:25] Speaker 5: All right. Any further questions? And we are also eager if you have any feedback.\n[2025-06-01 19:13:25] Speaker 5: can think about we are thinking in next roughly two weeks you can you can have access to all of\n[2025-06-01 19:13:25] Speaker 5: the health check scripts.\n[2025-06-01 19:13:27] Speaker 5: We'll get access to your repo\n[2025-06-01 19:13:29] Speaker 5: once you share your GitHub IDs.\n[2025-06-01 19:13:25] Speaker 5: can poke around and really we are here to listen and and build something that that makes uh\n[2025-06-01 19:13:25] Speaker 5: life easy right so we are here to listen and you know continue to iterate as fast as we\n[2025-06-01 19:13:25] Speaker 5: can to give you what you're looking for.\n[2025-06-01 19:13:27] Speaker 5: So.\n[2025-06-01 19:13:28] Speaker 1: Thanks, Samar.\n[2025-06-01 19:13:29] Speaker 1: I think overall this is, you know.\n[2025-06-01 19:13:25] Speaker 1: something that we've been asking for. So really, really happy it's, it's getting there.\n[2025-06-01 19:13:25] Speaker 1: and you all are focusing your efforts there.\n[2025-06-01 19:13:26] Speaker 1: It's definitely heading the right direction.\n[2025-06-01 19:13:25] Speaker 1: I think looking at what we have today that we built ourselves a best over the past year,\n[2025-06-01 19:13:29] Speaker 1: it's definitely...\n[2025-06-01 19:13:25] Speaker 1: not there yet. I think what we have right now is a bit more advanced just because we've built\n[2025-06-01 19:13:25] Speaker 1: our own set of scripts and like we've iterated on them and we have all that automation.\n[2025-06-01 19:13:25] Speaker 1: and integration with Kubernetes already.\n[2025-06-01 19:13:27] Speaker 1: But I like where this is going and yeah, excited too.\n[2025-06-01 19:13:25] Speaker 1: work with you all too.\n[2025-06-01 19:13:26] Speaker 1: It's great.\n[2025-06-01 19:13:27] Speaker 5: Okay.\n[2025-06-01 19:13:28] Speaker 5: Yeah.\n[2025-06-01 19:13:28] Speaker 5: We'll try to,\n[2025-06-01 19:13:30] Speaker 5: right now,\n[2025-06-01 19:13:31] Speaker 5: we are not overly\n[2025-06-01 19:13:25] Speaker 5: Kubernetes heavy features and if we will probably in the next iteration or a milestone\n[2025-06-01 19:13:25] Speaker 5: add all of those capabilities to bring Kubernetes metrics and that experience together.\n[2025-06-01 19:13:25] Speaker 5: And hopefully, Sisal, I'll post that.\n[2025-06-01 19:13:27] Speaker 5: You would be open to trying it out and giving us the feedback.\n[2025-06-01 19:13:25] Speaker 5: and iterating like that.\n[2025-06-01 19:13:29] Speaker 1: Sounds good.\n[2025-06-01 19:13:32] Speaker 5: Thanks for making the time to give us a demo.\n[2025-06-01 19:13:25] Speaker 1: Appreciate it.\n[2025-06-01 19:13:25] Speaker 5: Thanks for spending your time with us as well.\n[2025-06-01 19:13:28] Speaker 5: So anyone else, any questions?\n[2025-06-01 19:13:25] Speaker 1: Thank you.\n[2025-06-01 19:13:25] Speaker 1: Thank you.\n[2025-06-01 19:13:25] Speaker 1: Thank you.\n[2025-06-01 19:13:25] Speaker 1: Thank you.", + "summary": "Key Points:\n\n* The OCI Lens initiative is currently in the incubation phase and is not yet an MVP or closer to it.\n* The team is focused on continuous GPU and cluster-level monitoring of both NVIDIA and AMD GPUs.\n* The solution includes active health checks and active monitoring, which is a huge investment.\n* The team has built team-level tracking, which allows multiple teams to monitor their own subset of systems.\n* The solution also includes cost tracking, which allows users to see how much computer resources they have used.\n\nAction Items:\n\n* The team will show a demo of the OCI Lens initiative, which allows users to monitor either single, bare metal, virtual machine instances, or a full OKE cluster or an HPC cluster.\n* The team is working towards automatically fetching the nodes that are running an experiment based on Kubernetes scheduling.\n* The team will demonstrate the difference between performance monitoring and health checks, which goes very close to the layers that an ML engineer would operate under.\n\n---\n\nKey points:\n\n* PyTorch and JAX are being considered for performance testing.\n* The approach is to push metrics and health check data to Prometheus and the central control plane.\n* The origination of all metrics is within the network.\n* There are plenty of metrics, including NVIDIA DCGM exporter, AMD SMI metrics, and RDMA metrics.\n* Health checks include traditional disk IO usage and are adjusted based on performance seen.\n* The architecture is evolving and will be deployed as a dedicated OKE cluster with CPU nodes.\n* The footprint includes Prometheus, Grafana, open source, and control plane API.\n* The demo will be shown, and questions can be asked after.\n\nDecisions:\n\n* PyTorch and JAX will be considered for performance testing.\n* The approach is to push metrics and health check data to Prometheus and the central control plane.\n\nAction items:\n\n* None specified in the transcript.\n\n---\n\nKey Points:\n\n* The meeting discussed a plug-in model for monitoring OCI lengths.\n* To start monitoring, an existing instance or a new instance being provisioned must include a script.\n* The architecture is simple and straightforward, with the monitoring running directly on the bare metal host.\n* The monitoring can be scanned for Kubernetes clusters, RDMA cluster networks, or compute clusters.\n* The experience is that when you check a Kubernetes cluster, all GPU nodes under it are automatically scanned and added to the monitoring.\n* The meeting showed a portal for accessing the monitoring through REST API endpoints.\n* The monitoring allows for the creation of a monitoring ring, which can be used to bundle all the things.\n* Every monitoring ring comes with a dedicated Grafana board.\n* The health summary of all compute nodes that are part of the monitoring was demonstrated.\n* The health checks include performance-related checks that are done on the host when the plugin is activated.\n* The health checks can be run on demand, and a link will soon be provided to a JSON that is deeper on all the tests.\n\nDecisions:\n\n* The monitoring will be started by including a script in an existing instance or a new instance being provisioned.\n* The monitoring will run directly on the bare metal host.\n* The monitoring can be scanned for Kubernetes clusters, RDMA cluster networks, or compute clusters.\n* The monitoring allows for the creation of a monitoring ring, which can be used to bundle all the things.\n\nAction Items:\n\n* Provide a link to a JSON that is deeper on all the tests.\n\n---\n\nSummary of Meeting:\n\n* The speaker presented a tool that provides detailed data on GPU usage, health, and performance.\n* The tool is natively available in Prometheus and can be extended with custom labels.\n* The speaker added that they are working on improving the UI and automating some tasks to help users identify and resolve issues more efficiently.\n* The speaker mentioned that they are planning to add more data related to metadata, such as host metadata survey, to provide a more comprehensive view of the system.\n* The speaker also mentioned that they are planning to add more boards to the tool to provide more information about the system.\n* The speaker mentioned that they are planning to add more features to the tool, such as automating some tasks and improving the UI.\n* The speaker mentioned that they are planning to add more data related to metadata, such as host metadata survey, to provide a more comprehensive view of the system.\n* The speaker mentioned that they are planning to add more boards to the tool to provide more information about the system.\n* The speaker mentioned that they are planning to add more features to the tool, such as automating some tasks and improving the UI.\n* The speaker mentioned that they are planning to add more data related to metadata, such as host metadata survey, to provide a more comprehensive view of the system.\n* The speaker mentioned that they are planning to add more boards to the tool to provide more information about the system.\n* The speaker mentioned that they are planning to add more features to the tool, such as automating some tasks and improving the UI.\n* The speaker mentioned that they are planning to add more data related to metadata, such as host metadata survey, to provide a more comprehensive view of the system.\n* The speaker mentioned that they are planning to add more boards to the tool to provide more information about the system.\n* The speaker mentioned that they are planning to add more features to the tool, such as automating some tasks and improving the UI.\n* The speaker mentioned that they are planning to add more data related to metadata, such as host metadata survey, to provide a more comprehensive view of the system.\n* The speaker mentioned that they are planning to add more boards to the tool to provide more information about the system.\n* The speaker mentioned that they are planning to add more features to the tool, such as automating some tasks and improving the UI.\n* The speaker mentioned that they are planning to add more data\n\n---\n\nKey points:\n\n* The team is working on a solution to automatically detect and react to unhealthy GPUs in a Kubernetes cluster.\n* The solution will involve integrating with the Open Kubernetes Engine (OKE) and dynamically adding tags to metadata.\n* The team is also working on improving the static way of creating monitoring ranks.\n* The team is looking into using Prometheus to monitor the health of GPUs in a Kubernetes cluster.\n\nDecisions:\n\n* The team will integrate with OKE to dynamically add tags to metadata.\n* The team will prioritize the development of a solution to automatically detect and react to unhealthy GPUs.\n\nAction items:\n\n* The team will continue working on integrating with OKE and dynamically adding tags to metadata.\n* The team will prioritize the development of a solution to automatically detect and react to unhealthy GPUs.\n\n---\n\nKey Points:\n\n* The team is currently not automating the running of health checks on GPUs.\n* Feedback is needed on when to run health checks, such as at midnight every day or on demand.\n* The team is considering giving an on-demand way for ML engineers to invoke health check scripts through a REST API.\n* The team is also considering creating a health check recipe that can be run before large-scale training operations to understand the health of the infrastructure.\n* The team is considering scheduling health checks on demand, as well as having a periodic check that runs when the node is idle.\n* The team is aware of the need for an interruptible workload so that workloads can always schedule their priority.\n* The team is trying to learn from data and customer issues to start recommending specific fixes for GPU driver incompatibilities.\n\nAction Items:\n\n* Determine the best way to invoke health check scripts on demand.\n* Create a health check recipe that can be run before large-scale training operations.\n* Schedule health checks on demand as well as having a periodic check that runs when the node is idle.\n* Develop an agentic AI-type app to learn from data and customer issues to start recommending specific fixes for GPU driver incompatibilities.\n\n---\n\nKey points:\n\n* The team is working on a new approach to diagnose issues with hosts or experiments scheduled on them.\n* The goal is to narrow down the time it takes to fix these machines.\n* The team is working on building a consistent tool that both OCI support and customers can use to diagnose issues.\n* The team is also working on improving the historical results and access to them for customers.\n* The team is exploring the possibility of reusing existing primukes instead of setting up a CPU cluster or gate cluster to run Prometheus.\n* The team is working on building node exporter, DCGM, AMD exporter, and other metrics exporters depending on the GPU.\n* The team is also working on integrating OCI-specific metrics into Prometheus.\n* The team is using push gateway to collect metrics from scripts and scrape them by Prometheus.\n\nDecisions:\n\n* The team will work on building a consistent tool that both OCI support and customers can use to diagnose issues.\n* The team will explore the possibility of reusing existing primukes instead of setting up a CPU cluster or gate cluster to run Prometheus.\n\nAction items:\n\n* The team will work on building node exporter, DCGM, AMD exporter, and other metrics exporters depending on the GPU.\n* The team will integrate OCI-specific metrics into Prometheus.\n* The team will use push gateway to collect metrics from scripts and scrape them by Prometheus.\n\n---\n\nKey points:\n\n* Push Gateway is a service that should have its own service endpoint.\n* Any metrics generated, including DCGM, are pool metrics that are pushed directly to the Push Gateway.\n* Prometheus is used to scrape the Push Gateway, acting as an intermediate data store.\n* Access to health check scripts will be provided in roughly two weeks once GitHub IDs are shared.\n* The team is focused on building something that makes life easy for the user.\n* The Push Gateway is not currently Kubernetes heavy, but Kubernetes capabilities may be added in the next iteration or milestone.\n* Sisal is open to trying out and giving feedback on the Push Gateway.\n\nDecisions:\n\n* The Push Gateway should be installed as a service with its own service endpoint.\n* Any metrics generated should be pushed directly to the Push Gateway.\n* Prometheus should be used to scrape the Push Gateway.\n* Access to health check scripts will be provided in roughly two weeks.\n\nAction items:\n\n* Enable a few things for the Push Gateway.\n* Share GitHub IDs to access health check scripts.\n* Iterate on the Push Gateway to add Kubernetes capabilities.\n* Try out and give feedback on the Push Gateway." +} \ No newline at end of file diff --git a/docs/whisper_transcription/Examples/Test3/audio1788670787_all_transcripts_20250601_191710.txt b/docs/whisper_transcription/Examples/Test3/audio1788670787_all_transcripts_20250601_191710.txt new file mode 100644 index 0000000..72c04b4 --- /dev/null +++ b/docs/whisper_transcription/Examples/Test3/audio1788670787_all_transcripts_20250601_191710.txt @@ -0,0 +1,749 @@ +[2025-06-01 19:13:25] Speaker 5: All right. Thank you all. Morning, afternoon. So my name is Amar Gowda. I am one of the +[2025-06-01 19:13:30] Speaker 5: products. +[2025-06-01 19:13:25] Speaker 5: managers leading the OCI Lens initiative. So right now it's kind of what we call incubation. +[2025-06-01 19:13:25] Speaker 5: phases so we are not yet to be a mvp or closer to it so we're kind of getting early feedback +[2025-06-01 19:13:25] Speaker 5: and this is like think of this as a kind of private preview in a way right so a little earlier on +[2025-06-01 19:13:25] Speaker 5: on the journey. +[2025-06-01 19:13:26] Speaker 5: But this is something we started looking into, +[2025-06-01 19:13:31] Speaker 5: which we, I think we spoke with. +[2025-06-01 19:13:25] Speaker 5: you, Cecil, you and the rest of the team on what does it take to operate at scale? +[2025-06-01 19:13:25] Speaker 5: what are the things that we could improve on, especially the monitoring, the health check. +[2025-06-01 19:13:25] Speaker 5: side of the things, which has, you know, you gave a set of requirements and that led us. +[2025-06-01 19:13:25] Speaker 5: to building something to help customers like you. +[2025-06-01 19:13:30] Speaker 5: Okay, so I have a team here. +[2025-06-01 19:13:32] Speaker 5: Joletta is... +[2025-06-01 19:13:25] Speaker 5: another software engineers somia on the call is our data scientist selection ml engineer +[2025-06-01 19:13:30] Speaker 5: uh +[2025-06-01 19:13:25] Speaker 5: also have uh hithika who's our golang developer she's ex uh akis team so that's where if you all can +[2025-06-01 19:13:25] Speaker 5: connect a little bit too. +[2025-06-01 19:13:27] Speaker 5: All right. Today, I'm going to keep it more focused on the demo. +[2025-06-01 19:13:25] Speaker 5: and what this tool is all about. +[2025-06-01 19:13:27] Speaker 5: And then we'll, you know, answer questions +[2025-06-01 19:13:29] Speaker 5: and kind of talk through. +[2025-06-01 19:13:25] Speaker 5: But most importantly, we want to hear if the current version has a set of features you're +[2025-06-01 19:13:25] Speaker 5: looking for? Is it missing? Is it you want to see something more? Because we are kind of very +[2025-06-01 19:13:25] Speaker 5: rapidly on a weekly sprint uh releasing the features to this and we are really happy to incorporate +[2025-06-01 19:13:25] Speaker 5: anything and work on how you can onboard to this by the way +[2025-06-01 19:13:31] Speaker 5: uh so four areas where we are focused on +[2025-06-01 19:13:25] Speaker 5: this solution is continuous GPU and cluster level monitoring of both NVIDIA GPUs and AMD GPUs. +[2025-06-01 19:13:25] Speaker 5: together, right? So this is how we want to kind of platform +[2025-06-01 19:13:29] Speaker 5: agnostic solution that works for all GPUs. +[2025-06-01 19:13:25] Speaker 5: and has all the latest metrics available for you to digest and look at. +[2025-06-01 19:13:32] Speaker 5: The next thing which we've highlighted... +[2025-06-01 19:13:25] Speaker 5: focused on is the health checks piece to this right are my nodes healthy are they performing to +[2025-06-01 19:13:30] Speaker 5: what they +[2025-06-01 19:13:25] Speaker 5: should be uh active health checks active monitoring is is is a huge investment that we did uh and i'll +[2025-06-01 19:13:32] Speaker 5: go over some +[2025-06-01 19:13:25] Speaker 5: details about what all health checks that we have enabled including the RDMA cluster level. +[2025-06-01 19:13:25] Speaker 5: checks that we call MPI tests as part of the solution fully baked into. +[2025-06-01 19:13:30] Speaker 5: We kept it to cloud native. +[2025-06-01 19:13:25] Speaker 5: because I heard it very clear that it has to be Grafana, Prometheus, Native, or Otel, anything open source. +[2025-06-01 19:13:25] Speaker 5: Cloud Native Alliance so that it's easy to be move between clouds or on-prem and use +[2025-06-01 19:13:29] Speaker 5: the same existing. +[2025-06-01 19:13:25] Speaker 5: tools you always used for for monitoring another area that we've paid attention and we built is the +[2025-06-01 19:13:25] Speaker 5: team level tracking. A lot of these large size clusters are usually used by multiple teams and +[2025-06-01 19:13:25] Speaker 5: everybody wants to know the health of their own subset of systems or how their experiment is +[2025-06-01 19:13:30] Speaker 5: performing. +[2025-06-01 19:13:25] Speaker 5: if they have an unhealthy node, all of those combinations. +[2025-06-01 19:13:29] Speaker 5: And another important thing is about +[2025-06-01 19:13:25] Speaker 5: cost tracking right people want to see how much computer resources they've used uh whatever the +[2025-06-01 19:13:25] Speaker 5: the power the total gpu power they have used for running this experiment right those were a few +[2025-06-01 19:13:29] Speaker 5: things that +[2025-06-01 19:13:25] Speaker 5: that we also focused to build on this. +[2025-06-01 19:13:29] Speaker 5: The current set of features that we have +[2025-06-01 19:13:32] Speaker 5: and what we have built right now +[2025-06-01 19:13:25] Speaker 5: now is first thing tenancy level monitoring you no longer have region barriers right so we any +[2025-06-01 19:13:25] Speaker 5: data, any region, OCI region you may be in, everything is monitorable as a single instance +[2025-06-01 19:13:31] Speaker 5: for us. +[2025-06-01 19:13:25] Speaker 5: So I'm going to show you a demo. +[2025-06-01 19:13:27] Speaker 5: We allow you to monitor either single, bare metal, virtual machine +[2025-06-01 19:13:25] Speaker 5: instances or a full oke cluster or an hpc cluster that if you may be using slurm or a native uh +[2025-06-01 19:13:25] Speaker 5: called cluster network, compute cluster setup in OCI. +[2025-06-01 19:13:29] Speaker 5: The third part of the feature that we worked on +[2025-06-01 19:13:25] Speaker 5: and all of this we're going to see a demo quickly but i'm going to just spend maybe five more minutes +[2025-06-01 19:13:25] Speaker 5: and then go to the demo. +[2025-06-01 19:13:26] Speaker 5: Is it team level tracking? +[2025-06-01 19:13:27] Speaker 5: You can create team level tracking. +[2025-06-01 19:13:25] Speaker 5: handpicking whichever nodes are part of this experiment. +[2025-06-01 19:13:29] Speaker 5: We are also working towards... +[2025-06-01 19:13:25] Speaker 5: automatically fetching the nodes that are running experiment based on Kubernetes +[2025-06-01 19:13:29] Speaker 5: how the job system. +[2025-06-01 19:13:25] Speaker 5: scheduled. So that's another area we're working on automatically, dynamically pulling the nodes that +[2025-06-01 19:13:25] Speaker 5: part of that experiment rather than you allocating what compute nodes go to. +[2025-06-01 19:13:31] Speaker 5: This I've already said. +[2025-06-01 19:13:25] Speaker 5: already have both of them running and i'm going to show you that uh the difference between what we do +[2025-06-01 19:13:25] Speaker 5: with performance monitoring and health check is we go very close to the layers that an ML engineer would +[2025-06-01 19:13:25] Speaker 5: operate under, which is PyTorch for us, PyTorch and JAX primarily, right? So we picked up PyTorch base. +[2025-06-01 19:13:25] Speaker 5: matmal and linear regression based performance testing to achieve how many flops did we achieve +[2025-06-01 19:13:25] Speaker 5: on this compute node how much is each gpu performing and is it is it within the threshold of +[2025-06-01 19:13:25] Speaker 5: the specification from NVIDIA or AMD what we have traditionally seen. So we have a baseline. +[2025-06-01 19:13:25] Speaker 5: that we score the performance to because it's the standard set of precision based testing either FBA. +[2025-06-01 19:13:25] Speaker 5: FP16, FP32, it automatically does all of this. And most of this code, by the way, for the health +[2025-06-01 19:13:25] Speaker 5: will all be open source so you can see you can change the things if you like or add to it +[2025-06-01 19:13:25] Speaker 5: right and please feel free to contribute to it too. +[2025-06-01 19:13:28] Speaker 5: This you've seen it. +[2025-06-01 19:13:30] Speaker 5: The approach we have taken. +[2025-06-01 19:13:25] Speaker 5: here is instead of asking you to poke holes in your existing cluster or networks, we basically push the +[2025-06-01 19:13:25] Speaker 5: metrics and we push the health check data to the Prometheus and our central control plane. +[2025-06-01 19:13:25] Speaker 5: It could be looked at as, oh, I need to open this port, that port, no exception. +[2025-06-01 19:13:25] Speaker 5: if you're in a vc and in a production environment where everything is locked down egress is usually +[2025-06-01 19:13:25] Speaker 5: easier and you can always sniff the egress but these are all running within your own tenancy so +[2025-06-01 19:13:25] Speaker 5: So the origination of all of the metrics to where it's being sent is all within your network. +[2025-06-01 19:13:25] Speaker 5: nothing is going into the public. All right. So I'm going to go over this. +[2025-06-01 19:13:25] Speaker 5: but there are plenty of metrics, just a lot. +[2025-06-01 19:13:28] Speaker 5: And it's kind of an eye candy chart on Grafana's to look. +[2025-06-01 19:13:25] Speaker 5: at but i think you get the point right you have full nvidia dcgm exporter all metrics +[2025-06-01 19:13:25] Speaker 5: You have all AMD SMI metrics. +[2025-06-01 19:13:27] Speaker 5: You have all of the RDMA metrics that is particularly on our ROC. +[2025-06-01 19:13:25] Speaker 5: implementation that we sniffed the melanocs drivers for to capture that front end next +[2025-06-01 19:13:25] Speaker 5: node health is included, health check, full check is included, +[2025-06-01 19:13:29] Speaker 5: and the traditional disk IO usage. +[2025-06-01 19:13:25] Speaker 5: all of those that you usually get with Prometheus already included. +[2025-06-01 19:13:28] Speaker 5: So all of these is bundled as one. +[2025-06-01 19:13:25] Speaker 5: unpacking. +[2025-06-01 19:13:25] Speaker 5: On the health checks piece, these are all the tests we do today and we'll continue to add more or adjust. +[2025-06-01 19:13:25] Speaker 5: based on how performance we see we do extensive benchmarking to see where the threshold should +[2025-06-01 19:13:25] Speaker 5: be or the baseline should be and then we either go up and down based on how we are seeing. +[2025-06-01 19:13:25] Speaker 5: example the what should be the ideal mfu on a 140 gb against b200 or mi300x right and then we we lower it +[2025-06-01 19:13:25] Speaker 5: if needed right so but we work with our vendors to also see if this we already doing the right way or not +[2025-06-01 19:13:25] Speaker 5: So here are all the checks we run. +[2025-06-01 19:13:27] Speaker 5: And all of these, once they complete, you will see a... +[2025-06-01 19:13:25] Speaker 5: metrics flowing through Grafana and you'll also see this. +[2025-06-01 19:13:25] Speaker 5: Very simple. This is how the architecture is. I'm not going to spend too much time here either. +[2025-06-01 19:13:25] Speaker 5: because we are evolving how this is going to go right now the way we will release it is this will +[2025-06-01 19:13:25] Speaker 5: deploy as a we're not a kubernetes operator yet but you have to basically deploy a dedicated +[2025-06-01 19:13:25] Speaker 5: OKE cluster with just CPU nodes, which installs Prometheus Grafana open source and our control. +[2025-06-01 19:13:25] Speaker 5: plane api as well as portal right those are all the components that come as part of this +[2025-06-01 19:13:25] Speaker 5: And this is that footprint. +[2025-06-01 19:13:27] Speaker 5: We run a local Postgres for Profana storage. +[2025-06-01 19:13:30] Speaker 5: We also use Postgres for... +[2025-06-01 19:13:25] Speaker 5: kind of operating these things. +[2025-06-01 19:13:27] Speaker 5: This is our kind of MVP minus one approach, right? +[2025-06-01 19:13:32] Speaker 5: And once this... +[2025-06-01 19:13:25] Speaker 5: becomes a service this will start looking differently any questions so far +[2025-06-01 19:13:25] Speaker 1: I have a few questions. +[2025-06-01 19:13:26] Speaker 1: Do you prefer we wait till the end? +[2025-06-01 19:13:25] Speaker 5: Yeah, let's take it, Zizal. +[2025-06-01 19:13:26] Speaker 5: Or if you want to see the demo and then ask those questions, it's up to you. +[2025-06-01 19:13:25] Speaker 1: I guess let's do the demo and then we can come back to questions. +[2025-06-01 19:13:28] Speaker 1: Maybe that will answer some of it. +[2025-06-01 19:13:25] Speaker 1: questions perfect yep all right so the the way we we know you can install this is we you will +[2025-06-01 19:13:25] Speaker 5: have a GitHub repo where it could go and say deploy this and it sets up a Kubernetes cluster. +[2025-06-01 19:13:25] Speaker 5: Kubernetes cluster setup and all of the Grafana Prometeas, +[2025-06-01 19:13:29] Speaker 5: everything is installed. +[2025-06-01 19:13:25] Speaker 5: in your primary OSID or your tenancy region, +[2025-06-01 19:13:29] Speaker 5: you will get access to a portal like this. +[2025-06-01 19:13:25] Speaker 5: right we call it corino lens but it is basically the lens so once you are here +[2025-06-01 19:13:25] Speaker 5: right it will go it has the ability to do a multi-region query in a single api +[2025-06-01 19:13:25] Speaker 5: So everything is API first, you have portal and there is a REST API endpoint you can interact. +[2025-06-01 19:13:25] Speaker 5: all of that is deployed in your tenancy so you can lock it down if you like to +[2025-06-01 19:13:25] Speaker 5: Right now we run on a public domain just for a demo, but everything in here. +[2025-06-01 19:13:30] Speaker 5: So if you see here. +[2025-06-01 19:13:25] Speaker 5: This is done. If I refresh this page, it's going to come back with +[2025-06-01 19:13:25] Speaker 5: It's doing all the regions that you have subscribed in your OSIT or Oracle tenancy. +[2025-06-01 19:13:25] Speaker 5: Then it comes back and tells you, here are all the instances that I can monitor. +[2025-06-01 19:13:28] Speaker 5: These are basically... +[2025-06-01 19:13:25] Speaker 5: just GPU instances that are running okay if you have new instances running you can also add +[2025-06-01 19:13:25] Speaker 5: what we call kind of plug-in. +[2025-06-01 19:13:27] Speaker 5: So it's a plug-in model. +[2025-06-01 19:13:28] Speaker 5: You already say, okay, start monitoring. +[2025-06-01 19:13:25] Speaker 5: these to OCI lengths. And so if you look at here, I have Frankfurt and Ashburn machines showing. +[2025-06-01 19:13:25] Speaker 5: up and it allows me to combine all of this and say they're all part of the same team or same +[2025-06-01 19:13:25] Speaker 5: experiment or same way i want to monitor and i can create what's called monitoring right +[2025-06-01 19:13:25] Speaker 5: So before I go there, right, how can you start monitoring this? +[2025-06-01 19:13:29] Speaker 5: If you have an existing instance... +[2025-06-01 19:13:25] Speaker 5: running or a new instance being provisioned you have to just include this script for now right +[2025-06-01 19:13:25] Speaker 5: we will we will change the way it is we're going to make it a native oci plugin +[2025-06-01 19:13:25] Speaker 5: once we upstream all of this to our plugin framework. +[2025-06-01 19:13:27] Speaker 5: But for now, it's basically a tarz file. +[2025-06-01 19:13:25] Speaker 5: and it's a Golang-based code primarily. +[2025-06-01 19:13:28] Speaker 5: The health checks are all Python-based. +[2025-06-01 19:13:25] Speaker 5: And this is basically end of the day, just all these metrics are composed and pushed. +[2025-06-01 19:13:25] Speaker 5: through Prometheus Push gateway, +[2025-06-01 19:13:27] Speaker 5: which is getting scraped by Push. +[2025-06-01 19:13:28] Speaker 5: So architecture. +[2025-06-01 19:13:25] Speaker 5: this is nothing very different or rocket science here right so it's very simple +[2025-06-01 19:13:25] Speaker 5: straightforward, but what it runs and how it runs is what the value we are adding. +[2025-06-01 19:13:25] Speaker 5: This is like a cloud in its script as well. +[2025-06-01 19:13:27] Speaker 5: You can add it as part of a new provisioning. +[2025-06-01 19:13:25] Speaker 5: instance automatically every new instance that comes up has monitoring enabled and everything is +[2025-06-01 19:13:25] Speaker 5: running at the host level as a system service it's not operating at the kubernetes layers or at the +[2025-06-01 19:13:25] Speaker 5: the Slurm application layers. +[2025-06-01 19:13:27] Speaker 5: It's running directly on the bare metal host. +[2025-06-01 19:13:30] Speaker 5: And we felt that was. +[2025-06-01 19:13:25] Speaker 5: the ideal way and the right place for us to remove specific implementations to Kubernetes. +[2025-06-01 19:13:25] Speaker 5: or Slurm or others. +[2025-06-01 19:13:28] Speaker 5: So we can also scan Kubernetes clusters you have running +[2025-06-01 19:13:31] Speaker 5: or you have RDMA. +[2025-06-01 19:13:25] Speaker 5: cluster networks or what we call compute clusters and allow you to also monitor that. +[2025-06-01 19:13:25] Speaker 5: And when you check the whole Kubernetes cluster, all the GPU nodes under it are automatically +[2025-06-01 19:13:30] Speaker 5: scanned. +[2025-06-01 19:13:25] Speaker 5: and add it to the monitoring for us. +[2025-06-01 19:13:27] Speaker 5: That's the experience. +[2025-06-01 19:13:30] Speaker 5: Okay. +[2025-06-01 19:13:30] Speaker 5: So I'll quickly show you how... +[2025-06-01 19:13:25] Speaker 5: you know this this is basically a bare minimum um portal uh access we have everything is api +[2025-06-01 19:13:25] Speaker 5: first so you get exactly what you're doing here you could you could do that uh through +[2025-06-01 19:13:25] Speaker 5: REST API endpoints. So once this loads up, because it's doing like a close to 10 region query, +[2025-06-01 19:13:25] Speaker 5: takes roughly six seconds but six to ten seconds we will we'll make it better with some cash +[2025-06-01 19:13:25] Speaker 5: So, yeah, no, we have close to 30, 30 plus machines, right, Joleta? +[2025-06-01 19:13:33] Speaker 5: I think it's performing. +[2025-06-01 19:13:25] Speaker 5: okay but you know joletta is working hard to see how she can do parallel um parallelization +[2025-06-01 19:13:25] Speaker 5: of some of these runs and get back. +[2025-06-01 19:13:27] Speaker 5: But we got them better. +[2025-06-01 19:13:29] Speaker 5: We started from 30 seconds. +[2025-06-01 19:13:25] Speaker 5: by now at six seconds, but we'll keep pushing. +[2025-06-01 19:13:29] Speaker 5: All right. So it allows me to create a... +[2025-06-01 19:13:25] Speaker 5: monitoring ring, it's like just an arbitrary way for you to +[2025-06-01 19:13:28] Speaker 5: combine these resources and say +[2025-06-01 19:13:25] Speaker 5: I want a dedicated dashboard for this, +[2025-06-01 19:13:27] Speaker 5: these set of machines, right? +[2025-06-01 19:13:29] Speaker 5: That's what you. +[2025-06-01 19:13:25] Speaker 5: allows you to create a ring and if you go to monitoring rings here and you have OK, +[2025-06-01 19:13:30] Speaker 5: this is. +[2025-06-01 19:13:25] Speaker 5: for ML training, this is for the team, this is for cost center, whatever, however you +[2025-06-01 19:13:25] Speaker 5: want to use this for kind of bundling all the things. Right? And the thing about +[2025-06-01 19:13:25] Speaker 5: this is you can always come back and add more instances or remove the instances if they go +[2025-06-01 19:13:25] Speaker 5: offline or you have to turn back in whatever right so so every every ring you create monitor +[2025-06-01 19:13:25] Speaker 5: ring you create comes with a dedicated Grafana board which includes all the hosts that are +[2025-06-01 19:13:25] Speaker 5: part of this and I'm going to click you through the Grafana. +[2025-06-01 19:13:31] Speaker 5: So what you see here at the first part of +[2025-06-01 19:13:25] Speaker 5: this is the health summary of all the compute nodes that are part of this, right? So we added more nodes. +[2025-06-01 19:13:25] Speaker 5: which you're in this part just to show you demo, right? +[2025-06-01 19:13:27] Speaker 5: So if you see the first board here is this is the. +[2025-06-01 19:13:25] Speaker 5: actual health check of all the performance related checks that we did on the host when you activate +[2025-06-01 19:13:25] Speaker 5: the plugin we will also allow you to run these on demand so if you look at the categories here +[2025-06-01 19:13:25] Speaker 5: compute throughput, temperature check, all these check, check, check, +[2025-06-01 19:13:28] Speaker 5: model MFU based stuff. +[2025-06-01 19:13:25] Speaker 5: All of those are readily available for you and we will very soon have a link that will exactly take +[2025-06-01 19:13:25] Speaker 5: you to a JSON that is very much deeper on all the tests. This is basically the JSON that the health check. +[2025-06-01 19:13:25] Speaker 5: script create like what's the bandwidth achieved the tflops achieved for each of the node which +[2025-06-01 19:13:25] Speaker 5: GPU ran this for example right this is the GPU ID of an H100 we ran this is actually the JSON document +[2025-06-01 19:13:25] Speaker 5: you'll have access to this as well as the logs piece uh but in a grafana it looks +[2025-06-01 19:13:25] Speaker 5: Now, it's an easier way for you to digest all of this. +[2025-06-01 19:13:27] Speaker 5: Let me put it up. +[2025-06-01 19:13:25] Speaker 5: end of the day it's the same data but everything is a very detailed data that you may +[2025-06-01 19:13:25] Speaker 5: be looking for um than just a summary like this okay both are available okay so we will add more +[2025-06-01 19:13:25] Speaker 5: filters when it allows you to you know go by region and check the gpu types if you have mi300x +[2025-06-01 19:13:25] Speaker 5: So Osaka does not have, maybe I think other machines in Chicago have MI300X. +[2025-06-01 19:13:25] Speaker 5: see here yes you can see my 300x instance um so yeah these are all of both uh my 300x +[2025-06-01 19:13:25] Speaker 5: as well as GPU from NVidia are fully monitored. +[2025-06-01 19:13:29] Speaker 5: And we have health checks running for both. +[2025-06-01 19:13:25] Speaker 5: Okay, so all of this is the data that you may be interested in. +[2025-06-01 19:13:25] Speaker 5: that allows you to kind of baseline and benchmark all of this. Okay. So all of these metrics are +[2025-06-01 19:13:25] Speaker 5: natively available in Prometheus and if I have to show you so we push with OCI +[2025-06-01 19:13:25] Speaker 5: lens labels to this and this will allow you to extend it any further you like. +[2025-06-01 19:13:25] Speaker 5: custom boards if you like, or you want to merge left join, right join some datasets here to get to more. +[2025-06-01 19:13:25] Speaker 5: details everything is fully available that's just native krafanov right this is the lcl lens +[2025-06-01 19:13:25] Speaker 5: health check stuff too. There is quite a bit of +[2025-06-01 19:13:31] Speaker 5: health. Yeah, this is the health summary. +[2025-06-01 19:13:25] Speaker 5: So all of these metrics, the only thing that we are adding, +[2025-06-01 19:13:27] Speaker 5: this comes out as OCI lens. +[2025-06-01 19:13:25] Speaker 5: anything that comes directly from the vendor, +[2025-06-01 19:13:26] Speaker 5: like AMD or NVIDIA come as is, right? +[2025-06-01 19:13:29] Speaker 5: DCGM append it. +[2025-06-01 19:13:25] Speaker 5: So one additional thing we also did is if you know, let me go back here. +[2025-06-01 19:13:25] Speaker 5: So this is the overall health board. +[2025-06-01 19:13:28] Speaker 5: And there's another board that we bring is this is. +[2025-06-01 19:13:25] Speaker 5: per host data right this is the actual gpu rdma metrics uh you know you know node health and all +[2025-06-01 19:13:25] Speaker 5: that. Additional things that we are trying to add is if you come here and look at this table. +[2025-06-01 19:13:25] Speaker 5: We will improve the UI a little bit, +[2025-06-01 19:13:27] Speaker 5: but there is something called host metadata survey. +[2025-06-01 19:13:25] Speaker 5: It's basically a local service endpoint that we hit +[2025-06-01 19:13:28] Speaker 5: and we fetch the serial number because if the- +[2025-06-01 19:13:25] Speaker 5: host is underperforming and you have to turn back it in for repair. +[2025-06-01 19:13:29] Speaker 5: You know, some data that we have. +[2025-06-01 19:13:25] Speaker 5: going to publish here is going to be super helpful. +[2025-06-01 19:13:27] Speaker 5: And we are looking at how to put an agent. +[2025-06-01 19:13:25] Speaker 5: AI on top of this, right? Let's +[2025-06-01 19:13:26] Speaker 5: shortcut that, right? So we +[2025-06-01 19:13:28] Speaker 5: will instead of throwing more +[2025-06-01 19:13:25] Speaker 5: more boards, we will kind of curate an experience where if you see a underperforming node or a bad, +[2025-06-01 19:13:25] Speaker 5: node. We may automate some of that to see if we could automatically raise a ticket or turn the +[2025-06-01 19:13:25] Speaker 5: machine in or invoke folks in the support team to help you guys out. +[2025-06-01 19:13:30] Speaker 5: So that's the. +[2025-06-01 19:13:25] Speaker 5: future roadmap. So yeah, everything here, we're going to add more stuff like what's the +[2025-06-01 19:13:25] Speaker 5: the huda driver you're running what's the kernel versions you're running um what's the OS +[2025-06-01 19:13:25] Speaker 5: attached to, what's the Rockham driver version you're running, all of those data will be fully +[2025-06-01 19:13:25] Speaker 5: available everything related to metadata will be published here and you can continue to +[2025-06-01 19:13:25] Speaker 5: use that instead of you SSHing into the machine, +[2025-06-01 19:13:28] Speaker 5: running some commands and all of that. +[2025-06-01 19:13:25] Speaker 5: So, readily available. +[2025-06-01 19:13:27] Speaker 5: Everything else you see here is something you may have been using already. +[2025-06-01 19:13:25] Speaker 5: which is power, usage, temperature, utilization, +[2025-06-01 19:13:28] Speaker 5: all that stuff. +[2025-06-01 19:13:30] Speaker 5: Okay. +[2025-06-01 19:13:31] Speaker 5: So in an actual, this- +[2025-06-01 19:13:25] Speaker 5: this is this is what it is right so at least our first iteration of um can we get a monitor +[2025-06-01 19:13:25] Speaker 5: going which is native and health checks going which is what lci published health checks and +[2025-06-01 19:13:25] Speaker 5: just always vendor specified. So I'm gonna stop here and see if you folks have questions. +[2025-06-01 19:13:25] Speaker 5: and also talk about some feedback if you have for us. +[2025-06-01 19:13:32] Speaker 1: Thanks for the demo. +[2025-06-01 19:13:33] Speaker 1: I have a few... +[2025-06-01 19:13:25] Speaker 1: questions um so i think you mentioned in the first slide one of the things being like um you know +[2025-06-01 19:13:25] Speaker 1: Kubernetes native, cloud native, +[2025-06-01 19:13:26] Speaker 1: which was one of the things we discussed previously +[2025-06-01 19:13:28] Speaker 1: as being one of the... +[2025-06-01 19:13:25] Speaker 1: key things really important for us um what other levels of integrations with oke are you planning +[2025-06-01 19:13:25] Speaker 1: at all, if any. +[2025-06-01 19:13:26] Speaker 1: Things that top of mind would be super helpful +[2025-06-01 19:13:28] Speaker 1: for us would be being able to +[2025-06-01 19:13:25] Speaker 1: surface some of these unhealthy node conditions to OKE or to the actual Kubernetes object. +[2025-06-01 19:13:25] Speaker 1: so that we can use that in our tooling. +[2025-06-01 19:13:27] Speaker 1: And then, you know, when we're launching, submitting jobs. +[2025-06-01 19:13:25] Speaker 1: being able to detect that a node is unhealthy. +[2025-06-01 19:13:28] Speaker 1: The other thing I think is like the monitoring. +[2025-06-01 19:13:25] Speaker 1: the rings that you mentioned, I don't think that would be super helpful for us because we're not +[2025-06-01 19:13:25] Speaker 1: assigning nodes to people or teams. +[2025-06-01 19:13:28] Speaker 1: Instead, we're extremely dynamic. +[2025-06-01 19:13:31] Speaker 1: So we have... +[2025-06-01 19:13:25] Speaker 1: you know, super cluster of thousands of GPUs. +[2025-06-01 19:13:27] Speaker 1: And those nodes get used. +[2025-06-01 19:13:30] Speaker 1: We use Q, which is a native. +[2025-06-01 19:13:25] Speaker 1: Kubernetes project to schedule workloads. +[2025-06-01 19:13:30] Speaker 1: So at any given time, we want to know, like, this +[2025-06-01 19:13:25] Speaker 1: job is running with, you know, 128 GPUs, are any of the nodes unhealthy in this job? And so that job +[2025-06-01 19:13:25] Speaker 1: might be using a different set of nodes than another job would be for the same team the next +[2025-06-01 19:13:25] Speaker 1: day. So something like that would be helpful. Okay, wonderful. I think for the answer for the first +[2025-06-01 19:13:25] Speaker 5: one is uh we are working with okay team i think okay team also has added some features lately where +[2025-06-01 19:13:25] Speaker 5: they can detect an unhealthy gpu node and start tagging them as unallocatable uh for some some +[2025-06-01 19:13:25] Speaker 5: the checks that they ran. +[2025-06-01 19:13:27] Speaker 5: So we are actually working with them +[2025-06-01 19:13:29] Speaker 5: to see if anything... +[2025-06-01 19:13:25] Speaker 5: we find in our health check uh that uh that says that you know this node one of the gpus in the +[2025-06-01 19:13:25] Speaker 5: node is continuous to underperform and communicate back on unschedulable. +[2025-06-01 19:13:25] Speaker 5: we will have that integration point in the future for sure. +[2025-06-01 19:13:29] Speaker 5: Right. So, and the metrics from. +[2025-06-01 19:13:25] Speaker 5: OKE directly. So if there are part level, position volume level and lower +[2025-06-01 19:13:25] Speaker 5: of good healthy prometheus metrics that come we can directly pipe all of those into this lens with +[2025-06-01 19:13:25] Speaker 5: added set of things from our side to kind of consolidate and give you that one view that means +[2025-06-01 19:13:25] Speaker 5: be needed. So that definitely is the plan, but we are looking into, okay, now a customer has to run. +[2025-06-01 19:13:25] Speaker 5: a Prometheus instance in a Kubernetes cluster and that complication is what we're trying to do. +[2025-06-01 19:13:25] Speaker 5: tackle the next feature set. +[2025-06-01 19:13:29] Speaker 1: Okay, makes sense. +[2025-06-01 19:13:30] Speaker 1: Yeah, just to reemphasize. +[2025-06-01 19:13:25] Speaker 1: this is like probably the most important thing for us like really being able to +[2025-06-01 19:13:25] Speaker 1: because we want to be able to do this automatically, right? +[2025-06-01 19:13:28] Speaker 1: If a GPU breaks overnight, we don't want... +[2025-06-01 19:13:25] Speaker 1: someone to have to, you know, go do something, go look at a Grafana dashboard we want. +[2025-06-01 19:13:25] Speaker 1: our jobs to automatically react to these things. +[2025-06-01 19:13:28] Speaker 5: Yeah. So I think that that brings another second. +[2025-06-01 19:13:25] Speaker 5: question or even feedback you had for us is like the static way of creating monitoring ranks is +[2025-06-01 19:13:25] Speaker 5: not ideal we fully hear you uh i think because of the dynamic if you're using q where you know +[2025-06-01 19:13:25] Speaker 5: you get it booted out based on what team and priority and cohort and all of those stuff are. +[2025-06-01 19:13:25] Speaker 5: We will integrate with OKE and the way we will look at is where the job is running or where +[2025-06-01 19:13:25] Speaker 5: the deployment is running at a point of time and try to dynamically what we call add more tags to +[2025-06-01 19:13:25] Speaker 5: metadata that's coming in to exactly relate to what experiment who ran it when when did it +[2025-06-01 19:13:25] Speaker 5: complete timestamp it and try to pull those in. +[2025-06-01 19:13:28] Speaker 5: So it's a thing that we are working. +[2025-06-01 19:13:25] Speaker 5: towards, but we started with more non-Kubernetes-based experience, and then we would quick switch. +[2025-06-01 19:13:25] Speaker 5: move into the dynamic workload where we pull in the nodes and even the GPUs, right? +[2025-06-01 19:13:30] Speaker 5: Not full nodes. +[2025-06-01 19:13:25] Speaker 5: be running every experiment so we will try to say four out of that no GPUs were running for this +[2025-06-01 19:13:25] Speaker 5: experiment and that's what is aggregated into that board. +[2025-06-01 19:13:28] Speaker 5: We will, that's a good point. +[2025-06-01 19:13:25] Speaker 5: feedback and I think we will try to prioritize that. +[2025-06-01 19:13:27] Speaker 5: Yeah, I think you're not very far from. +[2025-06-01 19:13:25] Speaker 1: that honestly even without having to do like tagging of instances and trying to keep track of +[2025-06-01 19:13:25] Speaker 1: what's running what. I think you could do some queries in your dashboard that just... +[2025-06-01 19:13:25] Speaker 1: shows like your health metrics and then also use Kubernetes metrics and see like for +[2025-06-01 19:13:25] Speaker 1: any given job or deployment, like you said, show all the nodes of that job and show if +[2025-06-01 19:13:25] Speaker 1: of them are unhealthy so i think it's just a matter of like tying everything together +[2025-06-01 19:13:25] Speaker 5: Yeah, I fully agree. Actually, we started like that, by the way, and I think Joleta and... +[2025-06-01 19:13:25] Speaker 5: and most of my team knows that we started with a full Kubernetes-based experience, but we were nudged. +[2025-06-01 19:13:25] Speaker 5: on like well i think there are a lot of people who want a full monitoring and this is where we +[2025-06-01 19:13:25] Speaker 5: I think we have the both. We like to just pull it in like you said. +[2025-06-01 19:13:25] Speaker 5: and start relating where, where, what is running +[2025-06-01 19:13:28] Speaker 5: and start kind of creating that. +[2025-06-01 19:13:25] Speaker 5: key value pair mapping here. +[2025-06-01 19:13:25] Speaker 1: That's another question I'll have and then I'll pass it to like other. +[2025-06-01 19:13:25] Speaker 1: Luis, Ace, if you have any questions, is you mentioned health checks. Can you talk? +[2025-06-01 19:13:25] Speaker 1: a bit more about um how those would run like are you running anything that requires the workload +[2025-06-01 19:13:25] Speaker 1: to be idle and if so is it or the gpu note to be idle and if so is it going to be doing +[2025-06-01 19:13:25] Speaker 1: something like when nodes go idle to check the gpus or are these like running passively in the background +[2025-06-01 19:13:25] Speaker 5: So I'll ask Soumya to chime in as well on she's an ML engineer worked on the script. +[2025-06-01 19:13:25] Speaker 5: So we run this, we expect the GPUs to be idle and no workloads to be running. +[2025-06-01 19:13:25] Speaker 5: this is where the PyTorch is loading into the tensor so it's loading into the accelerator. +[2025-06-01 19:13:25] Speaker 5: running all these checks and coming back. +[2025-06-01 19:13:27] Speaker 5: So we expect no workloads to be running when this. +[2025-06-01 19:13:25] Speaker 5: this is running. We run it when you say you need to run it. +[2025-06-01 19:13:28] Speaker 5: Right now we have not automated it. +[2025-06-01 19:13:25] Speaker 5: And this is where we need feedback. +[2025-06-01 19:13:26] Speaker 5: Like, do you want to run at midnight, 12 o'clock every day? +[2025-06-01 19:13:25] Speaker 5: right you need that kind of no definitely not +[2025-06-01 19:13:25] Speaker 1: Yeah, if I'm an ML engineer, I'm like, okay, I know when my pipeline is at a pause where I say call. +[2025-06-01 19:13:25] Speaker 5: of my gpus are flushed out and idle i want to run the health checks for everything and make sure +[2025-06-01 19:13:25] Speaker 5: of that is good before we go to the next phase, right? Is that an experience you would look for? +[2025-06-01 19:13:25] Speaker 5: What we are thinking is we will give you a on-demand way for you to invoke health checkscripts. +[2025-06-01 19:13:25] Speaker 5: through a REST API. +[2025-06-01 19:13:26] Speaker 5: You have to just say +[2025-06-01 19:13:28] Speaker 5: when you want to run it +[2025-06-01 19:13:30] Speaker 5: and you on-demand run it. +[2025-06-01 19:13:25] Speaker 5: Because we don't know when your GPUs are idle, then they're not. +[2025-06-01 19:13:25] Speaker 1: Interesting. I was thinking like you might be able to figure that out like from a program. +[2025-06-01 19:13:25] Speaker 1: programmatic standpoint, you might be able to get that data and whenever they are idle. +[2025-06-01 19:13:25] Speaker 1: kick off like a workload that runs the Huff check. +[2025-06-01 19:13:30] Speaker 1: So the intuition behind creating +[2025-06-01 19:13:25] Speaker 3: this health check recipe is imagine a machine learning engineer or like you have a team that's +[2025-06-01 19:13:25] Speaker 3: going to do a multi-node training or like multi-node fine tuning, right? +[2025-06-01 19:13:29] Speaker 3: Before you go into doing... +[2025-06-01 19:13:25] Speaker 3: a large-scale training operation or like any kind of like workload that's going to take too much +[2025-06-01 19:13:25] Speaker 3: demand of the GPU itself. +[2025-06-01 19:13:27] Speaker 3: You'd want to run this health check before that to understand that +[2025-06-01 19:13:25] Speaker 3: health of your infrastructure right for example the way this health check is designed is it's based +[2025-06-01 19:13:25] Speaker 3: on just 10 0 matrix multiplication right so depending upon the matrix size you probably be +[2025-06-01 19:13:25] Speaker 3: loading like 818 like imagine like a matrix size of like 8192 by 818 +[2025-06-01 19:13:25] Speaker 3: too right you're going to be loading all these matrix into the gpu memory and we are going to +[2025-06-01 19:13:25] Speaker 3: pressurize and see how much your machine is able to take the computation throughput or like throat +[2025-06-01 19:13:25] Speaker 3: the power and things like that. So ideally you would want to do it when your machine is idle and to +[2025-06-01 19:13:25] Speaker 3: address your point yeah we could actually schedule and understand when the jobs are not running +[2025-06-01 19:13:25] Speaker 3: and we could just run the health check at that point. +[2025-06-01 19:13:27] Speaker 3: And we can detect when the machines are... +[2025-06-01 19:13:25] Speaker 3: Yeah, I think it'd be interesting to have the option to schedule it on demand, like as a... +[2025-06-01 19:13:25] Speaker 1: we hook like to the jobs like you said um but i don't i need to think more about this +[2025-06-01 19:13:25] Speaker 1: But I think, you know, my intuition is why not both, you know, like having a periodic check. +[2025-06-01 19:13:25] Speaker 1: that best effort runs when the node is sitting there doing nothing. +[2025-06-01 19:13:29] Speaker 1: So that if a node does go unhealthy, +[2025-06-01 19:13:25] Speaker 1: we can find out about it early and not wait for a job to get scheduled there to find out. +[2025-06-01 19:13:25] Speaker 5: Yeah, no, we'll take it as a good feedback. +[2025-06-01 19:13:28] Speaker 5: We have still, because we have to think. +[2025-06-01 19:13:25] Speaker 5: through a little bit on scheduling because it takes roughly five minutes for this head check +[2025-06-01 19:13:25] Speaker 5: to finish and in mix of this if your queue ends up scheduling another job uh they'll they'll +[2025-06-01 19:13:25] Speaker 5: it can't find an ideal GPU. So they go on pending. So we have to just think through this. +[2025-06-01 19:13:25] Speaker 5: Yeah, no, this would definitely, yes, this would definitely have to be an... +[2025-06-01 19:13:25] Speaker 1: interruptible workload so that our workloads can always schedule the priority. +[2025-06-01 19:13:25] Speaker 5: Exactly right you cannot boot this workload because this is running at the system +[2025-06-01 19:13:25] Speaker 5: level, right? And Kubernetes layers are at a much higher layer too. +[2025-06-01 19:13:25] Speaker 5: and we need to think of this a little bit but i think this is really good feedback um thank you +[2025-06-01 19:13:25] Speaker 1: Yeah, no, thank you. +[2025-06-01 19:13:26] Speaker 1: Louise Ace, you have anything? +[2025-06-01 19:13:25] Speaker 4: Mostly just echoing your thoughts. I mean, when we talk about node lifecycle, +[2025-06-01 19:13:25] Speaker 4: It's very much like an ongoing thing. +[2025-06-01 19:13:26] Speaker 4: It's not sort of a one off thing. +[2025-06-01 19:13:27] Speaker 4: So just, yeah. +[2025-06-01 19:13:25] Speaker 4: like having both elements and thinking about how nodes like proceed through. +[2025-06-01 19:13:25] Speaker 4: I don't want to say this process exactly, but like, I don't know, mostly just echoing Cecile's +[2025-06-01 19:13:25] Speaker 4: thoughts. Okay. Yeah. Thank you. So what, what we are trying to also avoid is before you shut this +[2025-06-01 19:13:25] Speaker 5: machine and turn it back into OCI to repair. We want to see what is the issue, right? +[2025-06-01 19:13:25] Speaker 5: deeper into what where is the concern is it a single gpu always coming back with a lower +[2025-06-01 19:13:25] Speaker 5: performance compared to the rest seven of them in the same node or is it consistent uh no +[2025-06-01 19:13:25] Speaker 5: RDMA nick flapping issues, errors that show up. +[2025-06-01 19:13:28] Speaker 5: And this is a start. +[2025-06-01 19:13:30] Speaker 5: And where we want to go. +[2025-06-01 19:13:25] Speaker 5: is if we start seeing a pattern of things which are very consistent with a lot of other customers +[2025-06-01 19:13:25] Speaker 5: and a lot of type of GPUs, +[2025-06-01 19:13:26] Speaker 5: we will come back with recommendations +[2025-06-01 19:13:28] Speaker 5: through an agentic AI type of app. +[2025-06-01 19:13:25] Speaker 5: flow where probably just a reboot may fix the things or it could be specific to a driver. +[2025-06-01 19:13:25] Speaker 5: the GPU driver you have, which has seen this incompatibility with the Mellanox drivers we have. +[2025-06-01 19:13:25] Speaker 5: example. So we are trying to learn from using the data as well as all the issues customers. +[2025-06-01 19:13:25] Speaker 5: give us to start recommending, right? +[2025-06-01 19:13:27] Speaker 5: So this is just a start for us to start collecting, +[2025-06-01 19:13:29] Speaker 5: but next approach for us is +[2025-06-01 19:13:25] Speaker 5: remove the noise and tell if there is really a issue with the host or if this is a transient issue. +[2025-06-01 19:13:25] Speaker 5: or not an issue with something related to an experiment that has been set up. +[2025-06-01 19:13:25] Speaker 5: workload is being scheduled on this or the topology that it's been deployed with. +[2025-06-01 19:13:29] Speaker 5: Any of those questions? +[2025-06-01 19:13:25] Speaker 5: Right, so that's where we want to get to. +[2025-06-01 19:13:25] Speaker 5: Awesome. So here's the log as well. +[2025-06-01 19:13:28] Speaker 5: This is this we'll build up more and more. +[2025-06-01 19:13:25] Speaker 5: stuff and it's all accessible either through portal or an api where you don't have to assess +[2025-06-01 19:13:25] Speaker 5: an instance. Because the scripts are going to be standardized that what you see against +[2025-06-01 19:13:25] Speaker 5: what OCI support sees everything will be consistent and maybe in the future once we have support team +[2025-06-01 19:13:25] Speaker 5: running full diagnosis they know where to narrow down the issue to like exact GPU or the NICs or +[2025-06-01 19:13:25] Speaker 5: anything instead of running this full you know six hours uh diagnosis on the host right +[2025-06-01 19:13:25] Speaker 5: So we're just trying to narrow down that time it takes to fix these machines too. +[2025-06-01 19:13:30] Speaker 5: So that will help. +[2025-06-01 19:13:25] Speaker 1: When you say consistent, is the idea that you give us a set of scripts and we run these? +[2025-06-01 19:13:25] Speaker 1: or they see the results of our health checks. +[2025-06-01 19:13:25] Speaker 1: So right now with the way it is integrated, the OCI support does not see the results, but you could just take... +[2025-06-01 19:13:25] Speaker 5: these results copy paste in your support ticket if you need to OCI support will have access +[2025-06-01 19:13:25] Speaker 5: the same health check tools and scripts that you are running. +[2025-06-01 19:13:29] Speaker 5: So it's a consistent tool. +[2025-06-01 19:13:25] Speaker 5: that you both are running, +[2025-06-01 19:13:26] Speaker 5: where what you have seen as the results, +[2025-06-01 19:13:29] Speaker 5: for example, the score, +[2025-06-01 19:13:25] Speaker 5: So here's the flops, for example. +[2025-06-01 19:13:28] Speaker 5: This is the duration, right? +[2025-06-01 19:13:30] Speaker 5: What you are seeing... +[2025-06-01 19:13:25] Speaker 5: when OCI support is trying to run this on their own. +[2025-06-01 19:13:28] Speaker 5: There is a lot of consistency. +[2025-06-01 19:13:25] Speaker 5: Right. And if you see this GPU consistently coming back with like, okay, it takes 31 seconds. +[2025-06-01 19:13:25] Speaker 5: It's easy to narrow down saying that GQ7 is something. +[2025-06-01 19:13:25] Speaker 5: that we need to request. +[2025-06-01 19:13:25] Speaker 1: I see. Yeah, I think it would be even better if we can get to a stage where they... +[2025-06-01 19:13:25] Speaker 1: like have access to historical results and they can just see things that are +[2025-06-01 19:13:28] Speaker 1: ZeeBrand. +[2025-06-01 19:13:25] Speaker 1: Yeah. +[2025-06-01 19:13:25] Speaker 1: instead of having to reproduce and copy pasting things. +[2025-06-01 19:13:25] Speaker 1: i fully agree i think that's our goal but we are not a oci service yet +[2025-06-01 19:13:25] Speaker 5: right and uh because it's all running the customer's tenancy uh there's obviously +[2025-06-01 19:13:25] Speaker 5: little bit of things about what we can collect, what we cannot. And we had to go through those. +[2025-06-01 19:13:25] Speaker 5: product approval phases. +[2025-06-01 19:13:28] Speaker 5: And then we will probably +[2025-06-01 19:13:29] Speaker 5: support and have access to this. +[2025-06-01 19:13:25] Speaker 1: Okay, makes sense. One other question for Prometheus. You mentioned at some point in the +[2025-06-01 19:13:25] Speaker 1: this slide something about setting up a CPU cluster or gate cluster to run Prometheus. +[2025-06-01 19:13:25] Speaker 1: can we reuse our existing primukes instead? +[2025-06-01 19:13:25] Speaker 5: Yes, yes, absolutely. You can use your existing. You need to just enable push gateway and add. +[2025-06-01 19:13:25] Speaker 5: push gateway configurations because that's how all of the scripts push their results and metrics. +[2025-06-01 19:13:25] Speaker 5: Okay, can you give a bit more detail about how this works? +[2025-06-01 19:13:25] Speaker 1: scripts push metrics or how does that work +[2025-06-01 19:13:25] Speaker 5: Yeah, Hrithika, can you elaborate on this? +[2025-06-01 19:13:25] Speaker 2: Hello? Can you hear me? +[2025-06-01 19:13:25] Speaker 5: Is VTK wrong? +[2025-06-01 19:13:25] Speaker 5: Yeah. +[2025-06-01 19:13:25] Speaker 2: All right. Could you give that question once again? +[2025-06-01 19:13:25] Speaker 1: It was just having a bit more detail about how the metrics get emitted. +[2025-06-01 19:13:25] Speaker 1: Because I understand there are scripts and then I'm just trying to figure out like how we could. +[2025-06-01 19:13:25] Speaker 1: scrape our own metrics. +[2025-06-01 19:13:25] Speaker 1: Oh yeah, so we basically build the node exporter, DCGM, AMD exporter, all of that depending on the GPU. +[2025-06-01 19:13:25] Speaker 2: it is and bring those up and there are also some metrics which are OCI specific which are +[2025-06-01 19:13:25] Speaker 2: return in a Go plugin and that plugin +[2025-06-01 19:13:29] Speaker 2: fits metrics to the push gateway at a regular interval. +[2025-06-01 19:13:25] Speaker 5: Yeah, all of the scripts here are... +[2025-06-01 19:13:27] Speaker 5: Does that make sense? +[2025-06-01 19:13:25] Speaker 5: push gateway if you see it here and +[2025-06-01 19:13:25] Speaker 5: And these are scraped by Prometheus +[2025-06-01 19:13:27] Speaker 5: based on Prometheus script config. +[2025-06-01 19:13:29] Speaker 5: And- +[2025-06-01 19:13:25] Speaker 5: Push gateway is part of a service that is default in Prometheus server install. +[2025-06-01 19:13:30] Speaker 5: And it's the same. +[2025-06-01 19:13:25] Speaker 5: mechanism. +[2025-06-01 19:13:27] Speaker 5: Gotcha. +[2025-06-01 19:13:28] Speaker 1: Okay. +[2025-06-01 19:13:28] Speaker 1: I'll look into this a bit more. +[2025-06-01 19:13:29] Speaker 1: I haven't used push gateway before. +[2025-06-01 19:13:25] Speaker 1: So that's probably where my gap comes from. +[2025-06-01 19:13:27] Speaker 1: Okay. +[2025-06-01 19:13:27] Speaker 5: Yeah, I think it's part of the setup. +[2025-06-01 19:13:25] Speaker 5: And if you go, if you run Prometheus through operator, +[2025-06-01 19:13:28] Speaker 5: Push Gateway is usually installed as a service. +[2025-06-01 19:13:25] Speaker 5: It should have its own service endpoint, +[2025-06-01 19:13:28] Speaker 5: and you have to just enable a few things. +[2025-06-01 19:13:25] Speaker 5: and you're good to go. Yeah. So, this is for instead of you going and reaching out. +[2025-06-01 19:13:25] Speaker 5: going and reaching out and pulling the metrics out which is where you need firewall exception +[2025-06-01 19:13:25] Speaker 5: and opening the ports and all of that. +[2025-06-01 19:13:27] Speaker 5: Here, any metrics you are generating, including DCGM, +[2025-06-01 19:13:25] Speaker 5: all of those are pool metrics, right? +[2025-06-01 19:13:27] Speaker 5: We push it directly to the push page. +[2025-06-01 19:13:25] Speaker 5: and your Prometheus is scraping push gateway basically. +[2025-06-01 19:13:29] Speaker 5: It's like an intermediate data store. +[2025-06-01 19:13:25] Speaker 5: All right. Any further questions? And we are also eager if you have any feedback. +[2025-06-01 19:13:25] Speaker 5: can think about we are thinking in next roughly two weeks you can you can have access to all of +[2025-06-01 19:13:25] Speaker 5: the health check scripts. +[2025-06-01 19:13:27] Speaker 5: We'll get access to your repo +[2025-06-01 19:13:29] Speaker 5: once you share your GitHub IDs. +[2025-06-01 19:13:25] Speaker 5: can poke around and really we are here to listen and and build something that that makes uh +[2025-06-01 19:13:25] Speaker 5: life easy right so we are here to listen and you know continue to iterate as fast as we +[2025-06-01 19:13:25] Speaker 5: can to give you what you're looking for. +[2025-06-01 19:13:27] Speaker 5: So. +[2025-06-01 19:13:28] Speaker 1: Thanks, Samar. +[2025-06-01 19:13:29] Speaker 1: I think overall this is, you know. +[2025-06-01 19:13:25] Speaker 1: something that we've been asking for. So really, really happy it's, it's getting there. +[2025-06-01 19:13:25] Speaker 1: and you all are focusing your efforts there. +[2025-06-01 19:13:26] Speaker 1: It's definitely heading the right direction. +[2025-06-01 19:13:25] Speaker 1: I think looking at what we have today that we built ourselves a best over the past year, +[2025-06-01 19:13:29] Speaker 1: it's definitely... +[2025-06-01 19:13:25] Speaker 1: not there yet. I think what we have right now is a bit more advanced just because we've built +[2025-06-01 19:13:25] Speaker 1: our own set of scripts and like we've iterated on them and we have all that automation. +[2025-06-01 19:13:25] Speaker 1: and integration with Kubernetes already. +[2025-06-01 19:13:27] Speaker 1: But I like where this is going and yeah, excited too. +[2025-06-01 19:13:25] Speaker 1: work with you all too. +[2025-06-01 19:13:26] Speaker 1: It's great. +[2025-06-01 19:13:27] Speaker 5: Okay. +[2025-06-01 19:13:28] Speaker 5: Yeah. +[2025-06-01 19:13:28] Speaker 5: We'll try to, +[2025-06-01 19:13:30] Speaker 5: right now, +[2025-06-01 19:13:31] Speaker 5: we are not overly +[2025-06-01 19:13:25] Speaker 5: Kubernetes heavy features and if we will probably in the next iteration or a milestone +[2025-06-01 19:13:25] Speaker 5: add all of those capabilities to bring Kubernetes metrics and that experience together. +[2025-06-01 19:13:25] Speaker 5: And hopefully, Sisal, I'll post that. +[2025-06-01 19:13:27] Speaker 5: You would be open to trying it out and giving us the feedback. +[2025-06-01 19:13:25] Speaker 5: and iterating like that. +[2025-06-01 19:13:29] Speaker 1: Sounds good. +[2025-06-01 19:13:32] Speaker 5: Thanks for making the time to give us a demo. +[2025-06-01 19:13:25] Speaker 1: Appreciate it. +[2025-06-01 19:13:25] Speaker 5: Thanks for spending your time with us as well. +[2025-06-01 19:13:28] Speaker 5: So anyone else, any questions? +[2025-06-01 19:13:25] Speaker 1: Thank you. +[2025-06-01 19:13:25] Speaker 1: Thank you. +[2025-06-01 19:13:25] Speaker 1: Thank you. +[2025-06-01 19:13:25] Speaker 1: Thank you. + +====== Summary ====== + +Key Points: + +* The OCI Lens initiative is currently in the incubation phase and is not yet an MVP or closer to it. +* The team is focused on continuous GPU and cluster-level monitoring of both NVIDIA and AMD GPUs. +* The solution includes active health checks and active monitoring, which is a huge investment. +* The team has built team-level tracking, which allows multiple teams to monitor their own subset of systems. +* The solution also includes cost tracking, which allows users to see how much computer resources they have used. + +Action Items: + +* The team will show a demo of the OCI Lens initiative, which allows users to monitor either single, bare metal, virtual machine instances, or a full OKE cluster or an HPC cluster. +* The team is working towards automatically fetching the nodes that are running an experiment based on Kubernetes scheduling. +* The team will demonstrate the difference between performance monitoring and health checks, which goes very close to the layers that an ML engineer would operate under. + +--- + +Key points: + +* PyTorch and JAX are being considered for performance testing. +* The approach is to push metrics and health check data to Prometheus and the central control plane. +* The origination of all metrics is within the network. +* There are plenty of metrics, including NVIDIA DCGM exporter, AMD SMI metrics, and RDMA metrics. +* Health checks include traditional disk IO usage and are adjusted based on performance seen. +* The architecture is evolving and will be deployed as a dedicated OKE cluster with CPU nodes. +* The footprint includes Prometheus, Grafana, open source, and control plane API. +* The demo will be shown, and questions can be asked after. + +Decisions: + +* PyTorch and JAX will be considered for performance testing. +* The approach is to push metrics and health check data to Prometheus and the central control plane. + +Action items: + +* None specified in the transcript. + +--- + +Key Points: + +* The meeting discussed a plug-in model for monitoring OCI lengths. +* To start monitoring, an existing instance or a new instance being provisioned must include a script. +* The architecture is simple and straightforward, with the monitoring running directly on the bare metal host. +* The monitoring can be scanned for Kubernetes clusters, RDMA cluster networks, or compute clusters. +* The experience is that when you check a Kubernetes cluster, all GPU nodes under it are automatically scanned and added to the monitoring. +* The meeting showed a portal for accessing the monitoring through REST API endpoints. +* The monitoring allows for the creation of a monitoring ring, which can be used to bundle all the things. +* Every monitoring ring comes with a dedicated Grafana board. +* The health summary of all compute nodes that are part of the monitoring was demonstrated. +* The health checks include performance-related checks that are done on the host when the plugin is activated. +* The health checks can be run on demand, and a link will soon be provided to a JSON that is deeper on all the tests. + +Decisions: + +* The monitoring will be started by including a script in an existing instance or a new instance being provisioned. +* The monitoring will run directly on the bare metal host. +* The monitoring can be scanned for Kubernetes clusters, RDMA cluster networks, or compute clusters. +* The monitoring allows for the creation of a monitoring ring, which can be used to bundle all the things. + +Action Items: + +* Provide a link to a JSON that is deeper on all the tests. + +--- + +Summary of Meeting: + +* The speaker presented a tool that provides detailed data on GPU usage, health, and performance. +* The tool is natively available in Prometheus and can be extended with custom labels. +* The speaker added that they are working on improving the UI and automating some tasks to help users identify and resolve issues more efficiently. +* The speaker mentioned that they are planning to add more data related to metadata, such as host metadata survey, to provide a more comprehensive view of the system. +* The speaker also mentioned that they are planning to add more boards to the tool to provide more information about the system. +* The speaker mentioned that they are planning to add more features to the tool, such as automating some tasks and improving the UI. +* The speaker mentioned that they are planning to add more data related to metadata, such as host metadata survey, to provide a more comprehensive view of the system. +* The speaker mentioned that they are planning to add more boards to the tool to provide more information about the system. +* The speaker mentioned that they are planning to add more features to the tool, such as automating some tasks and improving the UI. +* The speaker mentioned that they are planning to add more data related to metadata, such as host metadata survey, to provide a more comprehensive view of the system. +* The speaker mentioned that they are planning to add more boards to the tool to provide more information about the system. +* The speaker mentioned that they are planning to add more features to the tool, such as automating some tasks and improving the UI. +* The speaker mentioned that they are planning to add more data related to metadata, such as host metadata survey, to provide a more comprehensive view of the system. +* The speaker mentioned that they are planning to add more boards to the tool to provide more information about the system. +* The speaker mentioned that they are planning to add more features to the tool, such as automating some tasks and improving the UI. +* The speaker mentioned that they are planning to add more data related to metadata, such as host metadata survey, to provide a more comprehensive view of the system. +* The speaker mentioned that they are planning to add more boards to the tool to provide more information about the system. +* The speaker mentioned that they are planning to add more features to the tool, such as automating some tasks and improving the UI. +* The speaker mentioned that they are planning to add more data + +--- + +Key points: + +* The team is working on a solution to automatically detect and react to unhealthy GPUs in a Kubernetes cluster. +* The solution will involve integrating with the Open Kubernetes Engine (OKE) and dynamically adding tags to metadata. +* The team is also working on improving the static way of creating monitoring ranks. +* The team is looking into using Prometheus to monitor the health of GPUs in a Kubernetes cluster. + +Decisions: + +* The team will integrate with OKE to dynamically add tags to metadata. +* The team will prioritize the development of a solution to automatically detect and react to unhealthy GPUs. + +Action items: + +* The team will continue working on integrating with OKE and dynamically adding tags to metadata. +* The team will prioritize the development of a solution to automatically detect and react to unhealthy GPUs. + +--- + +Key Points: + +* The team is currently not automating the running of health checks on GPUs. +* Feedback is needed on when to run health checks, such as at midnight every day or on demand. +* The team is considering giving an on-demand way for ML engineers to invoke health check scripts through a REST API. +* The team is also considering creating a health check recipe that can be run before large-scale training operations to understand the health of the infrastructure. +* The team is considering scheduling health checks on demand, as well as having a periodic check that runs when the node is idle. +* The team is aware of the need for an interruptible workload so that workloads can always schedule their priority. +* The team is trying to learn from data and customer issues to start recommending specific fixes for GPU driver incompatibilities. + +Action Items: + +* Determine the best way to invoke health check scripts on demand. +* Create a health check recipe that can be run before large-scale training operations. +* Schedule health checks on demand as well as having a periodic check that runs when the node is idle. +* Develop an agentic AI-type app to learn from data and customer issues to start recommending specific fixes for GPU driver incompatibilities. + +--- + +Key points: + +* The team is working on a new approach to diagnose issues with hosts or experiments scheduled on them. +* The goal is to narrow down the time it takes to fix these machines. +* The team is working on building a consistent tool that both OCI support and customers can use to diagnose issues. +* The team is also working on improving the historical results and access to them for customers. +* The team is exploring the possibility of reusing existing primukes instead of setting up a CPU cluster or gate cluster to run Prometheus. +* The team is working on building node exporter, DCGM, AMD exporter, and other metrics exporters depending on the GPU. +* The team is also working on integrating OCI-specific metrics into Prometheus. +* The team is using push gateway to collect metrics from scripts and scrape them by Prometheus. + +Decisions: + +* The team will work on building a consistent tool that both OCI support and customers can use to diagnose issues. +* The team will explore the possibility of reusing existing primukes instead of setting up a CPU cluster or gate cluster to run Prometheus. + +Action items: + +* The team will work on building node exporter, DCGM, AMD exporter, and other metrics exporters depending on the GPU. +* The team will integrate OCI-specific metrics into Prometheus. +* The team will use push gateway to collect metrics from scripts and scrape them by Prometheus. + +--- + +Key points: + +* Push Gateway is a service that should have its own service endpoint. +* Any metrics generated, including DCGM, are pool metrics that are pushed directly to the Push Gateway. +* Prometheus is used to scrape the Push Gateway, acting as an intermediate data store. +* Access to health check scripts will be provided in roughly two weeks once GitHub IDs are shared. +* The team is focused on building something that makes life easy for the user. +* The Push Gateway is not currently Kubernetes heavy, but Kubernetes capabilities may be added in the next iteration or milestone. +* Sisal is open to trying out and giving feedback on the Push Gateway. + +Decisions: + +* The Push Gateway should be installed as a service with its own service endpoint. +* Any metrics generated should be pushed directly to the Push Gateway. +* Prometheus should be used to scrape the Push Gateway. +* Access to health check scripts will be provided in roughly two weeks. + +Action items: + +* Enable a few things for the Push Gateway. +* Share GitHub IDs to access health check scripts. +* Iterate on the Push Gateway to add Kubernetes capabilities. +* Try out and give feedback on the Push Gateway. \ No newline at end of file diff --git a/docs/whisper_transcription/Examples/Test3/transcription_log_20250601_191325.log b/docs/whisper_transcription/Examples/Test3/transcription_log_20250601_191325.log new file mode 100644 index 0000000..0f9da2b --- /dev/null +++ b/docs/whisper_transcription/Examples/Test3/transcription_log_20250601_191325.log @@ -0,0 +1,4490 @@ +2025-06-01 19:13:25,860 - INFO - Total audio files to process: 1 +2025-06-01 19:13:25,860 - INFO - Validating: /home/ubuntu/whisper/dd_amar/audio1788670787.m4a +2025-06-01 19:13:28,776 - INFO - Duration of /home/ubuntu/whisper/dd_amar/audio1788670787.m4a: 2310.76s +2025-06-01 19:13:28,777 - INFO - Processing: /home/ubuntu/whisper/dd_amar/audio1788670787.m4a +2025-06-01 19:13:31,063 - INFO - Running global speaker diarization... +2025-06-01 19:13:31,063 - INFO - Running speaker diarization on: /tmp/tmpla1p30kt.wav +2025-06-01 19:13:31,441 - DEBUG - Registered checkpoint save hook for _speechbrain_save +2025-06-01 19:13:31,441 - DEBUG - Registered checkpoint load hook for _speechbrain_load +2025-06-01 19:13:31,441 - DEBUG - Registered checkpoint save hook for save +2025-06-01 19:13:31,441 - DEBUG - Registered checkpoint load hook for load +2025-06-01 19:13:31,584 - INFO - Applied quirks (see `speechbrain.utils.quirks`): [disable_jit_profiling, allow_tf32] +2025-06-01 19:13:31,584 - INFO - Excluded quirks specified by the `SB_DISABLE_QUIRKS` environment (comma-separated list): [] +2025-06-01 19:13:31,584 - DEBUG - Registered checkpoint save hook for _save +2025-06-01 19:13:31,585 - DEBUG - Registered checkpoint load hook for _recover +2025-06-01 19:14:18,636 - INFO - Diarization returned 205 segments. Dumped to output_test3/diarization_debug.json. +2025-06-01 19:14:18,636 - INFO - Diarized segment: 0.03s - 0.05s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 0.06s - 0.22s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 3.37s - 55.79s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 56.68s - 110.29s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 63.92s - 64.16s -> SPEAKER_00 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 113.04s - 114.12s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 114.61s - 124.03s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 124.38s - 140.62s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 140.89s - 141.04s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 141.56s - 148.04s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 148.35s - 168.14s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 168.41s - 185.03s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 185.93s - 202.02s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 202.31s - 209.84s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 210.31s - 211.86s -> SPEAKER_04 +2025-06-01 19:14:18,636 - INFO - Diarized segment: 212.54s - 220.84s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 222.21s - 242.56s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 242.76s - 256.89s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 258.15s - 281.19s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 281.46s - 286.84s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 287.06s - 290.42s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 291.55s - 330.31s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 330.49s - 349.14s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 350.44s - 384.06s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 384.31s - 388.48s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 388.85s - 390.01s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 392.64s - 432.89s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 435.68s - 476.92s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 478.64s - 519.78s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 519.97s - 523.07s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 523.22s - 527.76s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 529.47s - 530.41s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 533.52s - 536.59s -> SPEAKER_00 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 535.22s - 535.31s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 536.96s - 540.84s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 542.21s - 542.24s -> SPEAKER_00 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 542.24s - 542.36s -> SPEAKER_01 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 542.36s - 542.65s -> SPEAKER_00 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 543.14s - 547.54s -> SPEAKER_00 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 547.91s - 548.62s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 548.77s - 637.10s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 637.45s - 641.43s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 641.96s - 643.69s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 644.18s - 661.09s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 661.70s - 664.69s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 664.94s - 667.64s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 668.01s - 678.47s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 679.05s - 691.86s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 693.39s - 693.56s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 695.04s - 708.41s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 708.81s - 750.16s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 751.68s - 756.17s -> SPEAKER_04 +2025-06-01 19:14:18,637 - INFO - Diarized segment: 756.59s - 768.38s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 769.51s - 780.20s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 781.04s - 783.47s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 783.71s - 806.05s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 806.08s - 858.46s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 861.16s - 939.38s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 941.30s - 941.49s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 941.76s - 942.48s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 942.84s - 971.00s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 972.62s - 993.41s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 994.22s - 995.86s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 996.60s - 1013.78s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1014.45s - 1015.06s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1016.13s - 1018.67s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1019.52s - 1020.19s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1022.50s - 1058.13s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1058.55s - 1060.56s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1060.83s - 1087.03s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1087.46s - 1095.67s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1096.16s - 1116.26s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1117.61s - 1151.36s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1152.90s - 1170.41s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1172.05s - 1178.19s -> SPEAKER_04 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1182.14s - 1191.22s -> SPEAKER_00 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1191.63s - 1199.08s -> SPEAKER_00 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1199.61s - 1215.94s -> SPEAKER_00 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1199.73s - 1200.00s -> SPEAKER_01 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1216.45s - 1219.54s -> SPEAKER_00 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1220.13s - 1226.05s -> SPEAKER_00 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1226.19s - 1234.13s -> SPEAKER_00 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1234.61s - 1251.36s -> SPEAKER_00 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1238.35s - 1238.76s -> SPEAKER_01 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1241.15s - 1241.17s -> SPEAKER_01 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1249.68s - 1250.05s -> SPEAKER_01 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1251.63s - 1253.44s -> SPEAKER_00 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1252.49s - 1252.78s -> SPEAKER_01 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1253.69s - 1262.91s -> SPEAKER_00 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1263.41s - 1269.23s -> SPEAKER_00 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1269.79s - 1270.13s -> SPEAKER_00 +2025-06-01 19:14:18,638 - INFO - Diarized segment: 1270.79s - 1272.36s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1272.84s - 1276.10s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1276.34s - 1312.79s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1313.36s - 1314.17s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1314.80s - 1351.25s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1353.22s - 1366.30s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1366.79s - 1367.97s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1368.19s - 1373.23s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1374.08s - 1394.02s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1394.07s - 1417.35s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1397.03s - 1397.38s -> SPEAKER_01 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1397.40s - 1397.42s -> SPEAKER_01 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1398.87s - 1399.02s -> SPEAKER_01 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1399.05s - 1399.07s -> SPEAKER_01 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1417.40s - 1426.04s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1426.49s - 1440.01s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1440.62s - 1442.44s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1441.14s - 1441.53s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1443.05s - 1446.45s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1447.18s - 1475.41s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1475.87s - 1477.13s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1477.47s - 1501.72s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1496.76s - 1496.79s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1504.81s - 1532.01s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1532.72s - 1570.08s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1570.57s - 1576.48s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1570.84s - 1571.73s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1572.41s - 1575.13s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1575.65s - 1605.27s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1605.97s - 1610.07s -> SPEAKER_04 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1611.56s - 1618.23s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1618.60s - 1626.58s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1619.36s - 1620.30s -> SPEAKER_01 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1620.55s - 1620.71s -> SPEAKER_01 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1620.71s - 1621.23s -> SPEAKER_02 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1621.23s - 1621.26s -> SPEAKER_01 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1625.82s - 1626.49s -> SPEAKER_02 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1628.28s - 1628.97s -> SPEAKER_00 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1628.42s - 1640.11s -> SPEAKER_02 +2025-06-01 19:14:18,639 - INFO - Diarized segment: 1640.60s - 1650.22s -> SPEAKER_02 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1650.95s - 1662.15s -> SPEAKER_02 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1662.64s - 1666.49s -> SPEAKER_02 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1666.74s - 1677.49s -> SPEAKER_02 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1677.88s - 1684.58s -> SPEAKER_02 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1685.08s - 1697.23s -> SPEAKER_02 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1698.82s - 1705.47s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1705.52s - 1707.29s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1705.57s - 1705.91s -> SPEAKER_01 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1707.75s - 1723.29s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1712.13s - 1712.42s -> SPEAKER_01 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1723.86s - 1724.79s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1725.87s - 1745.53s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1743.32s - 1745.50s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1746.07s - 1746.95s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1746.15s - 1746.75s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1747.20s - 1752.06s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1747.71s - 1747.83s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1752.25s - 1766.03s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1769.48s - 1772.23s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1776.19s - 1791.30s -> SPEAKER_03 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1793.08s - 1799.31s -> SPEAKER_03 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1801.02s - 1875.75s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1875.97s - 1880.63s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1881.00s - 1882.07s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1884.02s - 1886.50s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1889.24s - 1932.02s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1932.72s - 1933.86s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1935.00s - 1943.78s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1944.42s - 1944.44s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1944.44s - 1960.08s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1944.47s - 1945.23s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1960.59s - 1963.45s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1963.52s - 1992.65s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 1993.44s - 2004.53s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 2001.14s - 2001.93s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 2003.31s - 2003.80s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 2004.60s - 2024.04s -> SPEAKER_04 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 2026.62s - 2040.47s -> SPEAKER_00 +2025-06-01 19:14:18,640 - INFO - Diarized segment: 2040.64s - 2051.74s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2053.65s - 2054.66s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2054.26s - 2061.40s -> SPEAKER_00 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2062.07s - 2065.21s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2070.26s - 2071.62s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2071.84s - 2073.09s -> SPEAKER_01 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2073.33s - 2073.39s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2073.39s - 2073.53s -> SPEAKER_00 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2073.53s - 2073.63s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2074.69s - 2076.94s -> SPEAKER_01 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2078.34s - 2088.73s -> SPEAKER_00 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2089.53s - 2099.43s -> SPEAKER_01 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2101.34s - 2109.76s -> SPEAKER_01 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2110.27s - 2114.11s -> SPEAKER_01 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2115.92s - 2119.75s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2118.21s - 2119.65s -> SPEAKER_01 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2120.24s - 2124.68s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2125.93s - 2140.02s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2141.94s - 2142.40s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2142.04s - 2148.27s -> SPEAKER_00 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2147.91s - 2162.95s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2165.77s - 2175.88s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2175.91s - 2185.48s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2185.97s - 2186.04s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2186.34s - 2190.03s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2190.79s - 2192.28s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2199.60s - 2231.55s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2232.80s - 2233.34s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2233.34s - 2244.56s -> SPEAKER_00 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2244.59s - 2266.73s -> SPEAKER_00 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2266.90s - 2268.99s -> SPEAKER_00 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2269.62s - 2292.38s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2294.64s - 2295.44s -> SPEAKER_00 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2297.29s - 2297.61s -> SPEAKER_04 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2297.46s - 2300.25s -> SPEAKER_00 +2025-06-01 19:14:18,641 - INFO - Diarized segment: 2300.25s - 2305.43s -> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 0: 0.03s - 0.05s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 1: 0.06s - 0.22s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 2: 3.37s - 55.79s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 3: 56.68s - 110.29s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 4: 63.92s - 64.16s --> SPEAKER_00 +2025-06-01 19:14:18,642 - INFO - [DIAR] 5: 113.04s - 114.12s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 6: 114.61s - 124.03s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 7: 124.38s - 140.62s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 8: 140.89s - 141.04s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 9: 141.56s - 148.04s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 10: 148.35s - 168.14s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 11: 168.41s - 185.03s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 12: 185.93s - 202.02s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 13: 202.31s - 209.84s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 14: 210.31s - 211.86s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 15: 212.54s - 220.84s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 16: 222.21s - 242.56s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 17: 242.76s - 256.89s --> SPEAKER_04 +2025-06-01 19:14:18,642 - INFO - [DIAR] 18: 258.15s - 281.19s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 19: 281.46s - 286.84s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 20: 287.06s - 290.42s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 21: 291.55s - 330.31s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 22: 330.49s - 349.14s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 23: 350.44s - 384.06s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 24: 384.31s - 388.48s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 25: 388.85s - 390.01s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 26: 392.64s - 432.89s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 27: 435.68s - 476.92s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 28: 478.64s - 519.78s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 29: 519.97s - 523.07s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 30: 523.22s - 527.76s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 31: 529.47s - 530.41s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 32: 533.52s - 536.59s --> SPEAKER_00 +2025-06-01 19:14:18,643 - INFO - [DIAR] 33: 535.22s - 535.31s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 34: 536.96s - 540.84s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 35: 542.21s - 542.24s --> SPEAKER_00 +2025-06-01 19:14:18,643 - INFO - [DIAR] 36: 542.24s - 542.36s --> SPEAKER_01 +2025-06-01 19:14:18,643 - INFO - [DIAR] 37: 542.36s - 542.65s --> SPEAKER_00 +2025-06-01 19:14:18,643 - INFO - [DIAR] 38: 543.14s - 547.54s --> SPEAKER_00 +2025-06-01 19:14:18,643 - INFO - [DIAR] 39: 547.91s - 548.62s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 40: 548.77s - 637.10s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 41: 637.45s - 641.43s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 42: 641.96s - 643.69s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 43: 644.18s - 661.09s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 44: 661.70s - 664.69s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 45: 664.94s - 667.64s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 46: 668.01s - 678.47s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 47: 679.05s - 691.86s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 48: 693.39s - 693.56s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 49: 695.04s - 708.41s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 50: 708.81s - 750.16s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 51: 751.68s - 756.17s --> SPEAKER_04 +2025-06-01 19:14:18,643 - INFO - [DIAR] 52: 756.59s - 768.38s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 53: 769.51s - 780.20s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 54: 781.04s - 783.47s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 55: 783.71s - 806.05s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 56: 806.08s - 858.46s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 57: 861.16s - 939.38s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 58: 941.30s - 941.49s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 59: 941.76s - 942.48s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 60: 942.84s - 971.00s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 61: 972.62s - 993.41s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 62: 994.22s - 995.86s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 63: 996.60s - 1013.78s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 64: 1014.45s - 1015.06s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 65: 1016.13s - 1018.67s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 66: 1019.52s - 1020.19s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 67: 1022.50s - 1058.13s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 68: 1058.55s - 1060.56s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 69: 1060.83s - 1087.03s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 70: 1087.46s - 1095.67s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 71: 1096.16s - 1116.26s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 72: 1117.61s - 1151.36s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 73: 1152.90s - 1170.41s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 74: 1172.05s - 1178.19s --> SPEAKER_04 +2025-06-01 19:14:18,644 - INFO - [DIAR] 75: 1182.14s - 1191.22s --> SPEAKER_00 +2025-06-01 19:14:18,644 - INFO - [DIAR] 76: 1191.63s - 1199.08s --> SPEAKER_00 +2025-06-01 19:14:18,644 - INFO - [DIAR] 77: 1199.61s - 1215.94s --> SPEAKER_00 +2025-06-01 19:14:18,644 - INFO - [DIAR] 78: 1199.73s - 1200.00s --> SPEAKER_01 +2025-06-01 19:14:18,644 - INFO - [DIAR] 79: 1216.45s - 1219.54s --> SPEAKER_00 +2025-06-01 19:14:18,644 - INFO - [DIAR] 80: 1220.13s - 1226.05s --> SPEAKER_00 +2025-06-01 19:14:18,644 - INFO - [DIAR] 81: 1226.19s - 1234.13s --> SPEAKER_00 +2025-06-01 19:14:18,644 - INFO - [DIAR] 82: 1234.61s - 1251.36s --> SPEAKER_00 +2025-06-01 19:14:18,644 - INFO - [DIAR] 83: 1238.35s - 1238.76s --> SPEAKER_01 +2025-06-01 19:14:18,644 - INFO - [DIAR] 84: 1241.15s - 1241.17s --> SPEAKER_01 +2025-06-01 19:14:18,644 - INFO - [DIAR] 85: 1249.68s - 1250.05s --> SPEAKER_01 +2025-06-01 19:14:18,644 - INFO - [DIAR] 86: 1251.63s - 1253.44s --> SPEAKER_00 +2025-06-01 19:14:18,644 - INFO - [DIAR] 87: 1252.49s - 1252.78s --> SPEAKER_01 +2025-06-01 19:14:18,645 - INFO - [DIAR] 88: 1253.69s - 1262.91s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 89: 1263.41s - 1269.23s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 90: 1269.79s - 1270.13s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 91: 1270.79s - 1272.36s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 92: 1272.84s - 1276.10s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 93: 1276.34s - 1312.79s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 94: 1313.36s - 1314.17s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 95: 1314.80s - 1351.25s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 96: 1353.22s - 1366.30s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 97: 1366.79s - 1367.97s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 98: 1368.19s - 1373.23s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 99: 1374.08s - 1394.02s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 100: 1394.07s - 1417.35s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 101: 1397.03s - 1397.38s --> SPEAKER_01 +2025-06-01 19:14:18,645 - INFO - [DIAR] 102: 1397.40s - 1397.42s --> SPEAKER_01 +2025-06-01 19:14:18,645 - INFO - [DIAR] 103: 1398.87s - 1399.02s --> SPEAKER_01 +2025-06-01 19:14:18,645 - INFO - [DIAR] 104: 1399.05s - 1399.07s --> SPEAKER_01 +2025-06-01 19:14:18,645 - INFO - [DIAR] 105: 1417.40s - 1426.04s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 106: 1426.49s - 1440.01s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 107: 1440.62s - 1442.44s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 108: 1441.14s - 1441.53s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 109: 1443.05s - 1446.45s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 110: 1447.18s - 1475.41s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 111: 1475.87s - 1477.13s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 112: 1477.47s - 1501.72s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 113: 1496.76s - 1496.79s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 114: 1504.81s - 1532.01s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 115: 1532.72s - 1570.08s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 116: 1570.57s - 1576.48s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 117: 1570.84s - 1571.73s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 118: 1572.41s - 1575.13s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 119: 1575.65s - 1605.27s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 120: 1605.97s - 1610.07s --> SPEAKER_04 +2025-06-01 19:14:18,645 - INFO - [DIAR] 121: 1611.56s - 1618.23s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 122: 1618.60s - 1626.58s --> SPEAKER_00 +2025-06-01 19:14:18,645 - INFO - [DIAR] 123: 1619.36s - 1620.30s --> SPEAKER_01 +2025-06-01 19:14:18,645 - INFO - [DIAR] 124: 1620.55s - 1620.71s --> SPEAKER_01 +2025-06-01 19:14:18,645 - INFO - [DIAR] 125: 1620.71s - 1621.23s --> SPEAKER_02 +2025-06-01 19:14:18,646 - INFO - [DIAR] 126: 1621.23s - 1621.26s --> SPEAKER_01 +2025-06-01 19:14:18,646 - INFO - [DIAR] 127: 1625.82s - 1626.49s --> SPEAKER_02 +2025-06-01 19:14:18,646 - INFO - [DIAR] 128: 1628.28s - 1628.97s --> SPEAKER_00 +2025-06-01 19:14:18,646 - INFO - [DIAR] 129: 1628.42s - 1640.11s --> SPEAKER_02 +2025-06-01 19:14:18,646 - INFO - [DIAR] 130: 1640.60s - 1650.22s --> SPEAKER_02 +2025-06-01 19:14:18,646 - INFO - [DIAR] 131: 1650.95s - 1662.15s --> SPEAKER_02 +2025-06-01 19:14:18,646 - INFO - [DIAR] 132: 1662.64s - 1666.49s --> SPEAKER_02 +2025-06-01 19:14:18,646 - INFO - [DIAR] 133: 1666.74s - 1677.49s --> SPEAKER_02 +2025-06-01 19:14:18,646 - INFO - [DIAR] 134: 1677.88s - 1684.58s --> SPEAKER_02 +2025-06-01 19:14:18,646 - INFO - [DIAR] 135: 1685.08s - 1697.23s --> SPEAKER_02 +2025-06-01 19:14:18,646 - INFO - [DIAR] 136: 1698.82s - 1705.47s --> SPEAKER_00 +2025-06-01 19:14:18,646 - INFO - [DIAR] 137: 1705.52s - 1707.29s --> SPEAKER_00 +2025-06-01 19:14:18,646 - INFO - [DIAR] 138: 1705.57s - 1705.91s --> SPEAKER_01 +2025-06-01 19:14:18,646 - INFO - [DIAR] 139: 1707.75s - 1723.29s --> SPEAKER_00 +2025-06-01 19:14:18,646 - INFO - [DIAR] 140: 1712.13s - 1712.42s --> SPEAKER_01 +2025-06-01 19:14:18,646 - INFO - [DIAR] 141: 1723.86s - 1724.79s --> SPEAKER_00 +2025-06-01 19:14:18,646 - INFO - [DIAR] 142: 1725.87s - 1745.53s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 143: 1743.32s - 1745.50s --> SPEAKER_00 +2025-06-01 19:14:18,646 - INFO - [DIAR] 144: 1746.07s - 1746.95s --> SPEAKER_00 +2025-06-01 19:14:18,646 - INFO - [DIAR] 145: 1746.15s - 1746.75s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 146: 1747.20s - 1752.06s --> SPEAKER_00 +2025-06-01 19:14:18,646 - INFO - [DIAR] 147: 1747.71s - 1747.83s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 148: 1752.25s - 1766.03s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 149: 1769.48s - 1772.23s --> SPEAKER_00 +2025-06-01 19:14:18,646 - INFO - [DIAR] 150: 1776.19s - 1791.30s --> SPEAKER_03 +2025-06-01 19:14:18,646 - INFO - [DIAR] 151: 1793.08s - 1799.31s --> SPEAKER_03 +2025-06-01 19:14:18,646 - INFO - [DIAR] 152: 1801.02s - 1875.75s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 153: 1875.97s - 1880.63s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 154: 1881.00s - 1882.07s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 155: 1884.02s - 1886.50s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 156: 1889.24s - 1932.02s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 157: 1932.72s - 1933.86s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 158: 1935.00s - 1943.78s --> SPEAKER_00 +2025-06-01 19:14:18,646 - INFO - [DIAR] 159: 1944.42s - 1944.44s --> SPEAKER_00 +2025-06-01 19:14:18,646 - INFO - [DIAR] 160: 1944.44s - 1960.08s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 161: 1944.47s - 1945.23s --> SPEAKER_00 +2025-06-01 19:14:18,646 - INFO - [DIAR] 162: 1960.59s - 1963.45s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 163: 1963.52s - 1992.65s --> SPEAKER_04 +2025-06-01 19:14:18,646 - INFO - [DIAR] 164: 1993.44s - 2004.53s --> SPEAKER_00 +2025-06-01 19:14:18,647 - INFO - [DIAR] 165: 2001.14s - 2001.93s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 166: 2003.31s - 2003.80s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 167: 2004.60s - 2024.04s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 168: 2026.62s - 2040.47s --> SPEAKER_00 +2025-06-01 19:14:18,647 - INFO - [DIAR] 169: 2040.64s - 2051.74s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 170: 2053.65s - 2054.66s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 171: 2054.26s - 2061.40s --> SPEAKER_00 +2025-06-01 19:14:18,647 - INFO - [DIAR] 172: 2062.07s - 2065.21s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 173: 2070.26s - 2071.62s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 174: 2071.84s - 2073.09s --> SPEAKER_01 +2025-06-01 19:14:18,647 - INFO - [DIAR] 175: 2073.33s - 2073.39s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 176: 2073.39s - 2073.53s --> SPEAKER_00 +2025-06-01 19:14:18,647 - INFO - [DIAR] 177: 2073.53s - 2073.63s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 178: 2074.69s - 2076.94s --> SPEAKER_01 +2025-06-01 19:14:18,647 - INFO - [DIAR] 179: 2078.34s - 2088.73s --> SPEAKER_00 +2025-06-01 19:14:18,647 - INFO - [DIAR] 180: 2089.53s - 2099.43s --> SPEAKER_01 +2025-06-01 19:14:18,647 - INFO - [DIAR] 181: 2101.34s - 2109.76s --> SPEAKER_01 +2025-06-01 19:14:18,647 - INFO - [DIAR] 182: 2110.27s - 2114.11s --> SPEAKER_01 +2025-06-01 19:14:18,647 - INFO - [DIAR] 183: 2115.92s - 2119.75s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 184: 2118.21s - 2119.65s --> SPEAKER_01 +2025-06-01 19:14:18,647 - INFO - [DIAR] 185: 2120.24s - 2124.68s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 186: 2125.93s - 2140.02s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 187: 2141.94s - 2142.40s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 188: 2142.04s - 2148.27s --> SPEAKER_00 +2025-06-01 19:14:18,647 - INFO - [DIAR] 189: 2147.91s - 2162.95s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 190: 2165.77s - 2175.88s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 191: 2175.91s - 2185.48s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 192: 2185.97s - 2186.04s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 193: 2186.34s - 2190.03s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 194: 2190.79s - 2192.28s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 195: 2199.60s - 2231.55s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 196: 2232.80s - 2233.34s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 197: 2233.34s - 2244.56s --> SPEAKER_00 +2025-06-01 19:14:18,647 - INFO - [DIAR] 198: 2244.59s - 2266.73s --> SPEAKER_00 +2025-06-01 19:14:18,647 - INFO - [DIAR] 199: 2266.90s - 2268.99s --> SPEAKER_00 +2025-06-01 19:14:18,647 - INFO - [DIAR] 200: 2269.62s - 2292.38s --> SPEAKER_04 +2025-06-01 19:14:18,647 - INFO - [DIAR] 201: 2294.64s - 2295.44s --> SPEAKER_00 +2025-06-01 19:14:18,647 - INFO - [DIAR] 202: 2297.29s - 2297.61s --> SPEAKER_04 +2025-06-01 19:14:18,648 - INFO - [DIAR] 203: 2297.46s - 2300.25s --> SPEAKER_00 +2025-06-01 19:14:18,648 - INFO - [DIAR] 204: 2300.25s - 2305.43s --> SPEAKER_04 +2025-06-01 19:14:18,648 - INFO - Global diarization complete. 205 segments. +2025-06-01 19:14:20,737 - INFO - Running smart chunking... +2025-06-01 19:14:21,104 - INFO - Processing audio with duration 38:30.763 +2025-06-01 19:15:09,645 - INFO - Exported chunk: output_test5/chunks/chunk_000.wav (3.02s → 8.88s) +2025-06-01 19:15:09,645 - INFO - Exported chunk: output_test5/chunks/chunk_001.wav (8.88s → 16.16s) +2025-06-01 19:15:09,645 - INFO - Exported chunk: output_test5/chunks/chunk_002.wav (16.16s → 23.40s) +2025-06-01 19:15:09,646 - INFO - Exported chunk: output_test5/chunks/chunk_003.wav (23.40s → 28.22s) +2025-06-01 19:15:09,646 - INFO - Exported chunk: output_test5/chunks/chunk_004.wav (28.22s → 36.52s) +2025-06-01 19:15:09,646 - INFO - Exported chunk: output_test5/chunks/chunk_005.wav (36.52s → 42.70s) +2025-06-01 19:15:09,647 - INFO - Exported chunk: output_test5/chunks/chunk_006.wav (42.70s → 46.42s) +2025-06-01 19:15:09,647 - INFO - Exported chunk: output_test5/chunks/chunk_007.wav (46.42s → 51.48s) +2025-06-01 19:15:09,647 - INFO - Exported chunk: output_test5/chunks/chunk_008.wav (51.48s → 59.50s) +2025-06-01 19:15:09,648 - INFO - Exported chunk: output_test5/chunks/chunk_009.wav (59.50s → 65.74s) +2025-06-01 19:15:09,648 - INFO - Exported chunk: output_test5/chunks/chunk_010.wav (65.74s → 73.76s) +2025-06-01 19:15:09,649 - INFO - Exported chunk: output_test5/chunks/chunk_011.wav (73.76s → 81.80s) +2025-06-01 19:15:09,649 - INFO - Exported chunk: output_test5/chunks/chunk_012.wav (81.80s → 87.76s) +2025-06-01 19:15:09,649 - INFO - Exported chunk: output_test5/chunks/chunk_013.wav (88.84s → 95.10s) +2025-06-01 19:15:09,649 - INFO - Exported chunk: output_test5/chunks/chunk_014.wav (95.10s → 100.34s) +2025-06-01 19:15:09,650 - INFO - Exported chunk: output_test5/chunks/chunk_015.wav (100.34s → 106.84s) +2025-06-01 19:15:09,650 - INFO - Exported chunk: output_test5/chunks/chunk_016.wav (106.84s → 116.34s) +2025-06-01 19:15:09,651 - INFO - Exported chunk: output_test5/chunks/chunk_017.wav (116.34s → 126.88s) +2025-06-01 19:15:09,651 - INFO - Exported chunk: output_test5/chunks/chunk_018.wav (126.88s → 134.18s) +2025-06-01 19:15:09,651 - INFO - Exported chunk: output_test5/chunks/chunk_019.wav (134.18s → 143.16s) +2025-06-01 19:15:09,652 - INFO - Exported chunk: output_test5/chunks/chunk_020.wav (143.16s → 149.00s) +2025-06-01 19:15:09,652 - INFO - Exported chunk: output_test5/chunks/chunk_021.wav (149.00s → 156.74s) +2025-06-01 19:15:09,652 - INFO - Exported chunk: output_test5/chunks/chunk_022.wav (156.74s → 162.76s) +2025-06-01 19:15:09,653 - INFO - Exported chunk: output_test5/chunks/chunk_023.wav (162.76s → 169.96s) +2025-06-01 19:15:09,653 - INFO - Exported chunk: output_test5/chunks/chunk_024.wav (170.20s → 176.60s) +2025-06-01 19:15:09,653 - INFO - Exported chunk: output_test5/chunks/chunk_025.wav (176.74s → 182.42s) +2025-06-01 19:15:09,654 - INFO - Exported chunk: output_test5/chunks/chunk_026.wav (182.42s → 189.70s) +2025-06-01 19:15:09,654 - INFO - Exported chunk: output_test5/chunks/chunk_027.wav (189.70s → 194.10s) +2025-06-01 19:15:09,654 - INFO - Exported chunk: output_test5/chunks/chunk_028.wav (194.10s → 200.32s) +2025-06-01 19:15:09,655 - INFO - Exported chunk: output_test5/chunks/chunk_029.wav (200.32s → 206.02s) +2025-06-01 19:15:09,655 - INFO - Exported chunk: output_test5/chunks/chunk_030.wav (206.02s → 211.72s) +2025-06-01 19:15:09,655 - INFO - Exported chunk: output_test5/chunks/chunk_031.wav (212.48s → 217.86s) +2025-06-01 19:15:09,656 - INFO - Exported chunk: output_test5/chunks/chunk_032.wav (217.86s → 226.02s) +2025-06-01 19:15:09,656 - INFO - Exported chunk: output_test5/chunks/chunk_033.wav (226.02s → 233.40s) +2025-06-01 19:15:09,656 - INFO - Exported chunk: output_test5/chunks/chunk_034.wav (233.40s → 240.76s) +2025-06-01 19:15:09,657 - INFO - Exported chunk: output_test5/chunks/chunk_035.wav (240.90s → 246.58s) +2025-06-01 19:15:09,657 - INFO - Exported chunk: output_test5/chunks/chunk_036.wav (246.58s → 253.64s) +2025-06-01 19:15:09,657 - INFO - Exported chunk: output_test5/chunks/chunk_037.wav (253.64s → 260.82s) +2025-06-01 19:15:09,658 - INFO - Exported chunk: output_test5/chunks/chunk_038.wav (260.82s → 265.04s) +2025-06-01 19:15:09,658 - INFO - Exported chunk: output_test5/chunks/chunk_039.wav (265.04s → 269.62s) +2025-06-01 19:15:09,658 - INFO - Exported chunk: output_test5/chunks/chunk_040.wav (270.12s → 275.30s) +2025-06-01 19:15:09,659 - INFO - Exported chunk: output_test5/chunks/chunk_041.wav (275.30s → 280.70s) +2025-06-01 19:15:09,659 - INFO - Exported chunk: output_test5/chunks/chunk_042.wav (280.70s → 285.20s) +2025-06-01 19:15:09,659 - INFO - Exported chunk: output_test5/chunks/chunk_043.wav (285.20s → 293.24s) +2025-06-01 19:15:09,659 - INFO - Exported chunk: output_test5/chunks/chunk_044.wav (293.38s → 298.50s) +2025-06-01 19:15:09,660 - INFO - Exported chunk: output_test5/chunks/chunk_045.wav (298.50s → 303.96s) +2025-06-01 19:15:09,660 - INFO - Exported chunk: output_test5/chunks/chunk_046.wav (303.96s → 310.26s) +2025-06-01 19:15:09,661 - INFO - Exported chunk: output_test5/chunks/chunk_047.wav (310.26s → 317.74s) +2025-06-01 19:15:09,661 - INFO - Exported chunk: output_test5/chunks/chunk_048.wav (317.74s → 323.38s) +2025-06-01 19:15:09,661 - INFO - Exported chunk: output_test5/chunks/chunk_049.wav (323.38s → 328.16s) +2025-06-01 19:15:09,661 - INFO - Exported chunk: output_test5/chunks/chunk_050.wav (328.16s → 335.24s) +2025-06-01 19:15:09,662 - INFO - Exported chunk: output_test5/chunks/chunk_051.wav (335.24s → 340.98s) +2025-06-01 19:15:09,662 - INFO - Exported chunk: output_test5/chunks/chunk_052.wav (340.98s → 347.06s) +2025-06-01 19:15:09,662 - INFO - Exported chunk: output_test5/chunks/chunk_053.wav (347.24s → 353.52s) +2025-06-01 19:15:09,663 - INFO - Exported chunk: output_test5/chunks/chunk_054.wav (353.52s → 361.10s) +2025-06-01 19:15:09,663 - INFO - Exported chunk: output_test5/chunks/chunk_055.wav (361.10s → 367.14s) +2025-06-01 19:15:09,663 - INFO - Exported chunk: output_test5/chunks/chunk_056.wav (367.72s → 373.24s) +2025-06-01 19:15:09,664 - INFO - Exported chunk: output_test5/chunks/chunk_057.wav (373.26s → 378.14s) +2025-06-01 19:15:09,664 - INFO - Exported chunk: output_test5/chunks/chunk_058.wav (378.74s → 383.86s) +2025-06-01 19:15:09,664 - INFO - Exported chunk: output_test5/chunks/chunk_059.wav (383.86s → 388.50s) +2025-06-01 19:15:09,665 - INFO - Exported chunk: output_test5/chunks/chunk_060.wav (388.50s → 395.48s) +2025-06-01 19:15:09,665 - INFO - Exported chunk: output_test5/chunks/chunk_061.wav (395.58s → 401.38s) +2025-06-01 19:15:09,665 - INFO - Exported chunk: output_test5/chunks/chunk_062.wav (401.38s → 406.18s) +2025-06-01 19:15:09,666 - INFO - Exported chunk: output_test5/chunks/chunk_063.wav (406.64s → 413.20s) +2025-06-01 19:15:09,666 - INFO - Exported chunk: output_test5/chunks/chunk_064.wav (413.20s → 419.22s) +2025-06-01 19:15:09,666 - INFO - Exported chunk: output_test5/chunks/chunk_065.wav (419.22s → 426.10s) +2025-06-01 19:15:09,667 - INFO - Exported chunk: output_test5/chunks/chunk_066.wav (426.72s → 432.38s) +2025-06-01 19:15:09,667 - INFO - Exported chunk: output_test5/chunks/chunk_067.wav (432.38s → 432.78s) +2025-06-01 19:15:09,667 - INFO - Exported chunk: output_test5/chunks/chunk_068.wav (435.46s → 440.86s) +2025-06-01 19:15:09,667 - INFO - Exported chunk: output_test5/chunks/chunk_069.wav (441.42s → 447.40s) +2025-06-01 19:15:09,668 - INFO - Exported chunk: output_test5/chunks/chunk_070.wav (447.40s → 452.46s) +2025-06-01 19:15:09,668 - INFO - Exported chunk: output_test5/chunks/chunk_071.wav (452.46s → 461.26s) +2025-06-01 19:15:09,668 - INFO - Exported chunk: output_test5/chunks/chunk_072.wav (461.26s → 466.50s) +2025-06-01 19:15:09,669 - INFO - Exported chunk: output_test5/chunks/chunk_073.wav (466.98s → 473.00s) +2025-06-01 19:15:09,669 - INFO - Exported chunk: output_test5/chunks/chunk_074.wav (473.62s → 476.72s) +2025-06-01 19:15:09,669 - INFO - Exported chunk: output_test5/chunks/chunk_075.wav (478.42s → 482.54s) +2025-06-01 19:15:09,670 - INFO - Exported chunk: output_test5/chunks/chunk_076.wav (482.54s → 487.72s) +2025-06-01 19:15:09,670 - INFO - Exported chunk: output_test5/chunks/chunk_077.wav (487.72s → 496.56s) +2025-06-01 19:15:09,670 - INFO - Exported chunk: output_test5/chunks/chunk_078.wav (496.56s → 503.56s) +2025-06-01 19:15:09,671 - INFO - Exported chunk: output_test5/chunks/chunk_079.wav (503.56s → 508.32s) +2025-06-01 19:15:09,671 - INFO - Exported chunk: output_test5/chunks/chunk_080.wav (508.50s → 515.78s) +2025-06-01 19:15:09,671 - INFO - Exported chunk: output_test5/chunks/chunk_081.wav (515.78s → 524.68s) +2025-06-01 19:15:09,672 - INFO - Exported chunk: output_test5/chunks/chunk_082.wav (524.68s → 530.32s) +2025-06-01 19:15:09,672 - INFO - Exported chunk: output_test5/chunks/chunk_083.wav (533.12s → 536.40s) +2025-06-01 19:15:09,672 - INFO - Exported chunk: output_test5/chunks/chunk_084.wav (536.76s → 540.74s) +2025-06-01 19:15:09,673 - INFO - Exported chunk: output_test5/chunks/chunk_085.wav (542.86s → 546.90s) +2025-06-01 19:15:09,673 - INFO - Exported chunk: output_test5/chunks/chunk_086.wav (546.90s → 553.08s) +2025-06-01 19:15:09,673 - INFO - Exported chunk: output_test5/chunks/chunk_087.wav (553.08s → 560.18s) +2025-06-01 19:15:09,674 - INFO - Exported chunk: output_test5/chunks/chunk_088.wav (560.18s → 565.18s) +2025-06-01 19:15:09,674 - INFO - Exported chunk: output_test5/chunks/chunk_089.wav (565.18s → 572.08s) +2025-06-01 19:15:09,674 - INFO - Exported chunk: output_test5/chunks/chunk_090.wav (572.50s → 576.84s) +2025-06-01 19:15:09,674 - INFO - Exported chunk: output_test5/chunks/chunk_091.wav (577.00s → 582.78s) +2025-06-01 19:15:09,675 - INFO - Exported chunk: output_test5/chunks/chunk_092.wav (583.48s → 588.00s) +2025-06-01 19:15:09,675 - INFO - Exported chunk: output_test5/chunks/chunk_093.wav (588.00s → 592.40s) +2025-06-01 19:15:09,675 - INFO - Exported chunk: output_test5/chunks/chunk_094.wav (592.94s → 599.38s) +2025-06-01 19:15:09,676 - INFO - Exported chunk: output_test5/chunks/chunk_095.wav (599.76s → 603.88s) +2025-06-01 19:15:09,676 - INFO - Exported chunk: output_test5/chunks/chunk_096.wav (604.22s → 609.62s) +2025-06-01 19:15:09,676 - INFO - Exported chunk: output_test5/chunks/chunk_097.wav (610.28s → 614.26s) +2025-06-01 19:15:09,677 - INFO - Exported chunk: output_test5/chunks/chunk_098.wav (614.26s → 620.72s) +2025-06-01 19:15:09,677 - INFO - Exported chunk: output_test5/chunks/chunk_099.wav (620.88s → 625.80s) +2025-06-01 19:15:09,677 - INFO - Exported chunk: output_test5/chunks/chunk_100.wav (625.80s → 633.90s) +2025-06-01 19:15:09,677 - INFO - Exported chunk: output_test5/chunks/chunk_101.wav (633.90s → 639.12s) +2025-06-01 19:15:09,678 - INFO - Exported chunk: output_test5/chunks/chunk_102.wav (639.12s → 644.38s) +2025-06-01 19:15:09,678 - INFO - Exported chunk: output_test5/chunks/chunk_103.wav (644.48s → 649.74s) +2025-06-01 19:15:09,678 - INFO - Exported chunk: output_test5/chunks/chunk_104.wav (649.74s → 656.04s) +2025-06-01 19:15:09,679 - INFO - Exported chunk: output_test5/chunks/chunk_105.wav (656.10s → 661.96s) +2025-06-01 19:15:09,679 - INFO - Exported chunk: output_test5/chunks/chunk_106.wav (662.18s → 667.50s) +2025-06-01 19:15:09,679 - INFO - Exported chunk: output_test5/chunks/chunk_107.wav (667.50s → 672.64s) +2025-06-01 19:15:09,680 - INFO - Exported chunk: output_test5/chunks/chunk_108.wav (673.42s → 677.98s) +2025-06-01 19:15:09,680 - INFO - Exported chunk: output_test5/chunks/chunk_109.wav (678.82s → 683.76s) +2025-06-01 19:15:09,680 - INFO - Exported chunk: output_test5/chunks/chunk_110.wav (683.76s → 687.26s) +2025-06-01 19:15:09,680 - INFO - Exported chunk: output_test5/chunks/chunk_111.wav (687.48s → 693.50s) +2025-06-01 19:15:09,681 - INFO - Exported chunk: output_test5/chunks/chunk_112.wav (694.82s → 699.48s) +2025-06-01 19:15:09,681 - INFO - Exported chunk: output_test5/chunks/chunk_113.wav (699.48s → 705.10s) +2025-06-01 19:15:09,681 - INFO - Exported chunk: output_test5/chunks/chunk_114.wav (705.10s → 711.22s) +2025-06-01 19:15:09,682 - INFO - Exported chunk: output_test5/chunks/chunk_115.wav (711.22s → 718.32s) +2025-06-01 19:15:09,682 - INFO - Exported chunk: output_test5/chunks/chunk_116.wav (718.32s → 725.40s) +2025-06-01 19:15:09,682 - INFO - Exported chunk: output_test5/chunks/chunk_117.wav (725.40s → 733.96s) +2025-06-01 19:15:09,683 - INFO - Exported chunk: output_test5/chunks/chunk_118.wav (733.96s → 741.02s) +2025-06-01 19:15:09,683 - INFO - Exported chunk: output_test5/chunks/chunk_119.wav (741.02s → 746.48s) +2025-06-01 19:15:09,684 - INFO - Exported chunk: output_test5/chunks/chunk_120.wav (746.48s → 754.10s) +2025-06-01 19:15:09,684 - INFO - Exported chunk: output_test5/chunks/chunk_121.wav (754.10s → 761.90s) +2025-06-01 19:15:09,684 - INFO - Exported chunk: output_test5/chunks/chunk_122.wav (761.90s → 767.14s) +2025-06-01 19:15:09,685 - INFO - Exported chunk: output_test5/chunks/chunk_123.wav (767.14s → 774.28s) +2025-06-01 19:15:09,685 - INFO - Exported chunk: output_test5/chunks/chunk_124.wav (774.52s → 779.80s) +2025-06-01 19:15:09,685 - INFO - Exported chunk: output_test5/chunks/chunk_125.wav (780.94s → 790.34s) +2025-06-01 19:15:09,686 - INFO - Exported chunk: output_test5/chunks/chunk_126.wav (790.34s → 796.30s) +2025-06-01 19:15:09,686 - INFO - Exported chunk: output_test5/chunks/chunk_127.wav (796.30s → 801.88s) +2025-06-01 19:15:09,686 - INFO - Exported chunk: output_test5/chunks/chunk_128.wav (801.88s → 808.44s) +2025-06-01 19:15:09,687 - INFO - Exported chunk: output_test5/chunks/chunk_129.wav (808.44s → 813.46s) +2025-06-01 19:15:09,687 - INFO - Exported chunk: output_test5/chunks/chunk_130.wav (813.56s → 818.10s) +2025-06-01 19:15:09,687 - INFO - Exported chunk: output_test5/chunks/chunk_131.wav (818.28s → 823.70s) +2025-06-01 19:15:09,687 - INFO - Exported chunk: output_test5/chunks/chunk_132.wav (823.70s → 829.82s) +2025-06-01 19:15:09,688 - INFO - Exported chunk: output_test5/chunks/chunk_133.wav (829.82s → 836.16s) +2025-06-01 19:15:09,688 - INFO - Exported chunk: output_test5/chunks/chunk_134.wav (836.16s → 841.30s) +2025-06-01 19:15:09,688 - INFO - Exported chunk: output_test5/chunks/chunk_135.wav (841.30s → 848.90s) +2025-06-01 19:15:09,689 - INFO - Exported chunk: output_test5/chunks/chunk_136.wav (848.90s → 854.54s) +2025-06-01 19:15:09,689 - INFO - Exported chunk: output_test5/chunks/chunk_137.wav (854.54s → 863.24s) +2025-06-01 19:15:09,690 - INFO - Exported chunk: output_test5/chunks/chunk_138.wav (863.24s → 869.78s) +2025-06-01 19:15:09,690 - INFO - Exported chunk: output_test5/chunks/chunk_139.wav (869.78s → 875.56s) +2025-06-01 19:15:09,690 - INFO - Exported chunk: output_test5/chunks/chunk_140.wav (875.56s → 882.02s) +2025-06-01 19:15:09,691 - INFO - Exported chunk: output_test5/chunks/chunk_141.wav (882.02s → 888.40s) +2025-06-01 19:15:09,691 - INFO - Exported chunk: output_test5/chunks/chunk_142.wav (888.48s → 893.76s) +2025-06-01 19:15:09,691 - INFO - Exported chunk: output_test5/chunks/chunk_143.wav (893.96s → 900.94s) +2025-06-01 19:15:09,691 - INFO - Exported chunk: output_test5/chunks/chunk_144.wav (900.94s → 906.66s) +2025-06-01 19:15:09,692 - INFO - Exported chunk: output_test5/chunks/chunk_145.wav (906.66s → 911.94s) +2025-06-01 19:15:09,692 - INFO - Exported chunk: output_test5/chunks/chunk_146.wav (911.94s → 918.26s) +2025-06-01 19:15:09,692 - INFO - Exported chunk: output_test5/chunks/chunk_147.wav (918.26s → 925.58s) +2025-06-01 19:15:09,693 - INFO - Exported chunk: output_test5/chunks/chunk_148.wav (926.34s → 930.50s) +2025-06-01 19:15:09,693 - INFO - Exported chunk: output_test5/chunks/chunk_149.wav (930.56s → 934.30s) +2025-06-01 19:15:09,693 - INFO - Exported chunk: output_test5/chunks/chunk_150.wav (934.30s → 943.72s) +2025-06-01 19:15:09,694 - INFO - Exported chunk: output_test5/chunks/chunk_151.wav (943.72s → 950.26s) +2025-06-01 19:15:09,694 - INFO - Exported chunk: output_test5/chunks/chunk_152.wav (950.26s → 956.20s) +2025-06-01 19:15:09,694 - INFO - Exported chunk: output_test5/chunks/chunk_153.wav (956.20s → 964.78s) +2025-06-01 19:15:09,695 - INFO - Exported chunk: output_test5/chunks/chunk_154.wav (965.00s → 971.00s) +2025-06-01 19:15:09,695 - INFO - Exported chunk: output_test5/chunks/chunk_155.wav (972.68s → 979.18s) +2025-06-01 19:15:09,695 - INFO - Exported chunk: output_test5/chunks/chunk_156.wav (979.18s → 985.08s) +2025-06-01 19:15:09,696 - INFO - Exported chunk: output_test5/chunks/chunk_157.wav (985.08s → 993.12s) +2025-06-01 19:15:09,696 - INFO - Exported chunk: output_test5/chunks/chunk_158.wav (994.28s → 1001.86s) +2025-06-01 19:15:09,696 - INFO - Exported chunk: output_test5/chunks/chunk_159.wav (1001.86s → 1007.48s) +2025-06-01 19:15:09,697 - INFO - Exported chunk: output_test5/chunks/chunk_160.wav (1007.48s → 1014.64s) +2025-06-01 19:15:09,697 - INFO - Exported chunk: output_test5/chunks/chunk_161.wav (1016.12s → 1024.24s) +2025-06-01 19:15:09,697 - INFO - Exported chunk: output_test5/chunks/chunk_162.wav (1024.82s → 1029.48s) +2025-06-01 19:15:09,698 - INFO - Exported chunk: output_test5/chunks/chunk_163.wav (1029.60s → 1036.02s) +2025-06-01 19:15:09,698 - INFO - Exported chunk: output_test5/chunks/chunk_164.wav (1036.02s → 1043.68s) +2025-06-01 19:15:09,698 - INFO - Exported chunk: output_test5/chunks/chunk_165.wav (1044.36s → 1051.36s) +2025-06-01 19:15:09,699 - INFO - Exported chunk: output_test5/chunks/chunk_166.wav (1051.36s → 1060.04s) +2025-06-01 19:15:09,699 - INFO - Exported chunk: output_test5/chunks/chunk_167.wav (1060.04s → 1065.22s) +2025-06-01 19:15:09,699 - INFO - Exported chunk: output_test5/chunks/chunk_168.wav (1065.28s → 1070.76s) +2025-06-01 19:15:09,700 - INFO - Exported chunk: output_test5/chunks/chunk_169.wav (1070.92s → 1077.12s) +2025-06-01 19:15:09,700 - INFO - Exported chunk: output_test5/chunks/chunk_170.wav (1077.12s → 1083.92s) +2025-06-01 19:15:09,700 - INFO - Exported chunk: output_test5/chunks/chunk_171.wav (1083.92s → 1089.68s) +2025-06-01 19:15:09,701 - INFO - Exported chunk: output_test5/chunks/chunk_172.wav (1089.68s → 1094.48s) +2025-06-01 19:15:09,701 - INFO - Exported chunk: output_test5/chunks/chunk_173.wav (1094.48s → 1101.02s) +2025-06-01 19:15:09,701 - INFO - Exported chunk: output_test5/chunks/chunk_174.wav (1101.02s → 1108.34s) +2025-06-01 19:15:09,702 - INFO - Exported chunk: output_test5/chunks/chunk_175.wav (1108.34s → 1115.54s) +2025-06-01 19:15:09,702 - INFO - Exported chunk: output_test5/chunks/chunk_176.wav (1115.54s → 1120.92s) +2025-06-01 19:15:09,702 - INFO - Exported chunk: output_test5/chunks/chunk_177.wav (1120.92s → 1125.54s) +2025-06-01 19:15:09,703 - INFO - Exported chunk: output_test5/chunks/chunk_178.wav (1125.54s → 1130.48s) +2025-06-01 19:15:09,703 - INFO - Exported chunk: output_test5/chunks/chunk_179.wav (1130.48s → 1134.90s) +2025-06-01 19:15:09,703 - INFO - Exported chunk: output_test5/chunks/chunk_180.wav (1135.50s → 1141.02s) +2025-06-01 19:15:09,703 - INFO - Exported chunk: output_test5/chunks/chunk_181.wav (1141.12s → 1146.64s) +2025-06-01 19:15:09,704 - INFO - Exported chunk: output_test5/chunks/chunk_182.wav (1147.04s → 1154.96s) +2025-06-01 19:15:09,704 - INFO - Exported chunk: output_test5/chunks/chunk_183.wav (1154.96s → 1160.96s) +2025-06-01 19:15:09,704 - INFO - Exported chunk: output_test5/chunks/chunk_184.wav (1160.96s → 1167.84s) +2025-06-01 19:15:09,705 - INFO - Exported chunk: output_test5/chunks/chunk_185.wav (1167.84s → 1174.64s) +2025-06-01 19:15:09,705 - INFO - Exported chunk: output_test5/chunks/chunk_186.wav (1174.64s → 1184.82s) +2025-06-01 19:15:09,706 - INFO - Exported chunk: output_test5/chunks/chunk_187.wav (1184.82s → 1192.58s) +2025-06-01 19:15:09,706 - INFO - Exported chunk: output_test5/chunks/chunk_188.wav (1192.70s → 1197.34s) +2025-06-01 19:15:09,706 - INFO - Exported chunk: output_test5/chunks/chunk_189.wav (1197.34s → 1204.22s) +2025-06-01 19:15:09,706 - INFO - Exported chunk: output_test5/chunks/chunk_190.wav (1204.22s → 1210.12s) +2025-06-01 19:15:09,707 - INFO - Exported chunk: output_test5/chunks/chunk_191.wav (1210.12s → 1217.68s) +2025-06-01 19:15:09,707 - INFO - Exported chunk: output_test5/chunks/chunk_192.wav (1217.68s → 1223.26s) +2025-06-01 19:15:09,707 - INFO - Exported chunk: output_test5/chunks/chunk_193.wav (1223.26s → 1229.76s) +2025-06-01 19:15:09,708 - INFO - Exported chunk: output_test5/chunks/chunk_194.wav (1229.76s → 1234.04s) +2025-06-01 19:15:09,708 - INFO - Exported chunk: output_test5/chunks/chunk_195.wav (1234.82s → 1241.46s) +2025-06-01 19:15:09,708 - INFO - Exported chunk: output_test5/chunks/chunk_196.wav (1241.52s → 1248.52s) +2025-06-01 19:15:09,709 - INFO - Exported chunk: output_test5/chunks/chunk_197.wav (1248.52s → 1256.66s) +2025-06-01 19:15:09,709 - INFO - Exported chunk: output_test5/chunks/chunk_198.wav (1256.66s → 1264.66s) +2025-06-01 19:15:09,709 - INFO - Exported chunk: output_test5/chunks/chunk_199.wav (1264.66s → 1268.88s) +2025-06-01 19:15:09,710 - INFO - Exported chunk: output_test5/chunks/chunk_200.wav (1268.88s → 1275.04s) +2025-06-01 19:15:09,710 - INFO - Exported chunk: output_test5/chunks/chunk_201.wav (1275.04s → 1282.04s) +2025-06-01 19:15:09,710 - INFO - Exported chunk: output_test5/chunks/chunk_202.wav (1282.04s → 1288.96s) +2025-06-01 19:15:09,711 - INFO - Exported chunk: output_test5/chunks/chunk_203.wav (1288.96s → 1295.68s) +2025-06-01 19:15:09,711 - INFO - Exported chunk: output_test5/chunks/chunk_204.wav (1295.68s → 1302.50s) +2025-06-01 19:15:09,711 - INFO - Exported chunk: output_test5/chunks/chunk_205.wav (1302.50s → 1308.72s) +2025-06-01 19:15:09,712 - INFO - Exported chunk: output_test5/chunks/chunk_206.wav (1309.04s → 1316.64s) +2025-06-01 19:15:09,712 - INFO - Exported chunk: output_test5/chunks/chunk_207.wav (1316.64s → 1322.16s) +2025-06-01 19:15:09,712 - INFO - Exported chunk: output_test5/chunks/chunk_208.wav (1322.16s → 1329.20s) +2025-06-01 19:15:09,713 - INFO - Exported chunk: output_test5/chunks/chunk_209.wav (1329.20s → 1334.36s) +2025-06-01 19:15:09,713 - INFO - Exported chunk: output_test5/chunks/chunk_210.wav (1334.36s → 1341.72s) +2025-06-01 19:15:09,713 - INFO - Exported chunk: output_test5/chunks/chunk_211.wav (1341.72s → 1348.24s) +2025-06-01 19:15:09,714 - INFO - Exported chunk: output_test5/chunks/chunk_212.wav (1348.40s → 1355.08s) +2025-06-01 19:15:09,714 - INFO - Exported chunk: output_test5/chunks/chunk_213.wav (1355.08s → 1359.86s) +2025-06-01 19:15:09,714 - INFO - Exported chunk: output_test5/chunks/chunk_214.wav (1360.06s → 1365.26s) +2025-06-01 19:15:09,715 - INFO - Exported chunk: output_test5/chunks/chunk_215.wav (1365.26s → 1370.70s) +2025-06-01 19:15:09,715 - INFO - Exported chunk: output_test5/chunks/chunk_216.wav (1370.70s → 1376.88s) +2025-06-01 19:15:09,715 - INFO - Exported chunk: output_test5/chunks/chunk_217.wav (1376.88s → 1382.90s) +2025-06-01 19:15:09,716 - INFO - Exported chunk: output_test5/chunks/chunk_218.wav (1382.90s → 1389.08s) +2025-06-01 19:15:09,716 - INFO - Exported chunk: output_test5/chunks/chunk_219.wav (1389.18s → 1393.72s) +2025-06-01 19:15:09,716 - INFO - Exported chunk: output_test5/chunks/chunk_220.wav (1393.86s → 1400.74s) +2025-06-01 19:15:09,717 - INFO - Exported chunk: output_test5/chunks/chunk_221.wav (1400.74s → 1407.86s) +2025-06-01 19:15:09,717 - INFO - Exported chunk: output_test5/chunks/chunk_222.wav (1407.86s → 1414.32s) +2025-06-01 19:15:09,717 - INFO - Exported chunk: output_test5/chunks/chunk_223.wav (1414.32s → 1419.40s) +2025-06-01 19:15:09,717 - INFO - Exported chunk: output_test5/chunks/chunk_224.wav (1419.40s → 1424.86s) +2025-06-01 19:15:09,718 - INFO - Exported chunk: output_test5/chunks/chunk_225.wav (1424.86s → 1431.76s) +2025-06-01 19:15:09,718 - INFO - Exported chunk: output_test5/chunks/chunk_226.wav (1431.92s → 1437.42s) +2025-06-01 19:15:09,718 - INFO - Exported chunk: output_test5/chunks/chunk_227.wav (1437.42s → 1443.70s) +2025-06-01 19:15:09,719 - INFO - Exported chunk: output_test5/chunks/chunk_228.wav (1443.70s → 1448.52s) +2025-06-01 19:15:09,719 - INFO - Exported chunk: output_test5/chunks/chunk_229.wav (1448.52s → 1452.98s) +2025-06-01 19:15:09,719 - INFO - Exported chunk: output_test5/chunks/chunk_230.wav (1452.98s → 1458.16s) +2025-06-01 19:15:09,720 - INFO - Exported chunk: output_test5/chunks/chunk_231.wav (1458.16s → 1466.16s) +2025-06-01 19:15:09,720 - INFO - Exported chunk: output_test5/chunks/chunk_232.wav (1466.16s → 1472.56s) +2025-06-01 19:15:09,720 - INFO - Exported chunk: output_test5/chunks/chunk_233.wav (1472.56s → 1476.96s) +2025-06-01 19:15:09,720 - INFO - Exported chunk: output_test5/chunks/chunk_234.wav (1477.26s → 1481.82s) +2025-06-01 19:15:09,721 - INFO - Exported chunk: output_test5/chunks/chunk_235.wav (1481.82s → 1486.38s) +2025-06-01 19:15:09,721 - INFO - Exported chunk: output_test5/chunks/chunk_236.wav (1486.38s → 1490.48s) +2025-06-01 19:15:09,721 - INFO - Exported chunk: output_test5/chunks/chunk_237.wav (1490.48s → 1494.42s) +2025-06-01 19:15:09,721 - INFO - Exported chunk: output_test5/chunks/chunk_238.wav (1494.54s → 1500.00s) +2025-06-01 19:15:09,722 - INFO - Exported chunk: output_test5/chunks/chunk_239.wav (1500.24s → 1501.66s) +2025-06-01 19:15:09,722 - INFO - Exported chunk: output_test5/chunks/chunk_240.wav (1504.82s → 1508.62s) +2025-06-01 19:15:09,722 - INFO - Exported chunk: output_test5/chunks/chunk_241.wav (1508.88s → 1514.10s) +2025-06-01 19:15:09,723 - INFO - Exported chunk: output_test5/chunks/chunk_242.wav (1514.10s → 1520.06s) +2025-06-01 19:15:09,723 - INFO - Exported chunk: output_test5/chunks/chunk_243.wav (1520.06s → 1525.62s) +2025-06-01 19:15:09,723 - INFO - Exported chunk: output_test5/chunks/chunk_244.wav (1525.62s → 1531.80s) +2025-06-01 19:15:09,723 - INFO - Exported chunk: output_test5/chunks/chunk_245.wav (1532.68s → 1538.76s) +2025-06-01 19:15:09,724 - INFO - Exported chunk: output_test5/chunks/chunk_246.wav (1538.98s → 1546.40s) +2025-06-01 19:15:09,724 - INFO - Exported chunk: output_test5/chunks/chunk_247.wav (1546.40s → 1552.08s) +2025-06-01 19:15:09,724 - INFO - Exported chunk: output_test5/chunks/chunk_248.wav (1552.26s → 1557.66s) +2025-06-01 19:15:09,725 - INFO - Exported chunk: output_test5/chunks/chunk_249.wav (1557.74s → 1563.18s) +2025-06-01 19:15:09,725 - INFO - Exported chunk: output_test5/chunks/chunk_250.wav (1563.32s → 1568.10s) +2025-06-01 19:15:09,725 - INFO - Exported chunk: output_test5/chunks/chunk_251.wav (1568.46s → 1571.98s) +2025-06-01 19:15:09,726 - INFO - Exported chunk: output_test5/chunks/chunk_252.wav (1575.44s → 1582.84s) +2025-06-01 19:15:09,726 - INFO - Exported chunk: output_test5/chunks/chunk_253.wav (1582.84s → 1587.62s) +2025-06-01 19:15:09,726 - INFO - Exported chunk: output_test5/chunks/chunk_254.wav (1587.62s → 1593.08s) +2025-06-01 19:15:09,726 - INFO - Exported chunk: output_test5/chunks/chunk_255.wav (1593.42s → 1598.98s) +2025-06-01 19:15:09,727 - INFO - Exported chunk: output_test5/chunks/chunk_256.wav (1598.98s → 1605.30s) +2025-06-01 19:15:09,727 - INFO - Exported chunk: output_test5/chunks/chunk_257.wav (1605.80s → 1610.06s) +2025-06-01 19:15:09,727 - INFO - Exported chunk: output_test5/chunks/chunk_258.wav (1611.58s → 1616.90s) +2025-06-01 19:15:09,728 - INFO - Exported chunk: output_test5/chunks/chunk_259.wav (1616.90s → 1622.80s) +2025-06-01 19:15:09,728 - INFO - Exported chunk: output_test5/chunks/chunk_260.wav (1622.80s → 1631.16s) +2025-06-01 19:15:09,728 - INFO - Exported chunk: output_test5/chunks/chunk_261.wav (1631.16s → 1636.16s) +2025-06-01 19:15:09,729 - INFO - Exported chunk: output_test5/chunks/chunk_262.wav (1636.16s → 1642.72s) +2025-06-01 19:15:09,729 - INFO - Exported chunk: output_test5/chunks/chunk_263.wav (1642.72s → 1648.36s) +2025-06-01 19:15:09,729 - INFO - Exported chunk: output_test5/chunks/chunk_264.wav (1648.36s → 1654.28s) +2025-06-01 19:15:09,730 - INFO - Exported chunk: output_test5/chunks/chunk_265.wav (1654.28s → 1659.34s) +2025-06-01 19:15:09,730 - INFO - Exported chunk: output_test5/chunks/chunk_266.wav (1659.34s → 1665.92s) +2025-06-01 19:15:09,730 - INFO - Exported chunk: output_test5/chunks/chunk_267.wav (1665.92s → 1670.66s) +2025-06-01 19:15:09,731 - INFO - Exported chunk: output_test5/chunks/chunk_268.wav (1670.76s → 1675.24s) +2025-06-01 19:15:09,731 - INFO - Exported chunk: output_test5/chunks/chunk_269.wav (1675.24s → 1680.80s) +2025-06-01 19:15:09,731 - INFO - Exported chunk: output_test5/chunks/chunk_270.wav (1680.80s → 1687.10s) +2025-06-01 19:15:09,731 - INFO - Exported chunk: output_test5/chunks/chunk_271.wav (1687.10s → 1692.42s) +2025-06-01 19:15:09,732 - INFO - Exported chunk: output_test5/chunks/chunk_272.wav (1692.42s → 1696.92s) +2025-06-01 19:15:09,732 - INFO - Exported chunk: output_test5/chunks/chunk_273.wav (1696.92s → 1703.18s) +2025-06-01 19:15:09,732 - INFO - Exported chunk: output_test5/chunks/chunk_274.wav (1703.18s → 1709.12s) +2025-06-01 19:15:09,733 - INFO - Exported chunk: output_test5/chunks/chunk_275.wav (1709.18s → 1714.02s) +2025-06-01 19:15:09,733 - INFO - Exported chunk: output_test5/chunks/chunk_276.wav (1714.02s → 1719.68s) +2025-06-01 19:15:09,733 - INFO - Exported chunk: output_test5/chunks/chunk_277.wav (1719.80s → 1724.74s) +2025-06-01 19:15:09,733 - INFO - Exported chunk: output_test5/chunks/chunk_278.wav (1725.66s → 1730.58s) +2025-06-01 19:15:09,734 - INFO - Exported chunk: output_test5/chunks/chunk_279.wav (1730.58s → 1733.82s) +2025-06-01 19:15:09,734 - INFO - Exported chunk: output_test5/chunks/chunk_280.wav (1733.82s → 1739.30s) +2025-06-01 19:15:09,734 - INFO - Exported chunk: output_test5/chunks/chunk_281.wav (1739.64s → 1743.28s) +2025-06-01 19:15:09,734 - INFO - Exported chunk: output_test5/chunks/chunk_282.wav (1743.78s → 1748.44s) +2025-06-01 19:15:09,735 - INFO - Exported chunk: output_test5/chunks/chunk_283.wav (1748.44s → 1751.94s) +2025-06-01 19:15:09,735 - INFO - Exported chunk: output_test5/chunks/chunk_284.wav (1752.42s → 1756.04s) +2025-06-01 19:15:09,735 - INFO - Exported chunk: output_test5/chunks/chunk_285.wav (1756.04s → 1760.48s) +2025-06-01 19:15:09,736 - INFO - Exported chunk: output_test5/chunks/chunk_286.wav (1760.72s → 1765.54s) +2025-06-01 19:15:09,736 - INFO - Exported chunk: output_test5/chunks/chunk_287.wav (1769.16s → 1772.08s) +2025-06-01 19:15:09,736 - INFO - Exported chunk: output_test5/chunks/chunk_288.wav (1776.00s → 1782.78s) +2025-06-01 19:15:09,736 - INFO - Exported chunk: output_test5/chunks/chunk_289.wav (1782.78s → 1787.28s) +2025-06-01 19:15:09,737 - INFO - Exported chunk: output_test5/chunks/chunk_290.wav (1787.40s → 1790.98s) +2025-06-01 19:15:09,737 - INFO - Exported chunk: output_test5/chunks/chunk_291.wav (1792.82s → 1799.00s) +2025-06-01 19:15:09,737 - INFO - Exported chunk: output_test5/chunks/chunk_292.wav (1799.00s → 1806.74s) +2025-06-01 19:15:09,738 - INFO - Exported chunk: output_test5/chunks/chunk_293.wav (1806.74s → 1812.30s) +2025-06-01 19:15:09,738 - INFO - Exported chunk: output_test5/chunks/chunk_294.wav (1812.30s → 1818.10s) +2025-06-01 19:15:09,738 - INFO - Exported chunk: output_test5/chunks/chunk_295.wav (1818.10s → 1823.32s) +2025-06-01 19:15:09,738 - INFO - Exported chunk: output_test5/chunks/chunk_296.wav (1823.52s → 1830.08s) +2025-06-01 19:15:09,739 - INFO - Exported chunk: output_test5/chunks/chunk_297.wav (1830.08s → 1834.98s) +2025-06-01 19:15:09,739 - INFO - Exported chunk: output_test5/chunks/chunk_298.wav (1834.98s → 1839.86s) +2025-06-01 19:15:09,739 - INFO - Exported chunk: output_test5/chunks/chunk_299.wav (1839.86s → 1846.46s) +2025-06-01 19:15:09,740 - INFO - Exported chunk: output_test5/chunks/chunk_300.wav (1846.74s → 1852.78s) +2025-06-01 19:15:09,740 - INFO - Exported chunk: output_test5/chunks/chunk_301.wav (1852.88s → 1858.32s) +2025-06-01 19:15:09,740 - INFO - Exported chunk: output_test5/chunks/chunk_302.wav (1859.40s → 1865.00s) +2025-06-01 19:15:09,741 - INFO - Exported chunk: output_test5/chunks/chunk_303.wav (1865.00s → 1871.52s) +2025-06-01 19:15:09,741 - INFO - Exported chunk: output_test5/chunks/chunk_304.wav (1871.68s → 1876.50s) +2025-06-01 19:15:09,741 - INFO - Exported chunk: output_test5/chunks/chunk_305.wav (1876.50s → 1881.86s) +2025-06-01 19:15:09,741 - INFO - Exported chunk: output_test5/chunks/chunk_306.wav (1883.76s → 1886.40s) +2025-06-01 19:15:09,742 - INFO - Exported chunk: output_test5/chunks/chunk_307.wav (1889.04s → 1893.98s) +2025-06-01 19:15:09,742 - INFO - Exported chunk: output_test5/chunks/chunk_308.wav (1893.98s → 1899.88s) +2025-06-01 19:15:09,742 - INFO - Exported chunk: output_test5/chunks/chunk_309.wav (1899.88s → 1908.22s) +2025-06-01 19:15:09,743 - INFO - Exported chunk: output_test5/chunks/chunk_310.wav (1908.22s → 1913.92s) +2025-06-01 19:15:09,743 - INFO - Exported chunk: output_test5/chunks/chunk_311.wav (1913.92s → 1921.34s) +2025-06-01 19:15:09,743 - INFO - Exported chunk: output_test5/chunks/chunk_312.wav (1921.34s → 1927.54s) +2025-06-01 19:15:09,744 - INFO - Exported chunk: output_test5/chunks/chunk_313.wav (1927.70s → 1933.52s) +2025-06-01 19:15:09,744 - INFO - Exported chunk: output_test5/chunks/chunk_314.wav (1934.66s → 1939.46s) +2025-06-01 19:15:09,744 - INFO - Exported chunk: output_test5/chunks/chunk_315.wav (1939.46s → 1943.56s) +2025-06-01 19:15:09,744 - INFO - Exported chunk: output_test5/chunks/chunk_316.wav (1944.22s → 1950.76s) +2025-06-01 19:15:09,745 - INFO - Exported chunk: output_test5/chunks/chunk_317.wav (1950.76s → 1956.06s) +2025-06-01 19:15:09,745 - INFO - Exported chunk: output_test5/chunks/chunk_318.wav (1956.06s → 1961.90s) +2025-06-01 19:15:09,745 - INFO - Exported chunk: output_test5/chunks/chunk_319.wav (1961.90s → 1967.30s) +2025-06-01 19:15:09,746 - INFO - Exported chunk: output_test5/chunks/chunk_320.wav (1967.30s → 1973.44s) +2025-06-01 19:15:09,746 - INFO - Exported chunk: output_test5/chunks/chunk_321.wav (1973.56s → 1978.72s) +2025-06-01 19:15:09,746 - INFO - Exported chunk: output_test5/chunks/chunk_322.wav (1979.60s → 1985.58s) +2025-06-01 19:15:09,747 - INFO - Exported chunk: output_test5/chunks/chunk_323.wav (1986.32s → 1991.94s) +2025-06-01 19:15:09,747 - INFO - Exported chunk: output_test5/chunks/chunk_324.wav (1991.94s → 1992.78s) +2025-06-01 19:15:09,747 - INFO - Exported chunk: output_test5/chunks/chunk_325.wav (1992.78s → 1996.84s) +2025-06-01 19:15:09,747 - INFO - Exported chunk: output_test5/chunks/chunk_326.wav (1996.84s → 2000.92s) +2025-06-01 19:15:09,747 - INFO - Exported chunk: output_test5/chunks/chunk_327.wav (2001.16s → 2001.26s) +2025-06-01 19:15:09,748 - INFO - Exported chunk: output_test5/chunks/chunk_328.wav (2001.42s → 2004.32s) +2025-06-01 19:15:09,748 - INFO - Exported chunk: output_test5/chunks/chunk_329.wav (2004.50s → 2008.74s) +2025-06-01 19:15:09,748 - INFO - Exported chunk: output_test5/chunks/chunk_330.wav (2009.14s → 2014.52s) +2025-06-01 19:15:09,748 - INFO - Exported chunk: output_test5/chunks/chunk_331.wav (2014.52s → 2018.20s) +2025-06-01 19:15:09,749 - INFO - Exported chunk: output_test5/chunks/chunk_332.wav (2018.20s → 2023.98s) +2025-06-01 19:15:09,749 - INFO - Exported chunk: output_test5/chunks/chunk_333.wav (2026.36s → 2031.50s) +2025-06-01 19:15:09,749 - INFO - Exported chunk: output_test5/chunks/chunk_334.wav (2031.50s → 2037.42s) +2025-06-01 19:15:09,750 - INFO - Exported chunk: output_test5/chunks/chunk_335.wav (2037.98s → 2040.16s) +2025-06-01 19:15:09,750 - INFO - Exported chunk: output_test5/chunks/chunk_336.wav (2040.56s → 2045.82s) +2025-06-01 19:15:09,750 - INFO - Exported chunk: output_test5/chunks/chunk_337.wav (2045.82s → 2051.66s) +2025-06-01 19:15:09,750 - INFO - Exported chunk: output_test5/chunks/chunk_338.wav (2053.38s → 2058.88s) +2025-06-01 19:15:09,751 - INFO - Exported chunk: output_test5/chunks/chunk_339.wav (2058.88s → 2061.30s) +2025-06-01 19:15:09,751 - INFO - Exported chunk: output_test5/chunks/chunk_340.wav (2062.00s → 2065.12s) +2025-06-01 19:15:09,751 - INFO - Exported chunk: output_test5/chunks/chunk_341.wav (2069.82s → 2071.66s) +2025-06-01 19:15:09,751 - INFO - Exported chunk: output_test5/chunks/chunk_342.wav (2071.70s → 2072.98s) +2025-06-01 19:15:09,751 - INFO - Exported chunk: output_test5/chunks/chunk_343.wav (2073.26s → 2073.48s) +2025-06-01 19:15:09,752 - INFO - Exported chunk: output_test5/chunks/chunk_344.wav (2074.50s → 2076.76s) +2025-06-01 19:15:09,752 - INFO - Exported chunk: output_test5/chunks/chunk_345.wav (2078.06s → 2083.04s) +2025-06-01 19:15:09,752 - INFO - Exported chunk: output_test5/chunks/chunk_346.wav (2083.20s → 2087.38s) +2025-06-01 19:15:09,752 - INFO - Exported chunk: output_test5/chunks/chunk_347.wav (2087.38s → 2088.40s) +2025-06-01 19:15:09,753 - INFO - Exported chunk: output_test5/chunks/chunk_348.wav (2088.40s → 2097.66s) +2025-06-01 19:15:09,753 - INFO - Exported chunk: output_test5/chunks/chunk_349.wav (2097.66s → 2106.98s) +2025-06-01 19:15:09,753 - INFO - Exported chunk: output_test5/chunks/chunk_350.wav (2106.98s → 2113.92s) +2025-06-01 19:15:09,754 - INFO - Exported chunk: output_test5/chunks/chunk_351.wav (2115.48s → 2119.36s) +2025-06-01 19:15:09,754 - INFO - Exported chunk: output_test5/chunks/chunk_352.wav (2119.90s → 2124.54s) +2025-06-01 19:15:09,754 - INFO - Exported chunk: output_test5/chunks/chunk_353.wav (2125.98s → 2131.52s) +2025-06-01 19:15:09,755 - INFO - Exported chunk: output_test5/chunks/chunk_354.wav (2132.44s → 2139.42s) +2025-06-01 19:15:09,755 - INFO - Exported chunk: output_test5/chunks/chunk_355.wav (2139.42s → 2145.34s) +2025-06-01 19:15:09,755 - INFO - Exported chunk: output_test5/chunks/chunk_356.wav (2145.60s → 2150.54s) +2025-06-01 19:15:09,756 - INFO - Exported chunk: output_test5/chunks/chunk_357.wav (2150.54s → 2156.58s) +2025-06-01 19:15:09,756 - INFO - Exported chunk: output_test5/chunks/chunk_358.wav (2156.80s → 2161.94s) +2025-06-01 19:15:09,756 - INFO - Exported chunk: output_test5/chunks/chunk_359.wav (2161.94s → 2169.64s) +2025-06-01 19:15:09,757 - INFO - Exported chunk: output_test5/chunks/chunk_360.wav (2169.70s → 2173.98s) +2025-06-01 19:15:09,757 - INFO - Exported chunk: output_test5/chunks/chunk_361.wav (2173.98s → 2179.56s) +2025-06-01 19:15:09,757 - INFO - Exported chunk: output_test5/chunks/chunk_362.wav (2179.56s → 2185.04s) +2025-06-01 19:15:09,758 - INFO - Exported chunk: output_test5/chunks/chunk_363.wav (2186.36s → 2192.22s) +2025-06-01 19:15:09,758 - INFO - Exported chunk: output_test5/chunks/chunk_364.wav (2199.28s → 2205.08s) +2025-06-01 19:15:09,758 - INFO - Exported chunk: output_test5/chunks/chunk_365.wav (2205.08s → 2210.50s) +2025-06-01 19:15:09,758 - INFO - Exported chunk: output_test5/chunks/chunk_366.wav (2210.50s → 2217.24s) +2025-06-01 19:15:09,759 - INFO - Exported chunk: output_test5/chunks/chunk_367.wav (2217.24s → 2223.18s) +2025-06-01 19:15:09,759 - INFO - Exported chunk: output_test5/chunks/chunk_368.wav (2223.18s → 2230.00s) +2025-06-01 19:15:09,759 - INFO - Exported chunk: output_test5/chunks/chunk_369.wav (2230.00s → 2235.76s) +2025-06-01 19:15:09,760 - INFO - Exported chunk: output_test5/chunks/chunk_370.wav (2235.88s → 2240.40s) +2025-06-01 19:15:09,760 - INFO - Exported chunk: output_test5/chunks/chunk_371.wav (2240.40s → 2245.44s) +2025-06-01 19:15:09,760 - INFO - Exported chunk: output_test5/chunks/chunk_372.wav (2245.44s → 2250.80s) +2025-06-01 19:15:09,761 - INFO - Exported chunk: output_test5/chunks/chunk_373.wav (2250.80s → 2256.60s) +2025-06-01 19:15:09,761 - INFO - Exported chunk: output_test5/chunks/chunk_374.wav (2256.60s → 2260.68s) +2025-06-01 19:15:09,761 - INFO - Exported chunk: output_test5/chunks/chunk_375.wav (2260.68s → 2266.22s) +2025-06-01 19:15:09,761 - INFO - Exported chunk: output_test5/chunks/chunk_376.wav (2266.70s → 2274.14s) +2025-06-01 19:15:09,762 - INFO - Exported chunk: output_test5/chunks/chunk_377.wav (2274.14s → 2278.88s) +2025-06-01 19:15:09,762 - INFO - Exported chunk: output_test5/chunks/chunk_378.wav (2279.10s → 2283.68s) +2025-06-01 19:15:09,762 - INFO - Exported chunk: output_test5/chunks/chunk_379.wav (2283.92s → 2290.10s) +2025-06-01 19:15:09,763 - INFO - Exported chunk: output_test5/chunks/chunk_380.wav (2290.10s → 2299.56s) +2025-06-01 19:15:09,763 - INFO - Exported chunk: output_test5/chunks/chunk_381.wav (2299.70s → 2305.14s) +2025-06-01 19:15:09,763 - INFO - Exported chunk: output_test5/chunks/chunk_382.wav (2306.84s → 2307.18s) +2025-06-01 19:15:09,763 - INFO - Exported chunk: output_test5/chunks/chunk_383.wav (2307.20s → 2307.36s) +2025-06-01 19:15:09,763 - INFO - Exported chunk: output_test5/chunks/chunk_384.wav (2308.00s → 2308.34s) +2025-06-01 19:15:09,764 - INFO - Exported chunk: output_test5/chunks/chunk_385.wav (2310.50s → 2310.74s) +2025-06-01 19:15:09,765 - INFO - Total chunks: 386 +2025-06-01 19:15:11,160 - INFO - WhisperModel 'turbo' loaded on GPU 1 for thread 126000312616512 +2025-06-01 19:15:11,161 - INFO - WhisperModel 'turbo' loaded on GPU 0 for thread 126000597829184 +2025-06-01 19:15:11,161 - INFO - Processing audio with duration 7.280 seconds +2025-06-01 19:15:11,161 - INFO - Processing audio with duration 5.860 seconds +2025-06-01 19:15:11,349 - INFO - Processing audio with duration 00:05.860 +2025-06-01 19:15:11,472 - INFO - Processing audio with duration 00:07.280 +2025-06-01 19:15:11,608 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:11,689 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:11,790 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:11,790 - INFO - Got result for chunk: output_test5/chunks/chunk_000.wav +2025-06-01 19:15:11,790 - INFO - +====== Streaming Chunk 001 ====== +[2025-06-01 19:13:25] Speaker 5: All right. Thank you all. Morning, afternoon. So my name is Amar Gowda. I am one of the +[2025-06-01 19:13:30] Speaker 5: products. + +2025-06-01 19:15:11,790 - INFO - Processing audio with duration 7.240 seconds +2025-06-01 19:15:11,978 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:11,979 - INFO - Processing audio with duration 00:07.240 +2025-06-01 19:15:11,979 - INFO - Got result for chunk: output_test5/chunks/chunk_001.wav +2025-06-01 19:15:11,979 - INFO - Processing audio with duration 4.820 seconds +2025-06-01 19:15:11,980 - INFO - +====== Streaming Chunk 002 ====== +[2025-06-01 19:13:25] Speaker 5: managers leading the OCI Lens initiative. So right now it's kind of what we call incubation. + +2025-06-01 19:15:12,188 - INFO - Processing audio with duration 00:04.820 +2025-06-01 19:15:12,498 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:12,623 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:12,663 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:12,663 - INFO - Got result for chunk: output_test5/chunks/chunk_002.wav +2025-06-01 19:15:12,663 - INFO - +====== Streaming Chunk 003 ====== +[2025-06-01 19:13:25] Speaker 5: phases so we are not yet to be a mvp or closer to it so we're kind of getting early feedback + +2025-06-01 19:15:12,664 - INFO - Processing audio with duration 8.300 seconds +2025-06-01 19:15:12,853 - INFO - Processing audio with duration 00:08.300 +2025-06-01 19:15:12,895 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:12,895 - INFO - Got result for chunk: output_test5/chunks/chunk_003.wav +2025-06-01 19:15:12,896 - INFO - +====== Streaming Chunk 004 ====== +[2025-06-01 19:13:25] Speaker 5: and this is like think of this as a kind of private preview in a way right so a little earlier on + +2025-06-01 19:15:12,896 - INFO - Processing audio with duration 6.180 seconds +2025-06-01 19:15:13,080 - INFO - Processing audio with duration 00:06.180 +2025-06-01 19:15:13,083 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:13,209 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:13,255 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:13,255 - INFO - Got result for chunk: output_test5/chunks/chunk_004.wav +2025-06-01 19:15:13,255 - INFO - +====== Streaming Chunk 005 ====== +[2025-06-01 19:13:25] Speaker 5: on the journey. +[2025-06-01 19:13:26] Speaker 5: But this is something we started looking into, +[2025-06-01 19:13:31] Speaker 5: which we, I think we spoke with. + +2025-06-01 19:15:13,255 - INFO - Processing audio with duration 3.720 seconds +2025-06-01 19:15:13,442 - INFO - Processing audio with duration 00:03.720 +2025-06-01 19:15:13,481 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:13,481 - INFO - Got result for chunk: output_test5/chunks/chunk_005.wav +2025-06-01 19:15:13,481 - INFO - +====== Streaming Chunk 006 ====== +[2025-06-01 19:13:25] Speaker 5: you, Cecil, you and the rest of the team on what does it take to operate at scale? + +2025-06-01 19:15:13,482 - INFO - Processing audio with duration 5.060 seconds +2025-06-01 19:15:13,670 - INFO - Processing audio with duration 00:05.060 +2025-06-01 19:15:13,672 - INFO - Detected language 'en' with probability 0.99 +2025-06-01 19:15:13,799 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:13,831 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:13,831 - INFO - Got result for chunk: output_test5/chunks/chunk_006.wav +2025-06-01 19:15:13,831 - INFO - +====== Streaming Chunk 007 ====== +[2025-06-01 19:13:25] Speaker 5: what are the things that we could improve on, especially the monitoring, the health check. + +2025-06-01 19:15:13,831 - INFO - Processing audio with duration 8.020 seconds +2025-06-01 19:15:14,009 - INFO - Processing audio with duration 00:08.020 +2025-06-01 19:15:14,047 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:14,048 - INFO - Got result for chunk: output_test5/chunks/chunk_007.wav +2025-06-01 19:15:14,048 - INFO - +====== Streaming Chunk 008 ====== +[2025-06-01 19:13:25] Speaker 5: side of the things, which has, you know, you gave a set of requirements and that led us. + +2025-06-01 19:15:14,048 - INFO - Processing audio with duration 6.240 seconds +2025-06-01 19:15:14,217 - INFO - Processing audio with duration 00:06.240 +2025-06-01 19:15:14,219 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:14,347 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:14,393 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:14,393 - INFO - Got result for chunk: output_test5/chunks/chunk_008.wav +2025-06-01 19:15:14,394 - INFO - +====== Streaming Chunk 009 ====== +[2025-06-01 19:13:25] Speaker 5: to building something to help customers like you. +[2025-06-01 19:13:30] Speaker 5: Okay, so I have a team here. +[2025-06-01 19:13:32] Speaker 5: Joletta is... + +2025-06-01 19:15:14,394 - INFO - Processing audio with duration 8.020 seconds +2025-06-01 19:15:14,570 - INFO - Processing audio with duration 00:08.020 +2025-06-01 19:15:14,943 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:14,943 - INFO - Got result for chunk: output_test5/chunks/chunk_009.wav +2025-06-01 19:15:14,943 - INFO - +====== Streaming Chunk 010 ====== +[2025-06-01 19:13:25] Speaker 5: another software engineers somia on the call is our data scientist selection ml engineer +[2025-06-01 19:13:30] Speaker 5: uh + +2025-06-01 19:15:14,943 - INFO - Processing audio with duration 8.040 seconds +2025-06-01 19:15:15,111 - INFO - Processing audio with duration 00:08.040 +2025-06-01 19:15:15,113 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:15,239 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:15,283 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:15,283 - INFO - Got result for chunk: output_test5/chunks/chunk_010.wav +2025-06-01 19:15:15,284 - INFO - +====== Streaming Chunk 011 ====== +[2025-06-01 19:13:25] Speaker 5: also have uh hithika who's our golang developer she's ex uh akis team so that's where if you all can + +2025-06-01 19:15:15,284 - INFO - Processing audio with duration 5.960 seconds +2025-06-01 19:15:15,455 - INFO - Processing audio with duration 00:05.960 +2025-06-01 19:15:15,499 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:15,499 - INFO - Got result for chunk: output_test5/chunks/chunk_011.wav +2025-06-01 19:15:15,500 - INFO - +====== Streaming Chunk 012 ====== +[2025-06-01 19:13:25] Speaker 5: connect a little bit too. +[2025-06-01 19:13:27] Speaker 5: All right. Today, I'm going to keep it more focused on the demo. + +2025-06-01 19:15:15,500 - INFO - Processing audio with duration 6.260 seconds +2025-06-01 19:15:15,669 - INFO - Processing audio with duration 00:06.260 +2025-06-01 19:15:15,671 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:15,798 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:15,847 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:15,847 - INFO - Got result for chunk: output_test5/chunks/chunk_012.wav +2025-06-01 19:15:15,847 - INFO - +====== Streaming Chunk 013 ====== +[2025-06-01 19:13:25] Speaker 5: and what this tool is all about. +[2025-06-01 19:13:27] Speaker 5: And then we'll, you know, answer questions +[2025-06-01 19:13:29] Speaker 5: and kind of talk through. + +2025-06-01 19:15:15,848 - INFO - Processing audio with duration 5.240 seconds +2025-06-01 19:15:16,021 - INFO - Processing audio with duration 00:05.240 +2025-06-01 19:15:16,055 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:16,055 - INFO - Got result for chunk: output_test5/chunks/chunk_013.wav +2025-06-01 19:15:16,055 - INFO - +====== Streaming Chunk 014 ====== +[2025-06-01 19:13:25] Speaker 5: But most importantly, we want to hear if the current version has a set of features you're + +2025-06-01 19:15:16,056 - INFO - Processing audio with duration 6.500 seconds +2025-06-01 19:15:16,224 - INFO - Processing audio with duration 00:06.500 +2025-06-01 19:15:16,227 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:16,354 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:16,390 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:16,391 - INFO - Got result for chunk: output_test5/chunks/chunk_014.wav +2025-06-01 19:15:16,391 - INFO - +====== Streaming Chunk 015 ====== +[2025-06-01 19:13:25] Speaker 5: looking for? Is it missing? Is it you want to see something more? Because we are kind of very + +2025-06-01 19:15:16,391 - INFO - Processing audio with duration 9.500 seconds +2025-06-01 19:15:16,563 - INFO - Processing audio with duration 00:09.500 +2025-06-01 19:15:16,596 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:16,596 - INFO - Got result for chunk: output_test5/chunks/chunk_015.wav +2025-06-01 19:15:16,596 - INFO - +====== Streaming Chunk 016 ====== +[2025-06-01 19:13:25] Speaker 5: rapidly on a weekly sprint uh releasing the features to this and we are really happy to incorporate + +2025-06-01 19:15:16,597 - INFO - Processing audio with duration 10.540 seconds +2025-06-01 19:15:16,770 - INFO - Processing audio with duration 00:10.540 +2025-06-01 19:15:16,772 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:16,899 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:16,941 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:16,941 - INFO - Got result for chunk: output_test5/chunks/chunk_016.wav +2025-06-01 19:15:16,941 - INFO - +====== Streaming Chunk 017 ====== +[2025-06-01 19:13:25] Speaker 5: anything and work on how you can onboard to this by the way +[2025-06-01 19:13:31] Speaker 5: uh so four areas where we are focused on + +2025-06-01 19:15:16,942 - INFO - Processing audio with duration 7.300 seconds +2025-06-01 19:15:17,115 - INFO - Processing audio with duration 00:07.300 +2025-06-01 19:15:17,384 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:17,384 - INFO - Got result for chunk: output_test5/chunks/chunk_017.wav +2025-06-01 19:15:17,385 - INFO - +====== Streaming Chunk 018 ====== +[2025-06-01 19:13:25] Speaker 5: this solution is continuous GPU and cluster level monitoring of both NVIDIA GPUs and AMD GPUs. + +2025-06-01 19:15:17,385 - INFO - Processing audio with duration 8.980 seconds +2025-06-01 19:15:17,555 - INFO - Processing audio with duration 00:08.980 +2025-06-01 19:15:17,557 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:17,682 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:17,728 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:17,728 - INFO - Got result for chunk: output_test5/chunks/chunk_018.wav +2025-06-01 19:15:17,728 - INFO - +====== Streaming Chunk 019 ====== +[2025-06-01 19:13:25] Speaker 5: together, right? So this is how we want to kind of platform +[2025-06-01 19:13:29] Speaker 5: agnostic solution that works for all GPUs. + +2025-06-01 19:15:17,728 - INFO - Processing audio with duration 5.840 seconds +2025-06-01 19:15:17,899 - INFO - Processing audio with duration 00:05.840 +2025-06-01 19:15:17,943 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:17,943 - INFO - Got result for chunk: output_test5/chunks/chunk_019.wav +2025-06-01 19:15:17,944 - INFO - +====== Streaming Chunk 020 ====== +[2025-06-01 19:13:25] Speaker 5: and has all the latest metrics available for you to digest and look at. +[2025-06-01 19:13:32] Speaker 5: The next thing which we've highlighted... + +2025-06-01 19:15:17,944 - INFO - Processing audio with duration 7.740 seconds +2025-06-01 19:15:18,114 - INFO - Processing audio with duration 00:07.740 +2025-06-01 19:15:18,116 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:18,242 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:18,280 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:18,280 - INFO - Got result for chunk: output_test5/chunks/chunk_020.wav +2025-06-01 19:15:18,281 - INFO - +====== Streaming Chunk 021 ====== +[2025-06-01 19:13:25] Speaker 5: focused on is the health checks piece to this right are my nodes healthy are they performing to +[2025-06-01 19:13:30] Speaker 5: what they + +2025-06-01 19:15:18,281 - INFO - Processing audio with duration 6.020 seconds +2025-06-01 19:15:18,451 - INFO - Processing audio with duration 00:06.020 +2025-06-01 19:15:18,496 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:18,496 - INFO - Got result for chunk: output_test5/chunks/chunk_021.wav +2025-06-01 19:15:18,497 - INFO - +====== Streaming Chunk 022 ====== +[2025-06-01 19:13:25] Speaker 5: should be uh active health checks active monitoring is is is a huge investment that we did uh and i'll +[2025-06-01 19:13:32] Speaker 5: go over some + +2025-06-01 19:15:18,497 - INFO - Processing audio with duration 7.200 seconds +2025-06-01 19:15:18,666 - INFO - Processing audio with duration 00:07.200 +2025-06-01 19:15:18,668 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:18,795 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:18,824 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:18,825 - INFO - Got result for chunk: output_test5/chunks/chunk_022.wav +2025-06-01 19:15:18,825 - INFO - +====== Streaming Chunk 023 ====== +[2025-06-01 19:13:25] Speaker 5: details about what all health checks that we have enabled including the RDMA cluster level. + +2025-06-01 19:15:18,825 - INFO - Processing audio with duration 6.400 seconds +2025-06-01 19:15:18,997 - INFO - Processing audio with duration 00:06.400 +2025-06-01 19:15:19,041 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:19,041 - INFO - Got result for chunk: output_test5/chunks/chunk_023.wav +2025-06-01 19:15:19,041 - INFO - +====== Streaming Chunk 024 ====== +[2025-06-01 19:13:25] Speaker 5: checks that we call MPI tests as part of the solution fully baked into. +[2025-06-01 19:13:30] Speaker 5: We kept it to cloud native. + +2025-06-01 19:15:19,041 - INFO - Processing audio with duration 5.680 seconds +2025-06-01 19:15:19,211 - INFO - Processing audio with duration 00:05.680 +2025-06-01 19:15:19,214 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:19,340 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:19,387 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:19,387 - INFO - Got result for chunk: output_test5/chunks/chunk_024.wav +2025-06-01 19:15:19,387 - INFO - +====== Streaming Chunk 025 ====== +[2025-06-01 19:13:25] Speaker 5: because I heard it very clear that it has to be Grafana, Prometheus, Native, or Otel, anything open source. + +2025-06-01 19:15:19,387 - INFO - Processing audio with duration 7.280 seconds +2025-06-01 19:15:19,558 - INFO - Processing audio with duration 00:07.280 +2025-06-01 19:15:19,821 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:19,821 - INFO - Got result for chunk: output_test5/chunks/chunk_025.wav +2025-06-01 19:15:19,821 - INFO - +====== Streaming Chunk 026 ====== +[2025-06-01 19:13:25] Speaker 5: Cloud Native Alliance so that it's easy to be move between clouds or on-prem and use +[2025-06-01 19:13:29] Speaker 5: the same existing. + +2025-06-01 19:15:19,821 - INFO - Processing audio with duration 4.400 seconds +2025-06-01 19:15:19,989 - INFO - Processing audio with duration 00:04.400 +2025-06-01 19:15:19,991 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:20,118 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:20,150 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:20,151 - INFO - Got result for chunk: output_test5/chunks/chunk_026.wav +2025-06-01 19:15:20,151 - INFO - +====== Streaming Chunk 027 ====== +[2025-06-01 19:13:25] Speaker 5: tools you always used for for monitoring another area that we've paid attention and we built is the + +2025-06-01 19:15:20,151 - INFO - Processing audio with duration 6.220 seconds +2025-06-01 19:15:20,319 - INFO - Processing audio with duration 00:06.220 +2025-06-01 19:15:20,352 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:20,352 - INFO - Got result for chunk: output_test5/chunks/chunk_027.wav +2025-06-01 19:15:20,353 - INFO - +====== Streaming Chunk 028 ====== +[2025-06-01 19:13:25] Speaker 5: team level tracking. A lot of these large size clusters are usually used by multiple teams and + +2025-06-01 19:15:20,353 - INFO - Processing audio with duration 5.700 seconds +2025-06-01 19:15:20,522 - INFO - Processing audio with duration 00:05.700 +2025-06-01 19:15:20,525 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:20,651 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:20,688 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:20,688 - INFO - Got result for chunk: output_test5/chunks/chunk_028.wav +2025-06-01 19:15:20,689 - INFO - +====== Streaming Chunk 029 ====== +[2025-06-01 19:13:25] Speaker 5: everybody wants to know the health of their own subset of systems or how their experiment is +[2025-06-01 19:13:30] Speaker 5: performing. + +2025-06-01 19:15:20,689 - INFO - Processing audio with duration 5.700 seconds +2025-06-01 19:15:20,858 - INFO - Processing audio with duration 00:05.700 +2025-06-01 19:15:20,895 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:20,895 - INFO - Got result for chunk: output_test5/chunks/chunk_029.wav +2025-06-01 19:15:20,896 - INFO - +====== Streaming Chunk 030 ====== +[2025-06-01 19:13:25] Speaker 5: if they have an unhealthy node, all of those combinations. +[2025-06-01 19:13:29] Speaker 5: And another important thing is about + +2025-06-01 19:15:20,896 - INFO - Processing audio with duration 5.380 seconds +2025-06-01 19:15:21,065 - INFO - Processing audio with duration 00:05.380 +2025-06-01 19:15:21,067 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:21,194 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:21,224 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:21,224 - INFO - Got result for chunk: output_test5/chunks/chunk_030.wav +2025-06-01 19:15:21,224 - INFO - +====== Streaming Chunk 031 ====== +[2025-06-01 19:13:25] Speaker 5: cost tracking right people want to see how much computer resources they've used uh whatever the + +2025-06-01 19:15:21,224 - INFO - Processing audio with duration 8.160 seconds +2025-06-01 19:15:21,393 - INFO - Processing audio with duration 00:08.160 +2025-06-01 19:15:21,434 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:21,434 - INFO - Got result for chunk: output_test5/chunks/chunk_031.wav +2025-06-01 19:15:21,435 - INFO - +====== Streaming Chunk 032 ====== +[2025-06-01 19:13:25] Speaker 5: the power the total gpu power they have used for running this experiment right those were a few +[2025-06-01 19:13:29] Speaker 5: things that + +2025-06-01 19:15:21,435 - INFO - Processing audio with duration 7.380 seconds +2025-06-01 19:15:21,605 - INFO - Processing audio with duration 00:07.380 +2025-06-01 19:15:21,607 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:21,734 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:21,782 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:21,782 - INFO - Got result for chunk: output_test5/chunks/chunk_032.wav +2025-06-01 19:15:21,782 - INFO - +====== Streaming Chunk 033 ====== +[2025-06-01 19:13:25] Speaker 5: that we also focused to build on this. +[2025-06-01 19:13:29] Speaker 5: The current set of features that we have +[2025-06-01 19:13:32] Speaker 5: and what we have built right now + +2025-06-01 19:15:21,783 - INFO - Processing audio with duration 7.360 seconds +2025-06-01 19:15:21,952 - INFO - Processing audio with duration 00:07.360 +2025-06-01 19:15:22,238 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:22,239 - INFO - Got result for chunk: output_test5/chunks/chunk_033.wav +2025-06-01 19:15:22,239 - INFO - +====== Streaming Chunk 034 ====== +[2025-06-01 19:13:25] Speaker 5: now is first thing tenancy level monitoring you no longer have region barriers right so we any + +2025-06-01 19:15:22,239 - INFO - Processing audio with duration 5.680 seconds +2025-06-01 19:15:22,406 - INFO - Processing audio with duration 00:05.680 +2025-06-01 19:15:22,408 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:22,534 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:22,578 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:22,579 - INFO - Got result for chunk: output_test5/chunks/chunk_034.wav +2025-06-01 19:15:22,579 - INFO - +====== Streaming Chunk 035 ====== +[2025-06-01 19:13:25] Speaker 5: data, any region, OCI region you may be in, everything is monitorable as a single instance +[2025-06-01 19:13:31] Speaker 5: for us. + +2025-06-01 19:15:22,579 - INFO - Processing audio with duration 7.060 seconds +2025-06-01 19:15:22,748 - INFO - Processing audio with duration 00:07.060 +2025-06-01 19:15:22,792 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:22,792 - INFO - Got result for chunk: output_test5/chunks/chunk_035.wav +2025-06-01 19:15:22,792 - INFO - +====== Streaming Chunk 036 ====== +[2025-06-01 19:13:25] Speaker 5: So I'm going to show you a demo. +[2025-06-01 19:13:27] Speaker 5: We allow you to monitor either single, bare metal, virtual machine + +2025-06-01 19:15:22,792 - INFO - Processing audio with duration 7.180 seconds +2025-06-01 19:15:22,963 - INFO - Processing audio with duration 00:07.180 +2025-06-01 19:15:22,966 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:23,091 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:23,131 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:23,131 - INFO - Got result for chunk: output_test5/chunks/chunk_036.wav +2025-06-01 19:15:23,131 - INFO - +====== Streaming Chunk 037 ====== +[2025-06-01 19:13:25] Speaker 5: instances or a full oke cluster or an hpc cluster that if you may be using slurm or a native uh + +2025-06-01 19:15:23,131 - INFO - Processing audio with duration 4.220 seconds +2025-06-01 19:15:23,299 - INFO - Processing audio with duration 00:04.220 +2025-06-01 19:15:23,340 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:23,340 - INFO - Got result for chunk: output_test5/chunks/chunk_037.wav +2025-06-01 19:15:23,341 - INFO - +====== Streaming Chunk 038 ====== +[2025-06-01 19:13:25] Speaker 5: called cluster network, compute cluster setup in OCI. +[2025-06-01 19:13:29] Speaker 5: The third part of the feature that we worked on + +2025-06-01 19:15:23,341 - INFO - Processing audio with duration 4.580 seconds +2025-06-01 19:15:23,508 - INFO - Processing audio with duration 00:04.580 +2025-06-01 19:15:23,511 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:23,637 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:23,675 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:23,676 - INFO - Got result for chunk: output_test5/chunks/chunk_038.wav +2025-06-01 19:15:23,676 - INFO - +====== Streaming Chunk 039 ====== +[2025-06-01 19:13:25] Speaker 5: and all of this we're going to see a demo quickly but i'm going to just spend maybe five more minutes + +2025-06-01 19:15:23,676 - INFO - Processing audio with duration 5.180 seconds +2025-06-01 19:15:23,844 - INFO - Processing audio with duration 00:05.180 +2025-06-01 19:15:23,888 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:23,889 - INFO - Got result for chunk: output_test5/chunks/chunk_039.wav +2025-06-01 19:15:23,889 - INFO - +====== Streaming Chunk 040 ====== +[2025-06-01 19:13:25] Speaker 5: and then go to the demo. +[2025-06-01 19:13:26] Speaker 5: Is it team level tracking? +[2025-06-01 19:13:27] Speaker 5: You can create team level tracking. + +2025-06-01 19:15:23,889 - INFO - Processing audio with duration 5.400 seconds +2025-06-01 19:15:24,060 - INFO - Processing audio with duration 00:05.400 +2025-06-01 19:15:24,063 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:24,189 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:24,223 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:24,223 - INFO - Got result for chunk: output_test5/chunks/chunk_040.wav +2025-06-01 19:15:24,224 - INFO - +====== Streaming Chunk 041 ====== +[2025-06-01 19:13:25] Speaker 5: handpicking whichever nodes are part of this experiment. +[2025-06-01 19:13:29] Speaker 5: We are also working towards... + +2025-06-01 19:15:24,224 - INFO - Processing audio with duration 4.500 seconds +2025-06-01 19:15:24,393 - INFO - Processing audio with duration 00:04.500 +2025-06-01 19:15:24,677 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:24,677 - INFO - Got result for chunk: output_test5/chunks/chunk_041.wav +2025-06-01 19:15:24,677 - INFO - +====== Streaming Chunk 042 ====== +[2025-06-01 19:13:25] Speaker 5: automatically fetching the nodes that are running experiment based on Kubernetes +[2025-06-01 19:13:29] Speaker 5: how the job system. + +2025-06-01 19:15:24,677 - INFO - Processing audio with duration 8.040 seconds +2025-06-01 19:15:24,849 - INFO - Processing audio with duration 00:08.040 +2025-06-01 19:15:24,852 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:24,977 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:25,008 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:25,009 - INFO - Got result for chunk: output_test5/chunks/chunk_042.wav +2025-06-01 19:15:25,009 - INFO - +====== Streaming Chunk 043 ====== +[2025-06-01 19:13:25] Speaker 5: scheduled. So that's another area we're working on automatically, dynamically pulling the nodes that + +2025-06-01 19:15:25,009 - INFO - Processing audio with duration 5.120 seconds +2025-06-01 19:15:25,182 - INFO - Processing audio with duration 00:05.120 +2025-06-01 19:15:25,224 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:25,224 - INFO - Got result for chunk: output_test5/chunks/chunk_043.wav +2025-06-01 19:15:25,224 - INFO - +====== Streaming Chunk 044 ====== +[2025-06-01 19:13:25] Speaker 5: part of that experiment rather than you allocating what compute nodes go to. +[2025-06-01 19:13:31] Speaker 5: This I've already said. + +2025-06-01 19:15:25,224 - INFO - Processing audio with duration 5.460 seconds +2025-06-01 19:15:25,395 - INFO - Processing audio with duration 00:05.460 +2025-06-01 19:15:25,397 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:25,524 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:25,560 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:25,560 - INFO - Got result for chunk: output_test5/chunks/chunk_044.wav +2025-06-01 19:15:25,560 - INFO - +====== Streaming Chunk 045 ====== +[2025-06-01 19:13:25] Speaker 5: already have both of them running and i'm going to show you that uh the difference between what we do + +2025-06-01 19:15:25,560 - INFO - Processing audio with duration 6.300 seconds +2025-06-01 19:15:25,729 - INFO - Processing audio with duration 00:06.300 +2025-06-01 19:15:25,764 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:25,764 - INFO - Got result for chunk: output_test5/chunks/chunk_045.wav +2025-06-01 19:15:25,764 - INFO - +====== Streaming Chunk 046 ====== +[2025-06-01 19:13:25] Speaker 5: with performance monitoring and health check is we go very close to the layers that an ML engineer would + +2025-06-01 19:15:25,764 - INFO - Processing audio with duration 7.480 seconds +2025-06-01 19:15:25,934 - INFO - Processing audio with duration 00:07.480 +2025-06-01 19:15:25,936 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:26,062 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:26,115 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:26,115 - INFO - Got result for chunk: output_test5/chunks/chunk_046.wav +2025-06-01 19:15:26,115 - INFO - +====== Streaming Chunk 047 ====== +[2025-06-01 19:13:25] Speaker 5: operate under, which is PyTorch for us, PyTorch and JAX primarily, right? So we picked up PyTorch base. + +2025-06-01 19:15:26,115 - INFO - Processing audio with duration 5.640 seconds +2025-06-01 19:15:26,284 - INFO - Processing audio with duration 00:05.640 +2025-06-01 19:15:26,316 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:26,317 - INFO - Got result for chunk: output_test5/chunks/chunk_047.wav +2025-06-01 19:15:26,317 - INFO - +====== Streaming Chunk 048 ====== +[2025-06-01 19:13:25] Speaker 5: matmal and linear regression based performance testing to achieve how many flops did we achieve + +2025-06-01 19:15:26,317 - INFO - Processing audio with duration 4.780 seconds +2025-06-01 19:15:26,483 - INFO - Processing audio with duration 00:04.780 +2025-06-01 19:15:26,486 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:26,612 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:26,646 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:26,646 - INFO - Got result for chunk: output_test5/chunks/chunk_048.wav +2025-06-01 19:15:26,646 - INFO - +====== Streaming Chunk 049 ====== +[2025-06-01 19:13:25] Speaker 5: on this compute node how much is each gpu performing and is it is it within the threshold of + +2025-06-01 19:15:26,647 - INFO - Processing audio with duration 7.080 seconds +2025-06-01 19:15:26,820 - INFO - Processing audio with duration 00:07.080 +2025-06-01 19:15:27,104 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:27,104 - INFO - Got result for chunk: output_test5/chunks/chunk_049.wav +2025-06-01 19:15:27,105 - INFO - +====== Streaming Chunk 050 ====== +[2025-06-01 19:13:25] Speaker 5: the specification from NVIDIA or AMD what we have traditionally seen. So we have a baseline. + +2025-06-01 19:15:27,105 - INFO - Processing audio with duration 5.740 seconds +2025-06-01 19:15:27,274 - INFO - Processing audio with duration 00:05.740 +2025-06-01 19:15:27,276 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:27,403 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:27,437 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:27,438 - INFO - Got result for chunk: output_test5/chunks/chunk_050.wav +2025-06-01 19:15:27,438 - INFO - +====== Streaming Chunk 051 ====== +[2025-06-01 19:13:25] Speaker 5: that we score the performance to because it's the standard set of precision based testing either FBA. + +2025-06-01 19:15:27,438 - INFO - Processing audio with duration 6.080 seconds +2025-06-01 19:15:27,614 - INFO - Processing audio with duration 00:06.080 +2025-06-01 19:15:27,658 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:27,658 - INFO - Got result for chunk: output_test5/chunks/chunk_051.wav +2025-06-01 19:15:27,658 - INFO - +====== Streaming Chunk 052 ====== +[2025-06-01 19:13:25] Speaker 5: FP16, FP32, it automatically does all of this. And most of this code, by the way, for the health + +2025-06-01 19:15:27,659 - INFO - Processing audio with duration 6.280 seconds +2025-06-01 19:15:27,834 - INFO - Processing audio with duration 00:06.280 +2025-06-01 19:15:27,837 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:27,963 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:27,998 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:27,999 - INFO - Got result for chunk: output_test5/chunks/chunk_052.wav +2025-06-01 19:15:27,999 - INFO - +====== Streaming Chunk 053 ====== +[2025-06-01 19:13:25] Speaker 5: will all be open source so you can see you can change the things if you like or add to it + +2025-06-01 19:15:27,999 - INFO - Processing audio with duration 7.580 seconds +2025-06-01 19:15:28,169 - INFO - Processing audio with duration 00:07.580 +2025-06-01 19:15:28,218 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:28,218 - INFO - Got result for chunk: output_test5/chunks/chunk_053.wav +2025-06-01 19:15:28,218 - INFO - +====== Streaming Chunk 054 ====== +[2025-06-01 19:13:25] Speaker 5: right and please feel free to contribute to it too. +[2025-06-01 19:13:28] Speaker 5: This you've seen it. +[2025-06-01 19:13:30] Speaker 5: The approach we have taken. + +2025-06-01 19:15:28,218 - INFO - Processing audio with duration 6.040 seconds +2025-06-01 19:15:28,386 - INFO - Processing audio with duration 00:06.040 +2025-06-01 19:15:28,388 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:28,515 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:28,549 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:28,550 - INFO - Got result for chunk: output_test5/chunks/chunk_054.wav +2025-06-01 19:15:28,550 - INFO - +====== Streaming Chunk 055 ====== +[2025-06-01 19:13:25] Speaker 5: here is instead of asking you to poke holes in your existing cluster or networks, we basically push the + +2025-06-01 19:15:28,550 - INFO - Processing audio with duration 5.520 seconds +2025-06-01 19:15:28,719 - INFO - Processing audio with duration 00:05.520 +2025-06-01 19:15:28,754 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:28,754 - INFO - Got result for chunk: output_test5/chunks/chunk_055.wav +2025-06-01 19:15:28,754 - INFO - +====== Streaming Chunk 056 ====== +[2025-06-01 19:13:25] Speaker 5: metrics and we push the health check data to the Prometheus and our central control plane. + +2025-06-01 19:15:28,754 - INFO - Processing audio with duration 4.880 seconds +2025-06-01 19:15:28,926 - INFO - Processing audio with duration 00:04.880 +2025-06-01 19:15:28,928 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:29,054 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:29,091 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:29,092 - INFO - Got result for chunk: output_test5/chunks/chunk_056.wav +2025-06-01 19:15:29,092 - INFO - +====== Streaming Chunk 057 ====== +[2025-06-01 19:13:25] Speaker 5: It could be looked at as, oh, I need to open this port, that port, no exception. + +2025-06-01 19:15:29,092 - INFO - Processing audio with duration 5.120 seconds +2025-06-01 19:15:29,268 - INFO - Processing audio with duration 00:05.120 +2025-06-01 19:15:29,545 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:29,545 - INFO - Got result for chunk: output_test5/chunks/chunk_057.wav +2025-06-01 19:15:29,545 - INFO - +====== Streaming Chunk 058 ====== +[2025-06-01 19:13:25] Speaker 5: if you're in a vc and in a production environment where everything is locked down egress is usually + +2025-06-01 19:15:29,545 - INFO - Processing audio with duration 4.640 seconds +2025-06-01 19:15:29,717 - INFO - Processing audio with duration 00:04.640 +2025-06-01 19:15:29,719 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:29,846 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:29,880 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:29,880 - INFO - Got result for chunk: output_test5/chunks/chunk_058.wav +2025-06-01 19:15:29,880 - INFO - +====== Streaming Chunk 059 ====== +[2025-06-01 19:13:25] Speaker 5: easier and you can always sniff the egress but these are all running within your own tenancy so + +2025-06-01 19:15:29,881 - INFO - Processing audio with duration 6.980 seconds +2025-06-01 19:15:30,053 - INFO - Processing audio with duration 00:06.980 +2025-06-01 19:15:30,090 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:30,090 - INFO - Got result for chunk: output_test5/chunks/chunk_059.wav +2025-06-01 19:15:30,090 - INFO - +====== Streaming Chunk 060 ====== +[2025-06-01 19:13:25] Speaker 5: So the origination of all of the metrics to where it's being sent is all within your network. + +2025-06-01 19:15:30,091 - INFO - Processing audio with duration 5.800 seconds +2025-06-01 19:15:30,268 - INFO - Processing audio with duration 00:05.800 +2025-06-01 19:15:30,270 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:30,396 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:30,429 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:30,430 - INFO - Got result for chunk: output_test5/chunks/chunk_060.wav +2025-06-01 19:15:30,430 - INFO - +====== Streaming Chunk 061 ====== +[2025-06-01 19:13:25] Speaker 5: nothing is going into the public. All right. So I'm going to go over this. + +2025-06-01 19:15:30,430 - INFO - Processing audio with duration 4.800 seconds +2025-06-01 19:15:30,600 - INFO - Processing audio with duration 00:04.800 +2025-06-01 19:15:30,651 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:30,651 - INFO - Got result for chunk: output_test5/chunks/chunk_061.wav +2025-06-01 19:15:30,651 - INFO - +====== Streaming Chunk 062 ====== +[2025-06-01 19:13:25] Speaker 5: but there are plenty of metrics, just a lot. +[2025-06-01 19:13:28] Speaker 5: And it's kind of an eye candy chart on Grafana's to look. + +2025-06-01 19:15:30,651 - INFO - Processing audio with duration 6.560 seconds +2025-06-01 19:15:30,820 - INFO - Processing audio with duration 00:06.560 +2025-06-01 19:15:30,823 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:30,949 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:30,985 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:30,986 - INFO - Got result for chunk: output_test5/chunks/chunk_062.wav +2025-06-01 19:15:30,986 - INFO - +====== Streaming Chunk 063 ====== +[2025-06-01 19:13:25] Speaker 5: at but i think you get the point right you have full nvidia dcgm exporter all metrics + +2025-06-01 19:15:30,986 - INFO - Processing audio with duration 6.020 seconds +2025-06-01 19:15:31,157 - INFO - Processing audio with duration 00:06.020 +2025-06-01 19:15:31,202 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:31,202 - INFO - Got result for chunk: output_test5/chunks/chunk_063.wav +2025-06-01 19:15:31,202 - INFO - +====== Streaming Chunk 064 ====== +[2025-06-01 19:13:25] Speaker 5: You have all AMD SMI metrics. +[2025-06-01 19:13:27] Speaker 5: You have all of the RDMA metrics that is particularly on our ROC. + +2025-06-01 19:15:31,202 - INFO - Processing audio with duration 6.880 seconds +2025-06-01 19:15:31,370 - INFO - Processing audio with duration 00:06.880 +2025-06-01 19:15:31,373 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:31,499 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:31,529 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:31,529 - INFO - Got result for chunk: output_test5/chunks/chunk_064.wav +2025-06-01 19:15:31,530 - INFO - +====== Streaming Chunk 065 ====== +[2025-06-01 19:13:25] Speaker 5: implementation that we sniffed the melanocs drivers for to capture that front end next + +2025-06-01 19:15:31,530 - INFO - Processing audio with duration 5.660 seconds +2025-06-01 19:15:31,704 - INFO - Processing audio with duration 00:05.660 +2025-06-01 19:15:31,977 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:31,977 - INFO - Got result for chunk: output_test5/chunks/chunk_065.wav +2025-06-01 19:15:31,977 - INFO - +====== Streaming Chunk 066 ====== +[2025-06-01 19:13:25] Speaker 5: node health is included, health check, full check is included, +[2025-06-01 19:13:29] Speaker 5: and the traditional disk IO usage. + +2025-06-01 19:15:31,977 - INFO - Processing audio with duration 0.400 seconds +2025-06-01 19:15:32,135 - INFO - Processing audio with duration 00:00.400 +2025-06-01 19:15:32,137 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:32,262 - INFO - Detected language 'en' with probability 0.86 +2025-06-01 19:15:32,306 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:32,306 - INFO - Got result for chunk: output_test5/chunks/chunk_066.wav +2025-06-01 19:15:32,307 - INFO - +====== Streaming Chunk 067 ====== +[2025-06-01 19:13:25] Speaker 5: all of those that you usually get with Prometheus already included. +[2025-06-01 19:13:28] Speaker 5: So all of these is bundled as one. + +2025-06-01 19:15:32,307 - INFO - Processing audio with duration 5.400 seconds +2025-06-01 19:15:32,477 - INFO - Processing audio with duration 00:05.400 +2025-06-01 19:15:32,490 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:32,490 - INFO - Got result for chunk: output_test5/chunks/chunk_067.wav +2025-06-01 19:15:32,491 - INFO - +====== Streaming Chunk 068 ====== +[2025-06-01 19:13:25] Speaker 5: unpacking. + +2025-06-01 19:15:32,491 - INFO - Processing audio with duration 5.980 seconds +2025-06-01 19:15:32,662 - INFO - Processing audio with duration 00:05.980 +2025-06-01 19:15:32,665 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:32,791 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:32,830 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:32,831 - INFO - Got result for chunk: output_test5/chunks/chunk_068.wav +2025-06-01 19:15:32,831 - INFO - +====== Streaming Chunk 069 ====== +[2025-06-01 19:13:25] Speaker 5: On the health checks piece, these are all the tests we do today and we'll continue to add more or adjust. + +2025-06-01 19:15:32,831 - INFO - Processing audio with duration 5.060 seconds +2025-06-01 19:15:33,004 - INFO - Processing audio with duration 00:05.060 +2025-06-01 19:15:33,035 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:33,035 - INFO - Got result for chunk: output_test5/chunks/chunk_069.wav +2025-06-01 19:15:33,036 - INFO - +====== Streaming Chunk 070 ====== +[2025-06-01 19:13:25] Speaker 5: based on how performance we see we do extensive benchmarking to see where the threshold should + +2025-06-01 19:15:33,036 - INFO - Processing audio with duration 8.800 seconds +2025-06-01 19:15:33,204 - INFO - Processing audio with duration 00:08.800 +2025-06-01 19:15:33,206 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:33,332 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:33,367 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:33,368 - INFO - Got result for chunk: output_test5/chunks/chunk_070.wav +2025-06-01 19:15:33,368 - INFO - +====== Streaming Chunk 071 ====== +[2025-06-01 19:13:25] Speaker 5: be or the baseline should be and then we either go up and down based on how we are seeing. + +2025-06-01 19:15:33,368 - INFO - Processing audio with duration 5.240 seconds +2025-06-01 19:15:33,538 - INFO - Processing audio with duration 00:05.240 +2025-06-01 19:15:33,585 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:33,585 - INFO - Got result for chunk: output_test5/chunks/chunk_071.wav +2025-06-01 19:15:33,585 - INFO - +====== Streaming Chunk 072 ====== +[2025-06-01 19:13:25] Speaker 5: example the what should be the ideal mfu on a 140 gb against b200 or mi300x right and then we we lower it + +2025-06-01 19:15:33,585 - INFO - Processing audio with duration 6.020 seconds +2025-06-01 19:15:33,754 - INFO - Processing audio with duration 00:06.020 +2025-06-01 19:15:33,756 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:33,882 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:33,920 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:33,921 - INFO - Got result for chunk: output_test5/chunks/chunk_072.wav +2025-06-01 19:15:33,921 - INFO - +====== Streaming Chunk 073 ====== +[2025-06-01 19:13:25] Speaker 5: if needed right so but we work with our vendors to also see if this we already doing the right way or not + +2025-06-01 19:15:33,921 - INFO - Processing audio with duration 3.100 seconds +2025-06-01 19:15:34,089 - INFO - Processing audio with duration 00:03.100 +2025-06-01 19:15:34,405 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:34,405 - INFO - Got result for chunk: output_test5/chunks/chunk_073.wav +2025-06-01 19:15:34,405 - INFO - +====== Streaming Chunk 074 ====== +[2025-06-01 19:13:25] Speaker 5: So here are all the checks we run. +[2025-06-01 19:13:27] Speaker 5: And all of these, once they complete, you will see a... + +2025-06-01 19:15:34,405 - INFO - Processing audio with duration 4.120 seconds +2025-06-01 19:15:34,572 - INFO - Processing audio with duration 00:04.120 +2025-06-01 19:15:34,574 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:34,700 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:34,724 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:34,724 - INFO - Got result for chunk: output_test5/chunks/chunk_074.wav +2025-06-01 19:15:34,725 - INFO - +====== Streaming Chunk 075 ====== +[2025-06-01 19:13:25] Speaker 5: metrics flowing through Grafana and you'll also see this. + +2025-06-01 19:15:34,725 - INFO - Processing audio with duration 5.180 seconds +2025-06-01 19:15:34,892 - INFO - Processing audio with duration 00:05.180 +2025-06-01 19:15:34,930 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:34,931 - INFO - Got result for chunk: output_test5/chunks/chunk_075.wav +2025-06-01 19:15:34,931 - INFO - +====== Streaming Chunk 076 ====== +[2025-06-01 19:13:25] Speaker 5: Very simple. This is how the architecture is. I'm not going to spend too much time here either. + +2025-06-01 19:15:34,931 - INFO - Processing audio with duration 8.840 seconds +2025-06-01 19:15:35,101 - INFO - Processing audio with duration 00:08.840 +2025-06-01 19:15:35,104 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:35,230 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:35,265 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:35,265 - INFO - Got result for chunk: output_test5/chunks/chunk_076.wav +2025-06-01 19:15:35,266 - INFO - +====== Streaming Chunk 077 ====== +[2025-06-01 19:13:25] Speaker 5: because we are evolving how this is going to go right now the way we will release it is this will + +2025-06-01 19:15:35,266 - INFO - Processing audio with duration 7.000 seconds +2025-06-01 19:15:35,437 - INFO - Processing audio with duration 00:07.000 +2025-06-01 19:15:35,471 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:35,472 - INFO - Got result for chunk: output_test5/chunks/chunk_077.wav +2025-06-01 19:15:35,472 - INFO - +====== Streaming Chunk 078 ====== +[2025-06-01 19:13:25] Speaker 5: deploy as a we're not a kubernetes operator yet but you have to basically deploy a dedicated + +2025-06-01 19:15:35,472 - INFO - Processing audio with duration 4.760 seconds +2025-06-01 19:15:35,642 - INFO - Processing audio with duration 00:04.760 +2025-06-01 19:15:35,644 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:35,771 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:35,809 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:35,809 - INFO - Got result for chunk: output_test5/chunks/chunk_078.wav +2025-06-01 19:15:35,810 - INFO - +====== Streaming Chunk 079 ====== +[2025-06-01 19:13:25] Speaker 5: OKE cluster with just CPU nodes, which installs Prometheus Grafana open source and our control. + +2025-06-01 19:15:35,810 - INFO - Processing audio with duration 7.280 seconds +2025-06-01 19:15:35,982 - INFO - Processing audio with duration 00:07.280 +2025-06-01 19:15:36,016 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:36,016 - INFO - Got result for chunk: output_test5/chunks/chunk_079.wav +2025-06-01 19:15:36,016 - INFO - +====== Streaming Chunk 080 ====== +[2025-06-01 19:13:25] Speaker 5: plane api as well as portal right those are all the components that come as part of this + +2025-06-01 19:15:36,016 - INFO - Processing audio with duration 8.900 seconds +2025-06-01 19:15:36,186 - INFO - Processing audio with duration 00:08.900 +2025-06-01 19:15:36,188 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:36,314 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:36,363 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:36,363 - INFO - Got result for chunk: output_test5/chunks/chunk_080.wav +2025-06-01 19:15:36,363 - INFO - +====== Streaming Chunk 081 ====== +[2025-06-01 19:13:25] Speaker 5: And this is that footprint. +[2025-06-01 19:13:27] Speaker 5: We run a local Postgres for Profana storage. +[2025-06-01 19:13:30] Speaker 5: We also use Postgres for... + +2025-06-01 19:15:36,363 - INFO - Processing audio with duration 5.640 seconds +2025-06-01 19:15:36,539 - INFO - Processing audio with duration 00:05.640 +2025-06-01 19:15:36,842 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:36,843 - INFO - Got result for chunk: output_test5/chunks/chunk_081.wav +2025-06-01 19:15:36,843 - INFO - +====== Streaming Chunk 082 ====== +[2025-06-01 19:13:25] Speaker 5: kind of operating these things. +[2025-06-01 19:13:27] Speaker 5: This is our kind of MVP minus one approach, right? +[2025-06-01 19:13:32] Speaker 5: And once this... + +2025-06-01 19:15:36,843 - INFO - Processing audio with duration 3.280 seconds +2025-06-01 19:15:37,011 - INFO - Processing audio with duration 00:03.280 +2025-06-01 19:15:37,013 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:37,139 - INFO - Detected language 'en' with probability 0.99 +2025-06-01 19:15:37,162 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:37,162 - INFO - Got result for chunk: output_test5/chunks/chunk_082.wav +2025-06-01 19:15:37,162 - INFO - +====== Streaming Chunk 083 ====== +[2025-06-01 19:13:25] Speaker 5: becomes a service this will start looking differently any questions so far + +2025-06-01 19:15:37,162 - INFO - Processing audio with duration 3.980 seconds +2025-06-01 19:15:37,334 - INFO - Processing audio with duration 00:03.980 +2025-06-01 19:15:37,367 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:15:37,368 - INFO - Got result for chunk: output_test5/chunks/chunk_083.wav +2025-06-01 19:15:37,368 - INFO - +====== Streaming Chunk 084 ====== +[2025-06-01 19:13:25] Speaker 1: I have a few questions. +[2025-06-01 19:13:26] Speaker 1: Do you prefer we wait till the end? + +2025-06-01 19:15:37,368 - INFO - Processing audio with duration 4.040 seconds +2025-06-01 19:15:37,537 - INFO - Processing audio with duration 00:04.040 +2025-06-01 19:15:37,539 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:37,664 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:37,717 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:37,717 - INFO - Got result for chunk: output_test5/chunks/chunk_084.wav +2025-06-01 19:15:37,717 - INFO - +====== Streaming Chunk 085 ====== +[2025-06-01 19:13:25] Speaker 5: Yeah, let's take it, Zizal. +[2025-06-01 19:13:26] Speaker 5: Or if you want to see the demo and then ask those questions, it's up to you. + +2025-06-01 19:15:37,717 - INFO - Processing audio with duration 6.180 seconds +2025-06-01 19:15:37,890 - INFO - Processing audio with duration 00:06.180 +2025-06-01 19:15:37,935 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:15:37,935 - INFO - Got result for chunk: output_test5/chunks/chunk_085.wav +2025-06-01 19:15:37,935 - INFO - +====== Streaming Chunk 086 ====== +[2025-06-01 19:13:25] Speaker 1: I guess let's do the demo and then we can come back to questions. +[2025-06-01 19:13:28] Speaker 1: Maybe that will answer some of it. + +2025-06-01 19:15:37,936 - INFO - Processing audio with duration 7.100 seconds +2025-06-01 19:15:38,105 - INFO - Processing audio with duration 00:07.100 +2025-06-01 19:15:38,108 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:38,234 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:38,268 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:15:38,269 - INFO - Got result for chunk: output_test5/chunks/chunk_086.wav +2025-06-01 19:15:38,269 - INFO - +====== Streaming Chunk 087 ====== +[2025-06-01 19:13:25] Speaker 1: questions perfect yep all right so the the way we we know you can install this is we you will + +2025-06-01 19:15:38,269 - INFO - Processing audio with duration 5.000 seconds +2025-06-01 19:15:38,440 - INFO - Processing audio with duration 00:05.000 +2025-06-01 19:15:38,475 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:38,476 - INFO - Got result for chunk: output_test5/chunks/chunk_087.wav +2025-06-01 19:15:38,476 - INFO - +====== Streaming Chunk 088 ====== +[2025-06-01 19:13:25] Speaker 5: have a GitHub repo where it could go and say deploy this and it sets up a Kubernetes cluster. + +2025-06-01 19:15:38,476 - INFO - Processing audio with duration 6.900 seconds +2025-06-01 19:15:38,646 - INFO - Processing audio with duration 00:06.900 +2025-06-01 19:15:38,648 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:38,774 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:38,811 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:38,811 - INFO - Got result for chunk: output_test5/chunks/chunk_088.wav +2025-06-01 19:15:38,811 - INFO - +====== Streaming Chunk 089 ====== +[2025-06-01 19:13:25] Speaker 5: Kubernetes cluster setup and all of the Grafana Prometeas, +[2025-06-01 19:13:29] Speaker 5: everything is installed. + +2025-06-01 19:15:38,811 - INFO - Processing audio with duration 4.340 seconds +2025-06-01 19:15:38,983 - INFO - Processing audio with duration 00:04.340 +2025-06-01 19:15:39,273 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:39,273 - INFO - Got result for chunk: output_test5/chunks/chunk_089.wav +2025-06-01 19:15:39,274 - INFO - +====== Streaming Chunk 090 ====== +[2025-06-01 19:13:25] Speaker 5: in your primary OSID or your tenancy region, +[2025-06-01 19:13:29] Speaker 5: you will get access to a portal like this. + +2025-06-01 19:15:39,274 - INFO - Processing audio with duration 5.780 seconds +2025-06-01 19:15:39,442 - INFO - Processing audio with duration 00:05.780 +2025-06-01 19:15:39,444 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:39,571 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:39,603 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:39,603 - INFO - Got result for chunk: output_test5/chunks/chunk_090.wav +2025-06-01 19:15:39,603 - INFO - +====== Streaming Chunk 091 ====== +[2025-06-01 19:13:25] Speaker 5: right we call it corino lens but it is basically the lens so once you are here + +2025-06-01 19:15:39,603 - INFO - Processing audio with duration 4.520 seconds +2025-06-01 19:15:39,774 - INFO - Processing audio with duration 00:04.520 +2025-06-01 19:15:39,811 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:39,812 - INFO - Got result for chunk: output_test5/chunks/chunk_091.wav +2025-06-01 19:15:39,812 - INFO - +====== Streaming Chunk 092 ====== +[2025-06-01 19:13:25] Speaker 5: right it will go it has the ability to do a multi-region query in a single api + +2025-06-01 19:15:39,812 - INFO - Processing audio with duration 4.400 seconds +2025-06-01 19:15:39,984 - INFO - Processing audio with duration 00:04.400 +2025-06-01 19:15:39,987 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:40,113 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:40,148 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:40,148 - INFO - Got result for chunk: output_test5/chunks/chunk_092.wav +2025-06-01 19:15:40,148 - INFO - +====== Streaming Chunk 093 ====== +[2025-06-01 19:13:25] Speaker 5: So everything is API first, you have portal and there is a REST API endpoint you can interact. + +2025-06-01 19:15:40,149 - INFO - Processing audio with duration 6.440 seconds +2025-06-01 19:15:40,320 - INFO - Processing audio with duration 00:06.440 +2025-06-01 19:15:40,354 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:40,354 - INFO - Got result for chunk: output_test5/chunks/chunk_093.wav +2025-06-01 19:15:40,354 - INFO - +====== Streaming Chunk 094 ====== +[2025-06-01 19:13:25] Speaker 5: all of that is deployed in your tenancy so you can lock it down if you like to + +2025-06-01 19:15:40,355 - INFO - Processing audio with duration 4.120 seconds +2025-06-01 19:15:40,522 - INFO - Processing audio with duration 00:04.120 +2025-06-01 19:15:40,525 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:40,651 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:40,695 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:40,696 - INFO - Got result for chunk: output_test5/chunks/chunk_094.wav +2025-06-01 19:15:40,696 - INFO - +====== Streaming Chunk 095 ====== +[2025-06-01 19:13:25] Speaker 5: Right now we run on a public domain just for a demo, but everything in here. +[2025-06-01 19:13:30] Speaker 5: So if you see here. + +2025-06-01 19:15:40,696 - INFO - Processing audio with duration 5.400 seconds +2025-06-01 19:15:40,867 - INFO - Processing audio with duration 00:05.400 +2025-06-01 19:15:40,898 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:40,899 - INFO - Got result for chunk: output_test5/chunks/chunk_095.wav +2025-06-01 19:15:40,899 - INFO - Processing audio with duration 3.980 seconds +2025-06-01 19:15:40,899 - INFO - +====== Streaming Chunk 096 ====== +[2025-06-01 19:13:25] Speaker 5: This is done. If I refresh this page, it's going to come back with + +2025-06-01 19:15:41,068 - INFO - Processing audio with duration 00:03.980 +2025-06-01 19:15:41,070 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:41,196 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:41,229 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:41,229 - INFO - Got result for chunk: output_test5/chunks/chunk_096.wav +2025-06-01 19:15:41,230 - INFO - +====== Streaming Chunk 097 ====== +[2025-06-01 19:13:25] Speaker 5: It's doing all the regions that you have subscribed in your OSIT or Oracle tenancy. + +2025-06-01 19:15:41,230 - INFO - Processing audio with duration 6.460 seconds +2025-06-01 19:15:41,402 - INFO - Processing audio with duration 00:06.460 +2025-06-01 19:15:41,708 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:41,709 - INFO - Got result for chunk: output_test5/chunks/chunk_097.wav +2025-06-01 19:15:41,709 - INFO - +====== Streaming Chunk 098 ====== +[2025-06-01 19:13:25] Speaker 5: Then it comes back and tells you, here are all the instances that I can monitor. +[2025-06-01 19:13:28] Speaker 5: These are basically... + +2025-06-01 19:15:41,709 - INFO - Processing audio with duration 4.920 seconds +2025-06-01 19:15:41,879 - INFO - Processing audio with duration 00:04.920 +2025-06-01 19:15:41,882 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:42,009 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:42,039 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:42,040 - INFO - Got result for chunk: output_test5/chunks/chunk_098.wav +2025-06-01 19:15:42,040 - INFO - +====== Streaming Chunk 099 ====== +[2025-06-01 19:13:25] Speaker 5: just GPU instances that are running okay if you have new instances running you can also add + +2025-06-01 19:15:42,040 - INFO - Processing audio with duration 8.100 seconds +2025-06-01 19:15:42,212 - INFO - Processing audio with duration 00:08.100 +2025-06-01 19:15:42,266 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:42,266 - INFO - Got result for chunk: output_test5/chunks/chunk_099.wav +2025-06-01 19:15:42,267 - INFO - +====== Streaming Chunk 100 ====== +[2025-06-01 19:13:25] Speaker 5: what we call kind of plug-in. +[2025-06-01 19:13:27] Speaker 5: So it's a plug-in model. +[2025-06-01 19:13:28] Speaker 5: You already say, okay, start monitoring. + +2025-06-01 19:15:42,267 - INFO - Processing audio with duration 5.220 seconds +2025-06-01 19:15:42,435 - INFO - Processing audio with duration 00:05.220 +2025-06-01 19:15:42,437 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:42,564 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:42,602 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:42,602 - INFO - Got result for chunk: output_test5/chunks/chunk_100.wav +2025-06-01 19:15:42,603 - INFO - +====== Streaming Chunk 101 ====== +[2025-06-01 19:13:25] Speaker 5: these to OCI lengths. And so if you look at here, I have Frankfurt and Ashburn machines showing. + +2025-06-01 19:15:42,603 - INFO - Processing audio with duration 5.260 seconds +2025-06-01 19:15:42,774 - INFO - Processing audio with duration 00:05.260 +2025-06-01 19:15:42,812 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:42,812 - INFO - Got result for chunk: output_test5/chunks/chunk_101.wav +2025-06-01 19:15:42,813 - INFO - +====== Streaming Chunk 102 ====== +[2025-06-01 19:13:25] Speaker 5: up and it allows me to combine all of this and say they're all part of the same team or same + +2025-06-01 19:15:42,813 - INFO - Processing audio with duration 5.260 seconds +2025-06-01 19:15:42,982 - INFO - Processing audio with duration 00:05.260 +2025-06-01 19:15:42,985 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:43,112 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:43,142 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:43,143 - INFO - Got result for chunk: output_test5/chunks/chunk_102.wav +2025-06-01 19:15:43,143 - INFO - +====== Streaming Chunk 103 ====== +[2025-06-01 19:13:25] Speaker 5: experiment or same way i want to monitor and i can create what's called monitoring right + +2025-06-01 19:15:43,143 - INFO - Processing audio with duration 6.300 seconds +2025-06-01 19:15:43,314 - INFO - Processing audio with duration 00:06.300 +2025-06-01 19:15:43,357 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:43,357 - INFO - Got result for chunk: output_test5/chunks/chunk_103.wav +2025-06-01 19:15:43,357 - INFO - +====== Streaming Chunk 104 ====== +[2025-06-01 19:13:25] Speaker 5: So before I go there, right, how can you start monitoring this? +[2025-06-01 19:13:29] Speaker 5: If you have an existing instance... + +2025-06-01 19:15:43,357 - INFO - Processing audio with duration 5.860 seconds +2025-06-01 19:15:43,527 - INFO - Processing audio with duration 00:05.860 +2025-06-01 19:15:43,529 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:43,657 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:43,688 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:43,688 - INFO - Got result for chunk: output_test5/chunks/chunk_104.wav +2025-06-01 19:15:43,688 - INFO - +====== Streaming Chunk 105 ====== +[2025-06-01 19:13:25] Speaker 5: running or a new instance being provisioned you have to just include this script for now right + +2025-06-01 19:15:43,689 - INFO - Processing audio with duration 5.320 seconds +2025-06-01 19:15:43,858 - INFO - Processing audio with duration 00:05.320 +2025-06-01 19:15:44,138 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:44,138 - INFO - Got result for chunk: output_test5/chunks/chunk_105.wav +2025-06-01 19:15:44,138 - INFO - +====== Streaming Chunk 106 ====== +[2025-06-01 19:13:25] Speaker 5: we will we will change the way it is we're going to make it a native oci plugin + +2025-06-01 19:15:44,138 - INFO - Processing audio with duration 5.140 seconds +2025-06-01 19:15:44,311 - INFO - Processing audio with duration 00:05.140 +2025-06-01 19:15:44,313 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:44,441 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:44,483 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:44,483 - INFO - Got result for chunk: output_test5/chunks/chunk_106.wav +2025-06-01 19:15:44,483 - INFO - +====== Streaming Chunk 107 ====== +[2025-06-01 19:13:25] Speaker 5: once we upstream all of this to our plugin framework. +[2025-06-01 19:13:27] Speaker 5: But for now, it's basically a tarz file. + +2025-06-01 19:15:44,484 - INFO - Processing audio with duration 4.560 seconds +2025-06-01 19:15:44,652 - INFO - Processing audio with duration 00:04.560 +2025-06-01 19:15:44,692 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:44,692 - INFO - Got result for chunk: output_test5/chunks/chunk_107.wav +2025-06-01 19:15:44,692 - INFO - +====== Streaming Chunk 108 ====== +[2025-06-01 19:13:25] Speaker 5: and it's a Golang-based code primarily. +[2025-06-01 19:13:28] Speaker 5: The health checks are all Python-based. + +2025-06-01 19:15:44,693 - INFO - Processing audio with duration 4.940 seconds +2025-06-01 19:15:44,862 - INFO - Processing audio with duration 00:04.940 +2025-06-01 19:15:44,864 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:44,992 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:45,023 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:45,023 - INFO - Got result for chunk: output_test5/chunks/chunk_108.wav +2025-06-01 19:15:45,024 - INFO - +====== Streaming Chunk 109 ====== +[2025-06-01 19:13:25] Speaker 5: And this is basically end of the day, just all these metrics are composed and pushed. + +2025-06-01 19:15:45,024 - INFO - Processing audio with duration 3.500 seconds +2025-06-01 19:15:45,191 - INFO - Processing audio with duration 00:03.500 +2025-06-01 19:15:45,233 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:45,233 - INFO - Got result for chunk: output_test5/chunks/chunk_109.wav +2025-06-01 19:15:45,233 - INFO - +====== Streaming Chunk 110 ====== +[2025-06-01 19:13:25] Speaker 5: through Prometheus Push gateway, +[2025-06-01 19:13:27] Speaker 5: which is getting scraped by Push. +[2025-06-01 19:13:28] Speaker 5: So architecture. + +2025-06-01 19:15:45,234 - INFO - Processing audio with duration 6.020 seconds +2025-06-01 19:15:45,401 - INFO - Processing audio with duration 00:06.020 +2025-06-01 19:15:45,403 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:45,531 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:45,559 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:45,559 - INFO - Got result for chunk: output_test5/chunks/chunk_110.wav +2025-06-01 19:15:45,559 - INFO - +====== Streaming Chunk 111 ====== +[2025-06-01 19:13:25] Speaker 5: this is nothing very different or rocket science here right so it's very simple + +2025-06-01 19:15:45,559 - INFO - Processing audio with duration 4.660 seconds +2025-06-01 19:15:45,731 - INFO - Processing audio with duration 00:04.660 +2025-06-01 19:15:45,764 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:45,764 - INFO - Got result for chunk: output_test5/chunks/chunk_111.wav +2025-06-01 19:15:45,765 - INFO - +====== Streaming Chunk 112 ====== +[2025-06-01 19:13:25] Speaker 5: straightforward, but what it runs and how it runs is what the value we are adding. + +2025-06-01 19:15:45,765 - INFO - Processing audio with duration 5.620 seconds +2025-06-01 19:15:45,934 - INFO - Processing audio with duration 00:05.620 +2025-06-01 19:15:45,936 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:46,064 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:46,106 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:46,106 - INFO - Got result for chunk: output_test5/chunks/chunk_112.wav +2025-06-01 19:15:46,106 - INFO - +====== Streaming Chunk 113 ====== +[2025-06-01 19:13:25] Speaker 5: This is like a cloud in its script as well. +[2025-06-01 19:13:27] Speaker 5: You can add it as part of a new provisioning. + +2025-06-01 19:15:46,107 - INFO - Processing audio with duration 6.120 seconds +2025-06-01 19:15:46,277 - INFO - Processing audio with duration 00:06.120 +2025-06-01 19:15:46,570 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:46,570 - INFO - Got result for chunk: output_test5/chunks/chunk_113.wav +2025-06-01 19:15:46,570 - INFO - +====== Streaming Chunk 114 ====== +[2025-06-01 19:13:25] Speaker 5: instance automatically every new instance that comes up has monitoring enabled and everything is + +2025-06-01 19:15:46,570 - INFO - Processing audio with duration 7.100 seconds +2025-06-01 19:15:46,741 - INFO - Processing audio with duration 00:07.100 +2025-06-01 19:15:46,744 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:46,871 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:46,907 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:46,907 - INFO - Got result for chunk: output_test5/chunks/chunk_114.wav +2025-06-01 19:15:46,908 - INFO - +====== Streaming Chunk 115 ====== +[2025-06-01 19:13:25] Speaker 5: running at the host level as a system service it's not operating at the kubernetes layers or at the + +2025-06-01 19:15:46,908 - INFO - Processing audio with duration 7.080 seconds +2025-06-01 19:15:47,078 - INFO - Processing audio with duration 00:07.080 +2025-06-01 19:15:47,126 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:47,126 - INFO - Got result for chunk: output_test5/chunks/chunk_115.wav +2025-06-01 19:15:47,126 - INFO - +====== Streaming Chunk 116 ====== +[2025-06-01 19:13:25] Speaker 5: the Slurm application layers. +[2025-06-01 19:13:27] Speaker 5: It's running directly on the bare metal host. +[2025-06-01 19:13:30] Speaker 5: And we felt that was. + +2025-06-01 19:15:47,126 - INFO - Processing audio with duration 8.560 seconds +2025-06-01 19:15:47,297 - INFO - Processing audio with duration 00:08.560 +2025-06-01 19:15:47,300 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:47,427 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:47,456 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:47,457 - INFO - Got result for chunk: output_test5/chunks/chunk_116.wav +2025-06-01 19:15:47,457 - INFO - +====== Streaming Chunk 117 ====== +[2025-06-01 19:13:25] Speaker 5: the ideal way and the right place for us to remove specific implementations to Kubernetes. + +2025-06-01 19:15:47,457 - INFO - Processing audio with duration 7.060 seconds +2025-06-01 19:15:47,626 - INFO - Processing audio with duration 00:07.060 +2025-06-01 19:15:47,673 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:47,673 - INFO - Got result for chunk: output_test5/chunks/chunk_117.wav +2025-06-01 19:15:47,673 - INFO - +====== Streaming Chunk 118 ====== +[2025-06-01 19:13:25] Speaker 5: or Slurm or others. +[2025-06-01 19:13:28] Speaker 5: So we can also scan Kubernetes clusters you have running +[2025-06-01 19:13:31] Speaker 5: or you have RDMA. + +2025-06-01 19:15:47,674 - INFO - Processing audio with duration 5.460 seconds +2025-06-01 19:15:47,842 - INFO - Processing audio with duration 00:05.460 +2025-06-01 19:15:47,845 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:47,972 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:48,001 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:48,001 - INFO - Got result for chunk: output_test5/chunks/chunk_118.wav +2025-06-01 19:15:48,001 - INFO - +====== Streaming Chunk 119 ====== +[2025-06-01 19:13:25] Speaker 5: cluster networks or what we call compute clusters and allow you to also monitor that. + +2025-06-01 19:15:48,001 - INFO - Processing audio with duration 7.620 seconds +2025-06-01 19:15:48,172 - INFO - Processing audio with duration 00:07.620 +2025-06-01 19:15:48,210 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:48,211 - INFO - Got result for chunk: output_test5/chunks/chunk_119.wav +2025-06-01 19:15:48,211 - INFO - +====== Streaming Chunk 120 ====== +[2025-06-01 19:13:25] Speaker 5: And when you check the whole Kubernetes cluster, all the GPU nodes under it are automatically +[2025-06-01 19:13:30] Speaker 5: scanned. + +2025-06-01 19:15:48,211 - INFO - Processing audio with duration 7.800 seconds +2025-06-01 19:15:48,380 - INFO - Processing audio with duration 00:07.800 +2025-06-01 19:15:48,383 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:48,511 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:48,563 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 4}) +2025-06-01 19:15:48,564 - INFO - Got result for chunk: output_test5/chunks/chunk_120.wav +2025-06-01 19:15:48,564 - INFO - +====== Streaming Chunk 121 ====== +[2025-06-01 19:13:25] Speaker 5: and add it to the monitoring for us. +[2025-06-01 19:13:27] Speaker 5: That's the experience. +[2025-06-01 19:13:30] Speaker 5: Okay. +[2025-06-01 19:13:30] Speaker 5: So I'll quickly show you how... + +2025-06-01 19:15:48,564 - INFO - Processing audio with duration 5.240 seconds +2025-06-01 19:15:48,734 - INFO - Processing audio with duration 00:05.240 +2025-06-01 19:15:49,012 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:49,012 - INFO - Got result for chunk: output_test5/chunks/chunk_121.wav +2025-06-01 19:15:49,012 - INFO - +====== Streaming Chunk 122 ====== +[2025-06-01 19:13:25] Speaker 5: you know this this is basically a bare minimum um portal uh access we have everything is api + +2025-06-01 19:15:49,013 - INFO - Processing audio with duration 7.140 seconds +2025-06-01 19:15:49,184 - INFO - Processing audio with duration 00:07.140 +2025-06-01 19:15:49,187 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:49,314 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:49,345 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:49,346 - INFO - Got result for chunk: output_test5/chunks/chunk_122.wav +2025-06-01 19:15:49,346 - INFO - +====== Streaming Chunk 123 ====== +[2025-06-01 19:13:25] Speaker 5: first so you get exactly what you're doing here you could you could do that uh through + +2025-06-01 19:15:49,346 - INFO - Processing audio with duration 5.280 seconds +2025-06-01 19:15:49,516 - INFO - Processing audio with duration 00:05.280 +2025-06-01 19:15:49,557 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:49,557 - INFO - Got result for chunk: output_test5/chunks/chunk_123.wav +2025-06-01 19:15:49,557 - INFO - +====== Streaming Chunk 124 ====== +[2025-06-01 19:13:25] Speaker 5: REST API endpoints. So once this loads up, because it's doing like a close to 10 region query, + +2025-06-01 19:15:49,558 - INFO - Processing audio with duration 9.400 seconds +2025-06-01 19:15:49,729 - INFO - Processing audio with duration 00:09.400 +2025-06-01 19:15:49,731 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:49,858 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:49,891 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:49,891 - INFO - Got result for chunk: output_test5/chunks/chunk_124.wav +2025-06-01 19:15:49,891 - INFO - +====== Streaming Chunk 125 ====== +[2025-06-01 19:13:25] Speaker 5: takes roughly six seconds but six to ten seconds we will we'll make it better with some cash + +2025-06-01 19:15:49,891 - INFO - Processing audio with duration 5.960 seconds +2025-06-01 19:15:50,061 - INFO - Processing audio with duration 00:05.960 +2025-06-01 19:15:50,112 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:50,112 - INFO - Got result for chunk: output_test5/chunks/chunk_125.wav +2025-06-01 19:15:50,113 - INFO - +====== Streaming Chunk 126 ====== +[2025-06-01 19:13:25] Speaker 5: So, yeah, no, we have close to 30, 30 plus machines, right, Joleta? +[2025-06-01 19:13:33] Speaker 5: I think it's performing. + +2025-06-01 19:15:50,113 - INFO - Processing audio with duration 5.580 seconds +2025-06-01 19:15:50,279 - INFO - Processing audio with duration 00:05.580 +2025-06-01 19:15:50,282 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:50,409 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:50,443 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:50,443 - INFO - Got result for chunk: output_test5/chunks/chunk_126.wav +2025-06-01 19:15:50,443 - INFO - +====== Streaming Chunk 127 ====== +[2025-06-01 19:13:25] Speaker 5: okay but you know joletta is working hard to see how she can do parallel um parallelization + +2025-06-01 19:15:50,443 - INFO - Processing audio with duration 6.560 seconds +2025-06-01 19:15:50,613 - INFO - Processing audio with duration 00:06.560 +2025-06-01 19:15:50,658 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:50,659 - INFO - Got result for chunk: output_test5/chunks/chunk_127.wav +2025-06-01 19:15:50,659 - INFO - +====== Streaming Chunk 128 ====== +[2025-06-01 19:13:25] Speaker 5: of some of these runs and get back. +[2025-06-01 19:13:27] Speaker 5: But we got them better. +[2025-06-01 19:13:29] Speaker 5: We started from 30 seconds. + +2025-06-01 19:15:50,659 - INFO - Processing audio with duration 5.020 seconds +2025-06-01 19:15:50,830 - INFO - Processing audio with duration 00:05.020 +2025-06-01 19:15:50,833 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:50,960 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:51,002 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:51,002 - INFO - Got result for chunk: output_test5/chunks/chunk_128.wav +2025-06-01 19:15:51,002 - INFO - +====== Streaming Chunk 129 ====== +[2025-06-01 19:13:25] Speaker 5: by now at six seconds, but we'll keep pushing. +[2025-06-01 19:13:29] Speaker 5: All right. So it allows me to create a... + +2025-06-01 19:15:51,002 - INFO - Processing audio with duration 4.540 seconds +2025-06-01 19:15:51,172 - INFO - Processing audio with duration 00:04.540 +2025-06-01 19:15:51,462 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:51,462 - INFO - Got result for chunk: output_test5/chunks/chunk_129.wav +2025-06-01 19:15:51,462 - INFO - +====== Streaming Chunk 130 ====== +[2025-06-01 19:13:25] Speaker 5: monitoring ring, it's like just an arbitrary way for you to +[2025-06-01 19:13:28] Speaker 5: combine these resources and say + +2025-06-01 19:15:51,462 - INFO - Processing audio with duration 5.420 seconds +2025-06-01 19:15:51,631 - INFO - Processing audio with duration 00:05.420 +2025-06-01 19:15:51,633 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:51,760 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:51,803 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:15:51,803 - INFO - Got result for chunk: output_test5/chunks/chunk_130.wav +2025-06-01 19:15:51,803 - INFO - +====== Streaming Chunk 131 ====== +[2025-06-01 19:13:25] Speaker 5: I want a dedicated dashboard for this, +[2025-06-01 19:13:27] Speaker 5: these set of machines, right? +[2025-06-01 19:13:29] Speaker 5: That's what you. + +2025-06-01 19:15:51,803 - INFO - Processing audio with duration 6.120 seconds +2025-06-01 19:15:51,975 - INFO - Processing audio with duration 00:06.120 +2025-06-01 19:15:52,018 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:52,018 - INFO - Got result for chunk: output_test5/chunks/chunk_131.wav +2025-06-01 19:15:52,019 - INFO - +====== Streaming Chunk 132 ====== +[2025-06-01 19:13:25] Speaker 5: allows you to create a ring and if you go to monitoring rings here and you have OK, +[2025-06-01 19:13:30] Speaker 5: this is. + +2025-06-01 19:15:52,019 - INFO - Processing audio with duration 6.340 seconds +2025-06-01 19:15:52,188 - INFO - Processing audio with duration 00:06.340 +2025-06-01 19:15:52,191 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:52,318 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:52,352 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:52,352 - INFO - Got result for chunk: output_test5/chunks/chunk_132.wav +2025-06-01 19:15:52,352 - INFO - +====== Streaming Chunk 133 ====== +[2025-06-01 19:13:25] Speaker 5: for ML training, this is for the team, this is for cost center, whatever, however you + +2025-06-01 19:15:52,353 - INFO - Processing audio with duration 5.140 seconds +2025-06-01 19:15:52,524 - INFO - Processing audio with duration 00:05.140 +2025-06-01 19:15:52,558 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:52,559 - INFO - Got result for chunk: output_test5/chunks/chunk_133.wav +2025-06-01 19:15:52,559 - INFO - +====== Streaming Chunk 134 ====== +[2025-06-01 19:13:25] Speaker 5: want to use this for kind of bundling all the things. Right? And the thing about + +2025-06-01 19:15:52,559 - INFO - Processing audio with duration 7.600 seconds +2025-06-01 19:15:52,729 - INFO - Processing audio with duration 00:07.600 +2025-06-01 19:15:52,731 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:52,859 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:52,890 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:52,890 - INFO - Got result for chunk: output_test5/chunks/chunk_134.wav +2025-06-01 19:15:52,890 - INFO - +====== Streaming Chunk 135 ====== +[2025-06-01 19:13:25] Speaker 5: this is you can always come back and add more instances or remove the instances if they go + +2025-06-01 19:15:52,890 - INFO - Processing audio with duration 5.640 seconds +2025-06-01 19:15:53,064 - INFO - Processing audio with duration 00:05.640 +2025-06-01 19:15:53,096 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:53,097 - INFO - Got result for chunk: output_test5/chunks/chunk_135.wav +2025-06-01 19:15:53,097 - INFO - +====== Streaming Chunk 136 ====== +[2025-06-01 19:13:25] Speaker 5: offline or you have to turn back in whatever right so so every every ring you create monitor + +2025-06-01 19:15:53,097 - INFO - Processing audio with duration 8.700 seconds +2025-06-01 19:15:53,267 - INFO - Processing audio with duration 00:08.700 +2025-06-01 19:15:53,270 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:53,396 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:53,428 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:53,428 - INFO - Got result for chunk: output_test5/chunks/chunk_136.wav +2025-06-01 19:15:53,428 - INFO - +====== Streaming Chunk 137 ====== +[2025-06-01 19:13:25] Speaker 5: ring you create comes with a dedicated Grafana board which includes all the hosts that are + +2025-06-01 19:15:53,428 - INFO - Processing audio with duration 6.540 seconds +2025-06-01 19:15:53,598 - INFO - Processing audio with duration 00:06.540 +2025-06-01 19:15:53,910 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:53,911 - INFO - Got result for chunk: output_test5/chunks/chunk_137.wav +2025-06-01 19:15:53,911 - INFO - +====== Streaming Chunk 138 ====== +[2025-06-01 19:13:25] Speaker 5: part of this and I'm going to click you through the Grafana. +[2025-06-01 19:13:31] Speaker 5: So what you see here at the first part of + +2025-06-01 19:15:53,911 - INFO - Processing audio with duration 5.780 seconds +2025-06-01 19:15:54,080 - INFO - Processing audio with duration 00:05.780 +2025-06-01 19:15:54,083 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:54,210 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:54,250 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:54,250 - INFO - Got result for chunk: output_test5/chunks/chunk_138.wav +2025-06-01 19:15:54,250 - INFO - +====== Streaming Chunk 139 ====== +[2025-06-01 19:13:25] Speaker 5: this is the health summary of all the compute nodes that are part of this, right? So we added more nodes. + +2025-06-01 19:15:54,250 - INFO - Processing audio with duration 6.460 seconds +2025-06-01 19:15:54,419 - INFO - Processing audio with duration 00:06.460 +2025-06-01 19:15:54,469 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:54,469 - INFO - Got result for chunk: output_test5/chunks/chunk_139.wav +2025-06-01 19:15:54,469 - INFO - +====== Streaming Chunk 140 ====== +[2025-06-01 19:13:25] Speaker 5: which you're in this part just to show you demo, right? +[2025-06-01 19:13:27] Speaker 5: So if you see the first board here is this is the. + +2025-06-01 19:15:54,470 - INFO - Processing audio with duration 6.380 seconds +2025-06-01 19:15:54,638 - INFO - Processing audio with duration 00:06.380 +2025-06-01 19:15:54,640 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:54,767 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:54,798 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:54,799 - INFO - Got result for chunk: output_test5/chunks/chunk_140.wav +2025-06-01 19:15:54,799 - INFO - +====== Streaming Chunk 141 ====== +[2025-06-01 19:13:25] Speaker 5: actual health check of all the performance related checks that we did on the host when you activate + +2025-06-01 19:15:54,799 - INFO - Processing audio with duration 5.280 seconds +2025-06-01 19:15:54,970 - INFO - Processing audio with duration 00:05.280 +2025-06-01 19:15:55,005 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:55,006 - INFO - Got result for chunk: output_test5/chunks/chunk_141.wav +2025-06-01 19:15:55,006 - INFO - +====== Streaming Chunk 142 ====== +[2025-06-01 19:13:25] Speaker 5: the plugin we will also allow you to run these on demand so if you look at the categories here + +2025-06-01 19:15:55,006 - INFO - Processing audio with duration 6.980 seconds +2025-06-01 19:15:55,176 - INFO - Processing audio with duration 00:06.980 +2025-06-01 19:15:55,178 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:55,305 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:55,345 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:55,345 - INFO - Got result for chunk: output_test5/chunks/chunk_142.wav +2025-06-01 19:15:55,346 - INFO - +====== Streaming Chunk 143 ====== +[2025-06-01 19:13:25] Speaker 5: compute throughput, temperature check, all these check, check, check, +[2025-06-01 19:13:28] Speaker 5: model MFU based stuff. + +2025-06-01 19:15:55,346 - INFO - Processing audio with duration 5.720 seconds +2025-06-01 19:15:55,514 - INFO - Processing audio with duration 00:05.720 +2025-06-01 19:15:55,550 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:55,550 - INFO - Got result for chunk: output_test5/chunks/chunk_143.wav +2025-06-01 19:15:55,551 - INFO - +====== Streaming Chunk 144 ====== +[2025-06-01 19:13:25] Speaker 5: All of those are readily available for you and we will very soon have a link that will exactly take + +2025-06-01 19:15:55,551 - INFO - Processing audio with duration 5.280 seconds +2025-06-01 19:15:55,721 - INFO - Processing audio with duration 00:05.280 +2025-06-01 19:15:55,724 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:55,852 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:55,891 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:55,892 - INFO - Got result for chunk: output_test5/chunks/chunk_144.wav +2025-06-01 19:15:55,892 - INFO - +====== Streaming Chunk 145 ====== +[2025-06-01 19:13:25] Speaker 5: you to a JSON that is very much deeper on all the tests. This is basically the JSON that the health check. + +2025-06-01 19:15:55,892 - INFO - Processing audio with duration 6.320 seconds +2025-06-01 19:15:56,063 - INFO - Processing audio with duration 00:06.320 +2025-06-01 19:15:56,334 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:56,334 - INFO - Got result for chunk: output_test5/chunks/chunk_145.wav +2025-06-01 19:15:56,334 - INFO - +====== Streaming Chunk 146 ====== +[2025-06-01 19:13:25] Speaker 5: script create like what's the bandwidth achieved the tflops achieved for each of the node which + +2025-06-01 19:15:56,335 - INFO - Processing audio with duration 7.320 seconds +2025-06-01 19:15:56,507 - INFO - Processing audio with duration 00:07.320 +2025-06-01 19:15:56,509 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:56,637 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:56,675 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:56,675 - INFO - Got result for chunk: output_test5/chunks/chunk_146.wav +2025-06-01 19:15:56,676 - INFO - +====== Streaming Chunk 147 ====== +[2025-06-01 19:13:25] Speaker 5: GPU ran this for example right this is the GPU ID of an H100 we ran this is actually the JSON document + +2025-06-01 19:15:56,676 - INFO - Processing audio with duration 4.160 seconds +2025-06-01 19:15:56,845 - INFO - Processing audio with duration 00:04.160 +2025-06-01 19:15:56,882 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:56,883 - INFO - Got result for chunk: output_test5/chunks/chunk_147.wav +2025-06-01 19:15:56,883 - INFO - Processing audio with duration 3.740 seconds +2025-06-01 19:15:56,883 - INFO - +====== Streaming Chunk 148 ====== +[2025-06-01 19:13:25] Speaker 5: you'll have access to this as well as the logs piece uh but in a grafana it looks + +2025-06-01 19:15:57,054 - INFO - Processing audio with duration 00:03.740 +2025-06-01 19:15:57,057 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:57,183 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:57,223 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:57,223 - INFO - Got result for chunk: output_test5/chunks/chunk_148.wav +2025-06-01 19:15:57,223 - INFO - +====== Streaming Chunk 149 ====== +[2025-06-01 19:13:25] Speaker 5: Now, it's an easier way for you to digest all of this. +[2025-06-01 19:13:27] Speaker 5: Let me put it up. + +2025-06-01 19:15:57,224 - INFO - Processing audio with duration 9.420 seconds +2025-06-01 19:15:57,393 - INFO - Processing audio with duration 00:09.420 +2025-06-01 19:15:57,427 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:57,428 - INFO - Got result for chunk: output_test5/chunks/chunk_149.wav +2025-06-01 19:15:57,428 - INFO - +====== Streaming Chunk 150 ====== +[2025-06-01 19:13:25] Speaker 5: end of the day it's the same data but everything is a very detailed data that you may + +2025-06-01 19:15:57,428 - INFO - Processing audio with duration 6.540 seconds +2025-06-01 19:15:57,598 - INFO - Processing audio with duration 00:06.540 +2025-06-01 19:15:57,600 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:57,728 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:57,763 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:57,763 - INFO - Got result for chunk: output_test5/chunks/chunk_150.wav +2025-06-01 19:15:57,763 - INFO - +====== Streaming Chunk 151 ====== +[2025-06-01 19:13:25] Speaker 5: be looking for um than just a summary like this okay both are available okay so we will add more + +2025-06-01 19:15:57,763 - INFO - Processing audio with duration 5.940 seconds +2025-06-01 19:15:57,934 - INFO - Processing audio with duration 00:05.940 +2025-06-01 19:15:57,974 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:57,974 - INFO - Got result for chunk: output_test5/chunks/chunk_151.wav +2025-06-01 19:15:57,974 - INFO - +====== Streaming Chunk 152 ====== +[2025-06-01 19:13:25] Speaker 5: filters when it allows you to you know go by region and check the gpu types if you have mi300x + +2025-06-01 19:15:57,974 - INFO - Processing audio with duration 8.580 seconds +2025-06-01 19:15:58,144 - INFO - Processing audio with duration 00:08.580 +2025-06-01 19:15:58,146 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:58,272 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:58,303 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:58,304 - INFO - Got result for chunk: output_test5/chunks/chunk_152.wav +2025-06-01 19:15:58,304 - INFO - +====== Streaming Chunk 153 ====== +[2025-06-01 19:13:25] Speaker 5: So Osaka does not have, maybe I think other machines in Chicago have MI300X. + +2025-06-01 19:15:58,304 - INFO - Processing audio with duration 6.000 seconds +2025-06-01 19:15:58,477 - INFO - Processing audio with duration 00:06.000 +2025-06-01 19:15:58,774 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:58,774 - INFO - Got result for chunk: output_test5/chunks/chunk_153.wav +2025-06-01 19:15:58,774 - INFO - +====== Streaming Chunk 154 ====== +[2025-06-01 19:13:25] Speaker 5: see here yes you can see my 300x instance um so yeah these are all of both uh my 300x + +2025-06-01 19:15:58,774 - INFO - Processing audio with duration 6.500 seconds +2025-06-01 19:15:58,946 - INFO - Processing audio with duration 00:06.500 +2025-06-01 19:15:58,948 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:59,076 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:59,116 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:15:59,116 - INFO - Got result for chunk: output_test5/chunks/chunk_154.wav +2025-06-01 19:15:59,116 - INFO - +====== Streaming Chunk 155 ====== +[2025-06-01 19:13:25] Speaker 5: as well as GPU from NVidia are fully monitored. +[2025-06-01 19:13:29] Speaker 5: And we have health checks running for both. + +2025-06-01 19:15:59,117 - INFO - Processing audio with duration 5.900 seconds +2025-06-01 19:15:59,288 - INFO - Processing audio with duration 00:05.900 +2025-06-01 19:15:59,319 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:59,319 - INFO - Got result for chunk: output_test5/chunks/chunk_155.wav +2025-06-01 19:15:59,319 - INFO - +====== Streaming Chunk 156 ====== +[2025-06-01 19:13:25] Speaker 5: Okay, so all of this is the data that you may be interested in. + +2025-06-01 19:15:59,319 - INFO - Processing audio with duration 8.040 seconds +2025-06-01 19:15:59,488 - INFO - Processing audio with duration 00:08.040 +2025-06-01 19:15:59,490 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:59,617 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:15:59,652 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:59,653 - INFO - Got result for chunk: output_test5/chunks/chunk_156.wav +2025-06-01 19:15:59,653 - INFO - +====== Streaming Chunk 157 ====== +[2025-06-01 19:13:25] Speaker 5: that allows you to kind of baseline and benchmark all of this. Okay. So all of these metrics are + +2025-06-01 19:15:59,653 - INFO - Processing audio with duration 7.580 seconds +2025-06-01 19:15:59,823 - INFO - Processing audio with duration 00:07.580 +2025-06-01 19:15:59,858 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:15:59,859 - INFO - Got result for chunk: output_test5/chunks/chunk_157.wav +2025-06-01 19:15:59,859 - INFO - +====== Streaming Chunk 158 ====== +[2025-06-01 19:13:25] Speaker 5: natively available in Prometheus and if I have to show you so we push with OCI + +2025-06-01 19:15:59,859 - INFO - Processing audio with duration 5.620 seconds +2025-06-01 19:16:00,029 - INFO - Processing audio with duration 00:05.620 +2025-06-01 19:16:00,032 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:00,160 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:00,190 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:00,190 - INFO - Got result for chunk: output_test5/chunks/chunk_158.wav +2025-06-01 19:16:00,190 - INFO - +====== Streaming Chunk 159 ====== +[2025-06-01 19:13:25] Speaker 5: lens labels to this and this will allow you to extend it any further you like. + +2025-06-01 19:16:00,191 - INFO - Processing audio with duration 7.160 seconds +2025-06-01 19:16:00,363 - INFO - Processing audio with duration 00:07.160 +2025-06-01 19:16:00,404 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:00,404 - INFO - Got result for chunk: output_test5/chunks/chunk_159.wav +2025-06-01 19:16:00,405 - INFO - +====== Streaming Chunk 160 ====== +[2025-06-01 19:13:25] Speaker 5: custom boards if you like, or you want to merge left join, right join some datasets here to get to more. + +2025-06-01 19:16:00,405 - INFO - Processing audio with duration 8.120 seconds +2025-06-01 19:16:00,577 - INFO - Processing audio with duration 00:08.120 +2025-06-01 19:16:00,580 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:00,707 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:00,741 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:00,742 - INFO - Got result for chunk: output_test5/chunks/chunk_160.wav +2025-06-01 19:16:00,742 - INFO - +====== Streaming Chunk 161 ====== +[2025-06-01 19:13:25] Speaker 5: details everything is fully available that's just native krafanov right this is the lcl lens + +2025-06-01 19:16:00,742 - INFO - Processing audio with duration 4.660 seconds +2025-06-01 19:16:00,912 - INFO - Processing audio with duration 00:04.660 +2025-06-01 19:16:01,232 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:01,232 - INFO - Got result for chunk: output_test5/chunks/chunk_161.wav +2025-06-01 19:16:01,232 - INFO - +====== Streaming Chunk 162 ====== +[2025-06-01 19:13:25] Speaker 5: health check stuff too. There is quite a bit of +[2025-06-01 19:13:31] Speaker 5: health. Yeah, this is the health summary. + +2025-06-01 19:16:01,232 - INFO - Processing audio with duration 6.420 seconds +2025-06-01 19:16:01,400 - INFO - Processing audio with duration 00:06.420 +2025-06-01 19:16:01,403 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:01,530 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:01,571 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:01,571 - INFO - Got result for chunk: output_test5/chunks/chunk_162.wav +2025-06-01 19:16:01,572 - INFO - +====== Streaming Chunk 163 ====== +[2025-06-01 19:13:25] Speaker 5: So all of these metrics, the only thing that we are adding, +[2025-06-01 19:13:27] Speaker 5: this comes out as OCI lens. + +2025-06-01 19:16:01,572 - INFO - Processing audio with duration 7.660 seconds +2025-06-01 19:16:01,744 - INFO - Processing audio with duration 00:07.660 +2025-06-01 19:16:01,795 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:16:01,795 - INFO - Got result for chunk: output_test5/chunks/chunk_163.wav +2025-06-01 19:16:01,795 - INFO - +====== Streaming Chunk 164 ====== +[2025-06-01 19:13:25] Speaker 5: anything that comes directly from the vendor, +[2025-06-01 19:13:26] Speaker 5: like AMD or NVIDIA come as is, right? +[2025-06-01 19:13:29] Speaker 5: DCGM append it. + +2025-06-01 19:16:01,796 - INFO - Processing audio with duration 7.000 seconds +2025-06-01 19:16:01,968 - INFO - Processing audio with duration 00:07.000 +2025-06-01 19:16:01,970 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:02,097 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:02,129 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:02,129 - INFO - Got result for chunk: output_test5/chunks/chunk_164.wav +2025-06-01 19:16:02,129 - INFO - +====== Streaming Chunk 165 ====== +[2025-06-01 19:13:25] Speaker 5: So one additional thing we also did is if you know, let me go back here. + +2025-06-01 19:16:02,129 - INFO - Processing audio with duration 8.680 seconds +2025-06-01 19:16:02,302 - INFO - Processing audio with duration 00:08.680 +2025-06-01 19:16:02,342 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:02,342 - INFO - Got result for chunk: output_test5/chunks/chunk_165.wav +2025-06-01 19:16:02,343 - INFO - +====== Streaming Chunk 166 ====== +[2025-06-01 19:13:25] Speaker 5: So this is the overall health board. +[2025-06-01 19:13:28] Speaker 5: And there's another board that we bring is this is. + +2025-06-01 19:16:02,343 - INFO - Processing audio with duration 5.180 seconds +2025-06-01 19:16:02,513 - INFO - Processing audio with duration 00:05.180 +2025-06-01 19:16:02,516 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:02,643 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:02,681 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:02,681 - INFO - Got result for chunk: output_test5/chunks/chunk_166.wav +2025-06-01 19:16:02,681 - INFO - +====== Streaming Chunk 167 ====== +[2025-06-01 19:13:25] Speaker 5: per host data right this is the actual gpu rdma metrics uh you know you know node health and all + +2025-06-01 19:16:02,682 - INFO - Processing audio with duration 5.480 seconds +2025-06-01 19:16:02,851 - INFO - Processing audio with duration 00:05.480 +2025-06-01 19:16:02,888 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:02,889 - INFO - Got result for chunk: output_test5/chunks/chunk_167.wav +2025-06-01 19:16:02,889 - INFO - +====== Streaming Chunk 168 ====== +[2025-06-01 19:13:25] Speaker 5: that. Additional things that we are trying to add is if you come here and look at this table. + +2025-06-01 19:16:02,889 - INFO - Processing audio with duration 6.200 seconds +2025-06-01 19:16:03,064 - INFO - Processing audio with duration 00:06.200 +2025-06-01 19:16:03,066 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:03,193 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:03,229 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:03,229 - INFO - Got result for chunk: output_test5/chunks/chunk_168.wav +2025-06-01 19:16:03,229 - INFO - +====== Streaming Chunk 169 ====== +[2025-06-01 19:13:25] Speaker 5: We will improve the UI a little bit, +[2025-06-01 19:13:27] Speaker 5: but there is something called host metadata survey. + +2025-06-01 19:16:03,230 - INFO - Processing audio with duration 6.800 seconds +2025-06-01 19:16:03,403 - INFO - Processing audio with duration 00:06.800 +2025-06-01 19:16:03,687 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:03,688 - INFO - Got result for chunk: output_test5/chunks/chunk_169.wav +2025-06-01 19:16:03,688 - INFO - +====== Streaming Chunk 170 ====== +[2025-06-01 19:13:25] Speaker 5: It's basically a local service endpoint that we hit +[2025-06-01 19:13:28] Speaker 5: and we fetch the serial number because if the- + +2025-06-01 19:16:03,688 - INFO - Processing audio with duration 5.760 seconds +2025-06-01 19:16:03,858 - INFO - Processing audio with duration 00:05.760 +2025-06-01 19:16:03,861 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:03,987 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:04,032 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:04,033 - INFO - Got result for chunk: output_test5/chunks/chunk_170.wav +2025-06-01 19:16:04,033 - INFO - Processing audio with duration 4.800 seconds +2025-06-01 19:16:04,033 - INFO - +====== Streaming Chunk 171 ====== +[2025-06-01 19:13:25] Speaker 5: host is underperforming and you have to turn back it in for repair. +[2025-06-01 19:13:29] Speaker 5: You know, some data that we have. + +2025-06-01 19:16:04,205 - INFO - Processing audio with duration 00:04.800 +2025-06-01 19:16:04,248 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:04,248 - INFO - Got result for chunk: output_test5/chunks/chunk_171.wav +2025-06-01 19:16:04,248 - INFO - +====== Streaming Chunk 172 ====== +[2025-06-01 19:13:25] Speaker 5: going to publish here is going to be super helpful. +[2025-06-01 19:13:27] Speaker 5: And we are looking at how to put an agent. + +2025-06-01 19:16:04,248 - INFO - Processing audio with duration 6.540 seconds +2025-06-01 19:16:04,416 - INFO - Processing audio with duration 00:06.540 +2025-06-01 19:16:04,418 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:04,545 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:04,591 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:16:04,591 - INFO - Got result for chunk: output_test5/chunks/chunk_172.wav +2025-06-01 19:16:04,591 - INFO - +====== Streaming Chunk 173 ====== +[2025-06-01 19:13:25] Speaker 5: AI on top of this, right? Let's +[2025-06-01 19:13:26] Speaker 5: shortcut that, right? So we +[2025-06-01 19:13:28] Speaker 5: will instead of throwing more + +2025-06-01 19:16:04,591 - INFO - Processing audio with duration 7.320 seconds +2025-06-01 19:16:04,763 - INFO - Processing audio with duration 00:07.320 +2025-06-01 19:16:04,804 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:04,804 - INFO - Got result for chunk: output_test5/chunks/chunk_173.wav +2025-06-01 19:16:04,804 - INFO - +====== Streaming Chunk 174 ====== +[2025-06-01 19:13:25] Speaker 5: more boards, we will kind of curate an experience where if you see a underperforming node or a bad, + +2025-06-01 19:16:04,805 - INFO - Processing audio with duration 7.200 seconds +2025-06-01 19:16:04,974 - INFO - Processing audio with duration 00:07.200 +2025-06-01 19:16:04,976 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:05,103 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:05,138 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:05,138 - INFO - Got result for chunk: output_test5/chunks/chunk_174.wav +2025-06-01 19:16:05,138 - INFO - +====== Streaming Chunk 175 ====== +[2025-06-01 19:13:25] Speaker 5: node. We may automate some of that to see if we could automatically raise a ticket or turn the + +2025-06-01 19:16:05,138 - INFO - Processing audio with duration 5.380 seconds +2025-06-01 19:16:05,309 - INFO - Processing audio with duration 00:05.380 +2025-06-01 19:16:05,349 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:05,349 - INFO - Got result for chunk: output_test5/chunks/chunk_175.wav +2025-06-01 19:16:05,350 - INFO - +====== Streaming Chunk 176 ====== +[2025-06-01 19:13:25] Speaker 5: machine in or invoke folks in the support team to help you guys out. +[2025-06-01 19:13:30] Speaker 5: So that's the. + +2025-06-01 19:16:05,350 - INFO - Processing audio with duration 4.620 seconds +2025-06-01 19:16:05,522 - INFO - Processing audio with duration 00:04.620 +2025-06-01 19:16:05,524 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:05,650 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:05,684 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:05,684 - INFO - Got result for chunk: output_test5/chunks/chunk_176.wav +2025-06-01 19:16:05,684 - INFO - +====== Streaming Chunk 177 ====== +[2025-06-01 19:13:25] Speaker 5: future roadmap. So yeah, everything here, we're going to add more stuff like what's the + +2025-06-01 19:16:05,684 - INFO - Processing audio with duration 4.940 seconds +2025-06-01 19:16:05,857 - INFO - Processing audio with duration 00:04.940 +2025-06-01 19:16:06,117 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:06,117 - INFO - Got result for chunk: output_test5/chunks/chunk_177.wav +2025-06-01 19:16:06,117 - INFO - +====== Streaming Chunk 178 ====== +[2025-06-01 19:13:25] Speaker 5: the huda driver you're running what's the kernel versions you're running um what's the OS + +2025-06-01 19:16:06,118 - INFO - Processing audio with duration 4.420 seconds +2025-06-01 19:16:06,287 - INFO - Processing audio with duration 00:04.420 +2025-06-01 19:16:06,290 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:06,416 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:06,451 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:06,451 - INFO - Got result for chunk: output_test5/chunks/chunk_178.wav +2025-06-01 19:16:06,451 - INFO - +====== Streaming Chunk 179 ====== +[2025-06-01 19:13:25] Speaker 5: attached to, what's the Rockham driver version you're running, all of those data will be fully + +2025-06-01 19:16:06,451 - INFO - Processing audio with duration 5.520 seconds +2025-06-01 19:16:06,622 - INFO - Processing audio with duration 00:05.520 +2025-06-01 19:16:06,650 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:06,650 - INFO - Got result for chunk: output_test5/chunks/chunk_179.wav +2025-06-01 19:16:06,650 - INFO - +====== Streaming Chunk 180 ====== +[2025-06-01 19:13:25] Speaker 5: available everything related to metadata will be published here and you can continue to + +2025-06-01 19:16:06,650 - INFO - Processing audio with duration 5.520 seconds +2025-06-01 19:16:06,823 - INFO - Processing audio with duration 00:05.520 +2025-06-01 19:16:06,826 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:06,954 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:06,992 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:06,992 - INFO - Got result for chunk: output_test5/chunks/chunk_180.wav +2025-06-01 19:16:06,992 - INFO - +====== Streaming Chunk 181 ====== +[2025-06-01 19:13:25] Speaker 5: use that instead of you SSHing into the machine, +[2025-06-01 19:13:28] Speaker 5: running some commands and all of that. + +2025-06-01 19:16:06,993 - INFO - Processing audio with duration 7.920 seconds +2025-06-01 19:16:07,176 - INFO - Processing audio with duration 00:07.920 +2025-06-01 19:16:07,214 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:07,214 - INFO - Got result for chunk: output_test5/chunks/chunk_181.wav +2025-06-01 19:16:07,215 - INFO - +====== Streaming Chunk 182 ====== +[2025-06-01 19:13:25] Speaker 5: So, readily available. +[2025-06-01 19:13:27] Speaker 5: Everything else you see here is something you may have been using already. + +2025-06-01 19:16:07,215 - INFO - Processing audio with duration 6.000 seconds +2025-06-01 19:16:07,387 - INFO - Processing audio with duration 00:06.000 +2025-06-01 19:16:07,389 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:07,517 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:07,568 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 4}) +2025-06-01 19:16:07,568 - INFO - Got result for chunk: output_test5/chunks/chunk_182.wav +2025-06-01 19:16:07,568 - INFO - +====== Streaming Chunk 183 ====== +[2025-06-01 19:13:25] Speaker 5: which is power, usage, temperature, utilization, +[2025-06-01 19:13:28] Speaker 5: all that stuff. +[2025-06-01 19:13:30] Speaker 5: Okay. +[2025-06-01 19:13:31] Speaker 5: So in an actual, this- + +2025-06-01 19:16:07,569 - INFO - Processing audio with duration 6.880 seconds +2025-06-01 19:16:07,739 - INFO - Processing audio with duration 00:06.880 +2025-06-01 19:16:07,776 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:07,776 - INFO - Got result for chunk: output_test5/chunks/chunk_183.wav +2025-06-01 19:16:07,776 - INFO - +====== Streaming Chunk 184 ====== +[2025-06-01 19:13:25] Speaker 5: this is this is what it is right so at least our first iteration of um can we get a monitor + +2025-06-01 19:16:07,776 - INFO - Processing audio with duration 6.800 seconds +2025-06-01 19:16:07,950 - INFO - Processing audio with duration 00:06.800 +2025-06-01 19:16:07,952 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:08,078 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:08,108 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:08,109 - INFO - Got result for chunk: output_test5/chunks/chunk_184.wav +2025-06-01 19:16:08,109 - INFO - +====== Streaming Chunk 185 ====== +[2025-06-01 19:13:25] Speaker 5: going which is native and health checks going which is what lci published health checks and + +2025-06-01 19:16:08,109 - INFO - Processing audio with duration 10.180 seconds +2025-06-01 19:16:08,279 - INFO - Processing audio with duration 00:10.180 +2025-06-01 19:16:08,550 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:08,550 - INFO - Got result for chunk: output_test5/chunks/chunk_185.wav +2025-06-01 19:16:08,551 - INFO - +====== Streaming Chunk 186 ====== +[2025-06-01 19:13:25] Speaker 5: just always vendor specified. So I'm gonna stop here and see if you folks have questions. + +2025-06-01 19:16:08,551 - INFO - Processing audio with duration 7.760 seconds +2025-06-01 19:16:08,723 - INFO - Processing audio with duration 00:07.760 +2025-06-01 19:16:08,725 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:08,852 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:08,898 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2, 'Speaker 5': 1}) +2025-06-01 19:16:08,898 - INFO - Got result for chunk: output_test5/chunks/chunk_186.wav +2025-06-01 19:16:08,898 - INFO - +====== Streaming Chunk 187 ====== +[2025-06-01 19:13:25] Speaker 5: and also talk about some feedback if you have for us. +[2025-06-01 19:13:32] Speaker 1: Thanks for the demo. +[2025-06-01 19:13:33] Speaker 1: I have a few... + +2025-06-01 19:16:08,898 - INFO - Processing audio with duration 4.640 seconds +2025-06-01 19:16:09,068 - INFO - Processing audio with duration 00:04.640 +2025-06-01 19:16:09,104 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:09,104 - INFO - Got result for chunk: output_test5/chunks/chunk_187.wav +2025-06-01 19:16:09,104 - INFO - +====== Streaming Chunk 188 ====== +[2025-06-01 19:13:25] Speaker 1: questions um so i think you mentioned in the first slide one of the things being like um you know + +2025-06-01 19:16:09,104 - INFO - Processing audio with duration 6.880 seconds +2025-06-01 19:16:09,273 - INFO - Processing audio with duration 00:06.880 +2025-06-01 19:16:09,275 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:09,403 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:09,447 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 3}) +2025-06-01 19:16:09,447 - INFO - Got result for chunk: output_test5/chunks/chunk_188.wav +2025-06-01 19:16:09,447 - INFO - +====== Streaming Chunk 189 ====== +[2025-06-01 19:13:25] Speaker 1: Kubernetes native, cloud native, +[2025-06-01 19:13:26] Speaker 1: which was one of the things we discussed previously +[2025-06-01 19:13:28] Speaker 1: as being one of the... + +2025-06-01 19:16:09,448 - INFO - Processing audio with duration 5.900 seconds +2025-06-01 19:16:09,617 - INFO - Processing audio with duration 00:05.900 +2025-06-01 19:16:09,650 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:09,650 - INFO - Got result for chunk: output_test5/chunks/chunk_189.wav +2025-06-01 19:16:09,650 - INFO - +====== Streaming Chunk 190 ====== +[2025-06-01 19:13:25] Speaker 1: key things really important for us um what other levels of integrations with oke are you planning + +2025-06-01 19:16:09,651 - INFO - Processing audio with duration 7.560 seconds +2025-06-01 19:16:09,823 - INFO - Processing audio with duration 00:07.560 +2025-06-01 19:16:09,825 - INFO - Detected language 'en' with probability 0.99 +2025-06-01 19:16:09,952 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:09,998 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 3}) +2025-06-01 19:16:09,998 - INFO - Got result for chunk: output_test5/chunks/chunk_190.wav +2025-06-01 19:16:09,998 - INFO - +====== Streaming Chunk 191 ====== +[2025-06-01 19:13:25] Speaker 1: at all, if any. +[2025-06-01 19:13:26] Speaker 1: Things that top of mind would be super helpful +[2025-06-01 19:13:28] Speaker 1: for us would be being able to + +2025-06-01 19:16:09,998 - INFO - Processing audio with duration 5.580 seconds +2025-06-01 19:16:10,168 - INFO - Processing audio with duration 00:05.580 +2025-06-01 19:16:10,200 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:10,200 - INFO - Got result for chunk: output_test5/chunks/chunk_191.wav +2025-06-01 19:16:10,200 - INFO - +====== Streaming Chunk 192 ====== +[2025-06-01 19:13:25] Speaker 1: surface some of these unhealthy node conditions to OKE or to the actual Kubernetes object. + +2025-06-01 19:16:10,200 - INFO - Processing audio with duration 6.500 seconds +2025-06-01 19:16:10,370 - INFO - Processing audio with duration 00:06.500 +2025-06-01 19:16:10,373 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:10,500 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:10,544 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:16:10,545 - INFO - Got result for chunk: output_test5/chunks/chunk_192.wav +2025-06-01 19:16:10,545 - INFO - +====== Streaming Chunk 193 ====== +[2025-06-01 19:13:25] Speaker 1: so that we can use that in our tooling. +[2025-06-01 19:13:27] Speaker 1: And then, you know, when we're launching, submitting jobs. + +2025-06-01 19:16:10,545 - INFO - Processing audio with duration 4.280 seconds +2025-06-01 19:16:10,718 - INFO - Processing audio with duration 00:04.280 +2025-06-01 19:16:10,993 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:16:10,993 - INFO - Got result for chunk: output_test5/chunks/chunk_193.wav +2025-06-01 19:16:10,993 - INFO - +====== Streaming Chunk 194 ====== +[2025-06-01 19:13:25] Speaker 1: being able to detect that a node is unhealthy. +[2025-06-01 19:13:28] Speaker 1: The other thing I think is like the monitoring. + +2025-06-01 19:16:10,993 - INFO - Processing audio with duration 6.640 seconds +2025-06-01 19:16:11,164 - INFO - Processing audio with duration 00:06.640 +2025-06-01 19:16:11,166 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:11,294 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:11,329 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:11,330 - INFO - Got result for chunk: output_test5/chunks/chunk_194.wav +2025-06-01 19:16:11,330 - INFO - +====== Streaming Chunk 195 ====== +[2025-06-01 19:13:25] Speaker 1: the rings that you mentioned, I don't think that would be super helpful for us because we're not + +2025-06-01 19:16:11,330 - INFO - Processing audio with duration 7.000 seconds +2025-06-01 19:16:11,500 - INFO - Processing audio with duration 00:07.000 +2025-06-01 19:16:11,542 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 3}) +2025-06-01 19:16:11,542 - INFO - Got result for chunk: output_test5/chunks/chunk_195.wav +2025-06-01 19:16:11,542 - INFO - +====== Streaming Chunk 196 ====== +[2025-06-01 19:13:25] Speaker 1: assigning nodes to people or teams. +[2025-06-01 19:13:28] Speaker 1: Instead, we're extremely dynamic. +[2025-06-01 19:13:31] Speaker 1: So we have... + +2025-06-01 19:16:11,542 - INFO - Processing audio with duration 8.140 seconds +2025-06-01 19:16:11,715 - INFO - Processing audio with duration 00:08.140 +2025-06-01 19:16:11,718 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:11,844 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:11,895 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 3}) +2025-06-01 19:16:11,896 - INFO - Got result for chunk: output_test5/chunks/chunk_196.wav +2025-06-01 19:16:11,896 - INFO - +====== Streaming Chunk 197 ====== +[2025-06-01 19:13:25] Speaker 1: you know, super cluster of thousands of GPUs. +[2025-06-01 19:13:27] Speaker 1: And those nodes get used. +[2025-06-01 19:13:30] Speaker 1: We use Q, which is a native. + +2025-06-01 19:16:11,896 - INFO - Processing audio with duration 8.000 seconds +2025-06-01 19:16:12,068 - INFO - Processing audio with duration 00:08.000 +2025-06-01 19:16:12,108 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:16:12,108 - INFO - Got result for chunk: output_test5/chunks/chunk_197.wav +2025-06-01 19:16:12,108 - INFO - +====== Streaming Chunk 198 ====== +[2025-06-01 19:13:25] Speaker 1: Kubernetes project to schedule workloads. +[2025-06-01 19:13:30] Speaker 1: So at any given time, we want to know, like, this + +2025-06-01 19:16:12,109 - INFO - Processing audio with duration 4.220 seconds +2025-06-01 19:16:12,288 - INFO - Processing audio with duration 00:04.220 +2025-06-01 19:16:12,291 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:12,416 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:12,459 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:12,459 - INFO - Got result for chunk: output_test5/chunks/chunk_198.wav +2025-06-01 19:16:12,459 - INFO - +====== Streaming Chunk 199 ====== +[2025-06-01 19:13:25] Speaker 1: job is running with, you know, 128 GPUs, are any of the nodes unhealthy in this job? And so that job + +2025-06-01 19:16:12,459 - INFO - Processing audio with duration 6.160 seconds +2025-06-01 19:16:12,635 - INFO - Processing audio with duration 00:06.160 +2025-06-01 19:16:12,670 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:12,670 - INFO - Got result for chunk: output_test5/chunks/chunk_199.wav +2025-06-01 19:16:12,670 - INFO - +====== Streaming Chunk 200 ====== +[2025-06-01 19:13:25] Speaker 1: might be using a different set of nodes than another job would be for the same team the next + +2025-06-01 19:16:12,670 - INFO - Processing audio with duration 7.000 seconds +2025-06-01 19:16:12,851 - INFO - Processing audio with duration 00:07.000 +2025-06-01 19:16:12,854 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:12,982 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:13,022 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:13,022 - INFO - Got result for chunk: output_test5/chunks/chunk_200.wav +2025-06-01 19:16:13,022 - INFO - +====== Streaming Chunk 201 ====== +[2025-06-01 19:13:25] Speaker 1: day. So something like that would be helpful. Okay, wonderful. I think for the answer for the first + +2025-06-01 19:16:13,022 - INFO - Processing audio with duration 6.920 seconds +2025-06-01 19:16:13,200 - INFO - Processing audio with duration 00:06.920 +2025-06-01 19:16:13,428 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:13,428 - INFO - Got result for chunk: output_test5/chunks/chunk_201.wav +2025-06-01 19:16:13,429 - INFO - +====== Streaming Chunk 202 ====== +[2025-06-01 19:13:25] Speaker 5: one is uh we are working with okay team i think okay team also has added some features lately where + +2025-06-01 19:16:13,429 - INFO - Processing audio with duration 6.720 seconds +2025-06-01 19:16:13,607 - INFO - Processing audio with duration 00:06.720 +2025-06-01 19:16:13,609 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:13,736 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:13,773 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:13,773 - INFO - Got result for chunk: output_test5/chunks/chunk_202.wav +2025-06-01 19:16:13,774 - INFO - +====== Streaming Chunk 203 ====== +[2025-06-01 19:13:25] Speaker 5: they can detect an unhealthy gpu node and start tagging them as unallocatable uh for some some + +2025-06-01 19:16:13,774 - INFO - Processing audio with duration 6.820 seconds +2025-06-01 19:16:13,960 - INFO - Processing audio with duration 00:06.820 +2025-06-01 19:16:14,002 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:16:14,002 - INFO - Got result for chunk: output_test5/chunks/chunk_203.wav +2025-06-01 19:16:14,003 - INFO - +====== Streaming Chunk 204 ====== +[2025-06-01 19:13:25] Speaker 5: the checks that they ran. +[2025-06-01 19:13:27] Speaker 5: So we are actually working with them +[2025-06-01 19:13:29] Speaker 5: to see if anything... + +2025-06-01 19:16:14,003 - INFO - Processing audio with duration 6.220 seconds +2025-06-01 19:16:14,192 - INFO - Processing audio with duration 00:06.220 +2025-06-01 19:16:14,195 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:14,323 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:14,361 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:14,362 - INFO - Got result for chunk: output_test5/chunks/chunk_204.wav +2025-06-01 19:16:14,362 - INFO - +====== Streaming Chunk 205 ====== +[2025-06-01 19:13:25] Speaker 5: we find in our health check uh that uh that says that you know this node one of the gpus in the + +2025-06-01 19:16:14,362 - INFO - Processing audio with duration 7.600 seconds +2025-06-01 19:16:14,546 - INFO - Processing audio with duration 00:07.600 +2025-06-01 19:16:14,575 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:14,576 - INFO - Got result for chunk: output_test5/chunks/chunk_205.wav +2025-06-01 19:16:14,576 - INFO - +====== Streaming Chunk 206 ====== +[2025-06-01 19:13:25] Speaker 5: node is continuous to underperform and communicate back on unschedulable. + +2025-06-01 19:16:14,576 - INFO - Processing audio with duration 5.520 seconds +2025-06-01 19:16:14,765 - INFO - Processing audio with duration 00:05.520 +2025-06-01 19:16:14,767 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:14,895 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:14,935 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:14,936 - INFO - Got result for chunk: output_test5/chunks/chunk_206.wav +2025-06-01 19:16:14,936 - INFO - +====== Streaming Chunk 207 ====== +[2025-06-01 19:13:25] Speaker 5: we will have that integration point in the future for sure. +[2025-06-01 19:13:29] Speaker 5: Right. So, and the metrics from. + +2025-06-01 19:16:14,936 - INFO - Processing audio with duration 7.040 seconds +2025-06-01 19:16:15,120 - INFO - Processing audio with duration 00:07.040 +2025-06-01 19:16:15,150 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:15,151 - INFO - Got result for chunk: output_test5/chunks/chunk_207.wav +2025-06-01 19:16:15,151 - INFO - +====== Streaming Chunk 208 ====== +[2025-06-01 19:13:25] Speaker 5: OKE directly. So if there are part level, position volume level and lower + +2025-06-01 19:16:15,151 - INFO - Processing audio with duration 5.160 seconds +2025-06-01 19:16:15,341 - INFO - Processing audio with duration 00:05.160 +2025-06-01 19:16:15,344 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:15,837 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:15,869 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:15,870 - INFO - Got result for chunk: output_test5/chunks/chunk_208.wav +2025-06-01 19:16:15,870 - INFO - +====== Streaming Chunk 209 ====== +[2025-06-01 19:13:25] Speaker 5: of good healthy prometheus metrics that come we can directly pipe all of those into this lens with + +2025-06-01 19:16:15,870 - INFO - Processing audio with duration 7.360 seconds +2025-06-01 19:16:16,055 - INFO - Processing audio with duration 00:07.360 +2025-06-01 19:16:16,089 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:16,089 - INFO - Got result for chunk: output_test5/chunks/chunk_209.wav +2025-06-01 19:16:16,090 - INFO - +====== Streaming Chunk 210 ====== +[2025-06-01 19:13:25] Speaker 5: added set of things from our side to kind of consolidate and give you that one view that means + +2025-06-01 19:16:16,090 - INFO - Processing audio with duration 6.520 seconds +2025-06-01 19:16:16,279 - INFO - Processing audio with duration 00:06.520 +2025-06-01 19:16:16,282 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:16,410 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:16,450 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:16,451 - INFO - Got result for chunk: output_test5/chunks/chunk_210.wav +2025-06-01 19:16:16,451 - INFO - +====== Streaming Chunk 211 ====== +[2025-06-01 19:13:25] Speaker 5: be needed. So that definitely is the plan, but we are looking into, okay, now a customer has to run. + +2025-06-01 19:16:16,451 - INFO - Processing audio with duration 6.680 seconds +2025-06-01 19:16:16,636 - INFO - Processing audio with duration 00:06.680 +2025-06-01 19:16:16,674 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:16,675 - INFO - Got result for chunk: output_test5/chunks/chunk_211.wav +2025-06-01 19:16:16,675 - INFO - +====== Streaming Chunk 212 ====== +[2025-06-01 19:13:25] Speaker 5: a Prometheus instance in a Kubernetes cluster and that complication is what we're trying to do. + +2025-06-01 19:16:16,675 - INFO - Processing audio with duration 4.780 seconds +2025-06-01 19:16:16,863 - INFO - Processing audio with duration 00:04.780 +2025-06-01 19:16:16,866 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:16,992 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:17,034 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2, 'Speaker 5': 1}) +2025-06-01 19:16:17,035 - INFO - Got result for chunk: output_test5/chunks/chunk_212.wav +2025-06-01 19:16:17,035 - INFO - +====== Streaming Chunk 213 ====== +[2025-06-01 19:13:25] Speaker 5: tackle the next feature set. +[2025-06-01 19:13:29] Speaker 1: Okay, makes sense. +[2025-06-01 19:13:30] Speaker 1: Yeah, just to reemphasize. + +2025-06-01 19:16:17,035 - INFO - Processing audio with duration 5.200 seconds +2025-06-01 19:16:17,222 - INFO - Processing audio with duration 00:05.200 +2025-06-01 19:16:17,251 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:17,252 - INFO - Got result for chunk: output_test5/chunks/chunk_213.wav +2025-06-01 19:16:17,252 - INFO - +====== Streaming Chunk 214 ====== +[2025-06-01 19:13:25] Speaker 1: this is like probably the most important thing for us like really being able to + +2025-06-01 19:16:17,252 - INFO - Processing audio with duration 5.440 seconds +2025-06-01 19:16:17,448 - INFO - Processing audio with duration 00:05.440 +2025-06-01 19:16:17,450 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:17,578 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:17,623 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:16:17,623 - INFO - Got result for chunk: output_test5/chunks/chunk_214.wav +2025-06-01 19:16:17,623 - INFO - +====== Streaming Chunk 215 ====== +[2025-06-01 19:13:25] Speaker 1: because we want to be able to do this automatically, right? +[2025-06-01 19:13:28] Speaker 1: If a GPU breaks overnight, we don't want... + +2025-06-01 19:16:17,623 - INFO - Processing audio with duration 6.180 seconds +2025-06-01 19:16:17,810 - INFO - Processing audio with duration 00:06.180 +2025-06-01 19:16:17,850 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:17,850 - INFO - Got result for chunk: output_test5/chunks/chunk_215.wav +2025-06-01 19:16:17,850 - INFO - +====== Streaming Chunk 216 ====== +[2025-06-01 19:13:25] Speaker 1: someone to have to, you know, go do something, go look at a Grafana dashboard we want. + +2025-06-01 19:16:17,851 - INFO - Processing audio with duration 6.020 seconds +2025-06-01 19:16:18,043 - INFO - Processing audio with duration 00:06.020 +2025-06-01 19:16:18,295 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:18,413 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:18,459 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1, 'Speaker 5': 1}) +2025-06-01 19:16:18,460 - INFO - Got result for chunk: output_test5/chunks/chunk_216.wav +2025-06-01 19:16:18,460 - INFO - +====== Streaming Chunk 217 ====== +[2025-06-01 19:13:25] Speaker 1: our jobs to automatically react to these things. +[2025-06-01 19:13:28] Speaker 5: Yeah. So I think that that brings another second. + +2025-06-01 19:16:18,460 - INFO - Processing audio with duration 6.180 seconds +2025-06-01 19:16:18,647 - INFO - Processing audio with duration 00:06.180 +2025-06-01 19:16:18,680 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:18,680 - INFO - Got result for chunk: output_test5/chunks/chunk_217.wav +2025-06-01 19:16:18,680 - INFO - +====== Streaming Chunk 218 ====== +[2025-06-01 19:13:25] Speaker 5: question or even feedback you had for us is like the static way of creating monitoring ranks is + +2025-06-01 19:16:18,680 - INFO - Processing audio with duration 4.540 seconds +2025-06-01 19:16:18,873 - INFO - Processing audio with duration 00:04.540 +2025-06-01 19:16:18,875 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:19,002 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:19,038 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:19,038 - INFO - Got result for chunk: output_test5/chunks/chunk_218.wav +2025-06-01 19:16:19,038 - INFO - +====== Streaming Chunk 219 ====== +[2025-06-01 19:13:25] Speaker 5: not ideal we fully hear you uh i think because of the dynamic if you're using q where you know + +2025-06-01 19:16:19,038 - INFO - Processing audio with duration 6.880 seconds +2025-06-01 19:16:19,225 - INFO - Processing audio with duration 00:06.880 +2025-06-01 19:16:19,262 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:19,263 - INFO - Got result for chunk: output_test5/chunks/chunk_219.wav +2025-06-01 19:16:19,263 - INFO - +====== Streaming Chunk 220 ====== +[2025-06-01 19:13:25] Speaker 5: you get it booted out based on what team and priority and cohort and all of those stuff are. + +2025-06-01 19:16:19,263 - INFO - Processing audio with duration 7.120 seconds +2025-06-01 19:16:19,455 - INFO - Processing audio with duration 00:07.120 +2025-06-01 19:16:19,458 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:19,585 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:19,621 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:19,621 - INFO - Got result for chunk: output_test5/chunks/chunk_220.wav +2025-06-01 19:16:19,621 - INFO - +====== Streaming Chunk 221 ====== +[2025-06-01 19:13:25] Speaker 5: We will integrate with OKE and the way we will look at is where the job is running or where + +2025-06-01 19:16:19,621 - INFO - Processing audio with duration 6.460 seconds +2025-06-01 19:16:19,814 - INFO - Processing audio with duration 00:06.460 +2025-06-01 19:16:19,850 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:19,850 - INFO - Got result for chunk: output_test5/chunks/chunk_221.wav +2025-06-01 19:16:19,850 - INFO - +====== Streaming Chunk 222 ====== +[2025-06-01 19:13:25] Speaker 5: the deployment is running at a point of time and try to dynamically what we call add more tags to + +2025-06-01 19:16:19,850 - INFO - Processing audio with duration 5.080 seconds +2025-06-01 19:16:20,035 - INFO - Processing audio with duration 00:05.080 +2025-06-01 19:16:20,038 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:20,166 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:20,197 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:20,197 - INFO - Got result for chunk: output_test5/chunks/chunk_222.wav +2025-06-01 19:16:20,197 - INFO - +====== Streaming Chunk 223 ====== +[2025-06-01 19:13:25] Speaker 5: metadata that's coming in to exactly relate to what experiment who ran it when when did it + +2025-06-01 19:16:20,197 - INFO - Processing audio with duration 5.460 seconds +2025-06-01 19:16:20,373 - INFO - Processing audio with duration 00:05.460 +2025-06-01 19:16:20,760 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:20,760 - INFO - Got result for chunk: output_test5/chunks/chunk_223.wav +2025-06-01 19:16:20,760 - INFO - +====== Streaming Chunk 224 ====== +[2025-06-01 19:13:25] Speaker 5: complete timestamp it and try to pull those in. +[2025-06-01 19:13:28] Speaker 5: So it's a thing that we are working. + +2025-06-01 19:16:20,760 - INFO - Processing audio with duration 6.900 seconds +2025-06-01 19:16:20,933 - INFO - Processing audio with duration 00:06.900 +2025-06-01 19:16:20,935 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:21,063 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:21,100 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:21,100 - INFO - Got result for chunk: output_test5/chunks/chunk_224.wav +2025-06-01 19:16:21,100 - INFO - +====== Streaming Chunk 225 ====== +[2025-06-01 19:13:25] Speaker 5: towards, but we started with more non-Kubernetes-based experience, and then we would quick switch. + +2025-06-01 19:16:21,100 - INFO - Processing audio with duration 5.500 seconds +2025-06-01 19:16:21,269 - INFO - Processing audio with duration 00:05.500 +2025-06-01 19:16:21,313 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:21,314 - INFO - Got result for chunk: output_test5/chunks/chunk_225.wav +2025-06-01 19:16:21,314 - INFO - +====== Streaming Chunk 226 ====== +[2025-06-01 19:13:25] Speaker 5: move into the dynamic workload where we pull in the nodes and even the GPUs, right? +[2025-06-01 19:13:30] Speaker 5: Not full nodes. + +2025-06-01 19:16:21,314 - INFO - Processing audio with duration 6.280 seconds +2025-06-01 19:16:21,486 - INFO - Processing audio with duration 00:06.280 +2025-06-01 19:16:21,489 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:21,616 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:21,652 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:21,652 - INFO - Got result for chunk: output_test5/chunks/chunk_226.wav +2025-06-01 19:16:21,652 - INFO - +====== Streaming Chunk 227 ====== +[2025-06-01 19:13:25] Speaker 5: be running every experiment so we will try to say four out of that no GPUs were running for this + +2025-06-01 19:16:21,652 - INFO - Processing audio with duration 4.820 seconds +2025-06-01 19:16:21,822 - INFO - Processing audio with duration 00:04.820 +2025-06-01 19:16:21,863 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:21,863 - INFO - Got result for chunk: output_test5/chunks/chunk_227.wav +2025-06-01 19:16:21,864 - INFO - +====== Streaming Chunk 228 ====== +[2025-06-01 19:13:25] Speaker 5: experiment and that's what is aggregated into that board. +[2025-06-01 19:13:28] Speaker 5: We will, that's a good point. + +2025-06-01 19:16:21,864 - INFO - Processing audio with duration 4.460 seconds +2025-06-01 19:16:22,038 - INFO - Processing audio with duration 00:04.460 +2025-06-01 19:16:22,041 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:22,166 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:22,207 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:22,208 - INFO - Got result for chunk: output_test5/chunks/chunk_228.wav +2025-06-01 19:16:22,208 - INFO - +====== Streaming Chunk 229 ====== +[2025-06-01 19:13:25] Speaker 5: feedback and I think we will try to prioritize that. +[2025-06-01 19:13:27] Speaker 5: Yeah, I think you're not very far from. + +2025-06-01 19:16:22,208 - INFO - Processing audio with duration 5.180 seconds +2025-06-01 19:16:22,381 - INFO - Processing audio with duration 00:05.180 +2025-06-01 19:16:22,414 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:22,415 - INFO - Got result for chunk: output_test5/chunks/chunk_229.wav +2025-06-01 19:16:22,415 - INFO - +====== Streaming Chunk 230 ====== +[2025-06-01 19:13:25] Speaker 1: that honestly even without having to do like tagging of instances and trying to keep track of + +2025-06-01 19:16:22,415 - INFO - Processing audio with duration 8.000 seconds +2025-06-01 19:16:22,593 - INFO - Processing audio with duration 00:08.000 +2025-06-01 19:16:22,595 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:22,722 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:22,754 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:22,754 - INFO - Got result for chunk: output_test5/chunks/chunk_230.wav +2025-06-01 19:16:22,754 - INFO - +====== Streaming Chunk 231 ====== +[2025-06-01 19:13:25] Speaker 1: what's running what. I think you could do some queries in your dashboard that just... + +2025-06-01 19:16:22,754 - INFO - Processing audio with duration 6.400 seconds +2025-06-01 19:16:22,928 - INFO - Processing audio with duration 00:06.400 +2025-06-01 19:16:23,213 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:23,213 - INFO - Got result for chunk: output_test5/chunks/chunk_231.wav +2025-06-01 19:16:23,213 - INFO - +====== Streaming Chunk 232 ====== +[2025-06-01 19:13:25] Speaker 1: shows like your health metrics and then also use Kubernetes metrics and see like for + +2025-06-01 19:16:23,214 - INFO - Processing audio with duration 4.400 seconds +2025-06-01 19:16:23,388 - INFO - Processing audio with duration 00:04.400 +2025-06-01 19:16:23,391 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:23,517 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:23,552 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:23,552 - INFO - Got result for chunk: output_test5/chunks/chunk_232.wav +2025-06-01 19:16:23,552 - INFO - +====== Streaming Chunk 233 ====== +[2025-06-01 19:13:25] Speaker 1: any given job or deployment, like you said, show all the nodes of that job and show if + +2025-06-01 19:16:23,552 - INFO - Processing audio with duration 4.560 seconds +2025-06-01 19:16:23,725 - INFO - Processing audio with duration 00:04.560 +2025-06-01 19:16:23,756 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:23,757 - INFO - Got result for chunk: output_test5/chunks/chunk_233.wav +2025-06-01 19:16:23,757 - INFO - +====== Streaming Chunk 234 ====== +[2025-06-01 19:13:25] Speaker 1: of them are unhealthy so i think it's just a matter of like tying everything together + +2025-06-01 19:16:23,757 - INFO - Processing audio with duration 4.560 seconds +2025-06-01 19:16:23,931 - INFO - Processing audio with duration 00:04.560 +2025-06-01 19:16:23,933 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:24,059 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:24,100 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:24,100 - INFO - Got result for chunk: output_test5/chunks/chunk_234.wav +2025-06-01 19:16:24,101 - INFO - +====== Streaming Chunk 235 ====== +[2025-06-01 19:13:25] Speaker 5: Yeah, I fully agree. Actually, we started like that, by the way, and I think Joleta and... + +2025-06-01 19:16:24,101 - INFO - Processing audio with duration 4.100 seconds +2025-06-01 19:16:24,274 - INFO - Processing audio with duration 00:04.100 +2025-06-01 19:16:24,314 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:24,314 - INFO - Got result for chunk: output_test5/chunks/chunk_235.wav +2025-06-01 19:16:24,314 - INFO - +====== Streaming Chunk 236 ====== +[2025-06-01 19:13:25] Speaker 5: and most of my team knows that we started with a full Kubernetes-based experience, but we were nudged. + +2025-06-01 19:16:24,314 - INFO - Processing audio with duration 3.940 seconds +2025-06-01 19:16:24,485 - INFO - Processing audio with duration 00:03.940 +2025-06-01 19:16:24,488 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:24,614 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:24,649 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:24,649 - INFO - Got result for chunk: output_test5/chunks/chunk_236.wav +2025-06-01 19:16:24,650 - INFO - +====== Streaming Chunk 237 ====== +[2025-06-01 19:13:25] Speaker 5: on like well i think there are a lot of people who want a full monitoring and this is where we + +2025-06-01 19:16:24,650 - INFO - Processing audio with duration 5.460 seconds +2025-06-01 19:16:24,827 - INFO - Processing audio with duration 00:05.460 +2025-06-01 19:16:24,864 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:24,864 - INFO - Got result for chunk: output_test5/chunks/chunk_237.wav +2025-06-01 19:16:24,864 - INFO - Processing audio with duration 1.420 seconds +2025-06-01 19:16:24,864 - INFO - +====== Streaming Chunk 238 ====== +[2025-06-01 19:13:25] Speaker 5: I think we have the both. We like to just pull it in like you said. + +2025-06-01 19:16:25,037 - INFO - Processing audio with duration 00:01.420 +2025-06-01 19:16:25,040 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:25,165 - INFO - Detected language 'en' with probability 0.69 +2025-06-01 19:16:25,200 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:25,200 - INFO - Got result for chunk: output_test5/chunks/chunk_238.wav +2025-06-01 19:16:25,200 - INFO - +====== Streaming Chunk 239 ====== +[2025-06-01 19:13:25] Speaker 5: and start relating where, where, what is running +[2025-06-01 19:13:28] Speaker 5: and start kind of creating that. + +2025-06-01 19:16:25,200 - INFO - Processing audio with duration 3.800 seconds +2025-06-01 19:16:25,374 - INFO - Processing audio with duration 00:03.800 +2025-06-01 19:16:25,627 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:25,627 - INFO - Got result for chunk: output_test5/chunks/chunk_239.wav +2025-06-01 19:16:25,627 - INFO - +====== Streaming Chunk 240 ====== +[2025-06-01 19:13:25] Speaker 5: key value pair mapping here. + +2025-06-01 19:16:25,628 - INFO - Processing audio with duration 5.220 seconds +2025-06-01 19:16:25,800 - INFO - Processing audio with duration 00:05.220 +2025-06-01 19:16:25,803 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:25,929 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:25,959 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:25,959 - INFO - Got result for chunk: output_test5/chunks/chunk_240.wav +2025-06-01 19:16:25,960 - INFO - +====== Streaming Chunk 241 ====== +[2025-06-01 19:13:25] Speaker 1: That's another question I'll have and then I'll pass it to like other. + +2025-06-01 19:16:25,960 - INFO - Processing audio with duration 5.960 seconds +2025-06-01 19:16:26,129 - INFO - Processing audio with duration 00:05.960 +2025-06-01 19:16:26,169 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:26,170 - INFO - Got result for chunk: output_test5/chunks/chunk_241.wav +2025-06-01 19:16:26,170 - INFO - +====== Streaming Chunk 242 ====== +[2025-06-01 19:13:25] Speaker 1: Luis, Ace, if you have any questions, is you mentioned health checks. Can you talk? + +2025-06-01 19:16:26,170 - INFO - Processing audio with duration 5.560 seconds +2025-06-01 19:16:26,342 - INFO - Processing audio with duration 00:05.560 +2025-06-01 19:16:26,344 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:26,472 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:26,503 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:26,503 - INFO - Got result for chunk: output_test5/chunks/chunk_242.wav +2025-06-01 19:16:26,503 - INFO - +====== Streaming Chunk 243 ====== +[2025-06-01 19:13:25] Speaker 1: a bit more about um how those would run like are you running anything that requires the workload + +2025-06-01 19:16:26,503 - INFO - Processing audio with duration 6.180 seconds +2025-06-01 19:16:26,673 - INFO - Processing audio with duration 00:06.180 +2025-06-01 19:16:26,719 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:26,719 - INFO - Got result for chunk: output_test5/chunks/chunk_243.wav +2025-06-01 19:16:26,720 - INFO - +====== Streaming Chunk 244 ====== +[2025-06-01 19:13:25] Speaker 1: to be idle and if so is it or the gpu note to be idle and if so is it going to be doing + +2025-06-01 19:16:26,720 - INFO - Processing audio with duration 6.080 seconds +2025-06-01 19:16:26,894 - INFO - Processing audio with duration 00:06.080 +2025-06-01 19:16:26,896 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:27,023 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:27,059 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:27,059 - INFO - Got result for chunk: output_test5/chunks/chunk_244.wav +2025-06-01 19:16:27,060 - INFO - +====== Streaming Chunk 245 ====== +[2025-06-01 19:13:25] Speaker 1: something like when nodes go idle to check the gpus or are these like running passively in the background + +2025-06-01 19:16:27,060 - INFO - Processing audio with duration 7.420 seconds +2025-06-01 19:16:27,248 - INFO - Processing audio with duration 00:07.420 +2025-06-01 19:16:27,291 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:27,292 - INFO - Got result for chunk: output_test5/chunks/chunk_245.wav +2025-06-01 19:16:27,292 - INFO - +====== Streaming Chunk 246 ====== +[2025-06-01 19:13:25] Speaker 5: So I'll ask Soumya to chime in as well on she's an ML engineer worked on the script. + +2025-06-01 19:16:27,292 - INFO - Processing audio with duration 5.680 seconds +2025-06-01 19:16:27,468 - INFO - Processing audio with duration 00:05.680 +2025-06-01 19:16:27,471 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:27,598 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:27,632 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:27,633 - INFO - Got result for chunk: output_test5/chunks/chunk_246.wav +2025-06-01 19:16:27,633 - INFO - +====== Streaming Chunk 247 ====== +[2025-06-01 19:13:25] Speaker 5: So we run this, we expect the GPUs to be idle and no workloads to be running. + +2025-06-01 19:16:27,633 - INFO - Processing audio with duration 5.400 seconds +2025-06-01 19:16:27,805 - INFO - Processing audio with duration 00:05.400 +2025-06-01 19:16:28,081 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:28,081 - INFO - Got result for chunk: output_test5/chunks/chunk_247.wav +2025-06-01 19:16:28,081 - INFO - +====== Streaming Chunk 248 ====== +[2025-06-01 19:13:25] Speaker 5: this is where the PyTorch is loading into the tensor so it's loading into the accelerator. + +2025-06-01 19:16:28,081 - INFO - Processing audio with duration 5.440 seconds +2025-06-01 19:16:28,258 - INFO - Processing audio with duration 00:05.440 +2025-06-01 19:16:28,260 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:28,387 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:28,425 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:28,426 - INFO - Got result for chunk: output_test5/chunks/chunk_248.wav +2025-06-01 19:16:28,426 - INFO - +====== Streaming Chunk 249 ====== +[2025-06-01 19:13:25] Speaker 5: running all these checks and coming back. +[2025-06-01 19:13:27] Speaker 5: So we expect no workloads to be running when this. + +2025-06-01 19:16:28,426 - INFO - Processing audio with duration 4.780 seconds +2025-06-01 19:16:28,599 - INFO - Processing audio with duration 00:04.780 +2025-06-01 19:16:28,644 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:28,645 - INFO - Got result for chunk: output_test5/chunks/chunk_249.wav +2025-06-01 19:16:28,645 - INFO - +====== Streaming Chunk 250 ====== +[2025-06-01 19:13:25] Speaker 5: this is running. We run it when you say you need to run it. +[2025-06-01 19:13:28] Speaker 5: Right now we have not automated it. + +2025-06-01 19:16:28,645 - INFO - Processing audio with duration 3.520 seconds +2025-06-01 19:16:28,813 - INFO - Processing audio with duration 00:03.520 +2025-06-01 19:16:28,816 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:28,942 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:28,987 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:28,988 - INFO - Got result for chunk: output_test5/chunks/chunk_250.wav +2025-06-01 19:16:28,988 - INFO - +====== Streaming Chunk 251 ====== +[2025-06-01 19:13:25] Speaker 5: And this is where we need feedback. +[2025-06-01 19:13:26] Speaker 5: Like, do you want to run at midnight, 12 o'clock every day? + +2025-06-01 19:16:28,988 - INFO - Processing audio with duration 7.400 seconds +2025-06-01 19:16:29,165 - INFO - Processing audio with duration 00:07.400 +2025-06-01 19:16:29,186 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:29,187 - INFO - Got result for chunk: output_test5/chunks/chunk_251.wav +2025-06-01 19:16:29,187 - INFO - +====== Streaming Chunk 252 ====== +[2025-06-01 19:13:25] Speaker 5: right you need that kind of no definitely not + +2025-06-01 19:16:29,187 - INFO - Processing audio with duration 4.780 seconds +2025-06-01 19:16:29,356 - INFO - Processing audio with duration 00:04.780 +2025-06-01 19:16:29,359 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:29,486 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:29,532 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:29,532 - INFO - Got result for chunk: output_test5/chunks/chunk_252.wav +2025-06-01 19:16:29,532 - INFO - +====== Streaming Chunk 253 ====== +[2025-06-01 19:13:25] Speaker 1: Yeah, if I'm an ML engineer, I'm like, okay, I know when my pipeline is at a pause where I say call. + +2025-06-01 19:16:29,533 - INFO - Processing audio with duration 5.460 seconds +2025-06-01 19:16:29,704 - INFO - Processing audio with duration 00:05.460 +2025-06-01 19:16:29,742 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:29,743 - INFO - Got result for chunk: output_test5/chunks/chunk_253.wav +2025-06-01 19:16:29,743 - INFO - +====== Streaming Chunk 254 ====== +[2025-06-01 19:13:25] Speaker 5: of my gpus are flushed out and idle i want to run the health checks for everything and make sure + +2025-06-01 19:16:29,743 - INFO - Processing audio with duration 5.560 seconds +2025-06-01 19:16:29,912 - INFO - Processing audio with duration 00:05.560 +2025-06-01 19:16:29,914 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:30,041 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:30,079 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:30,079 - INFO - Got result for chunk: output_test5/chunks/chunk_254.wav +2025-06-01 19:16:30,079 - INFO - +====== Streaming Chunk 255 ====== +[2025-06-01 19:13:25] Speaker 5: of that is good before we go to the next phase, right? Is that an experience you would look for? + +2025-06-01 19:16:30,080 - INFO - Processing audio with duration 6.320 seconds +2025-06-01 19:16:30,253 - INFO - Processing audio with duration 00:06.320 +2025-06-01 19:16:30,519 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:30,519 - INFO - Got result for chunk: output_test5/chunks/chunk_255.wav +2025-06-01 19:16:30,519 - INFO - +====== Streaming Chunk 256 ====== +[2025-06-01 19:13:25] Speaker 5: What we are thinking is we will give you a on-demand way for you to invoke health checkscripts. + +2025-06-01 19:16:30,519 - INFO - Processing audio with duration 4.260 seconds +2025-06-01 19:16:30,696 - INFO - Processing audio with duration 00:04.260 +2025-06-01 19:16:30,698 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:30,826 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:30,881 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 4}) +2025-06-01 19:16:30,881 - INFO - Got result for chunk: output_test5/chunks/chunk_256.wav +2025-06-01 19:16:30,881 - INFO - +====== Streaming Chunk 257 ====== +[2025-06-01 19:13:25] Speaker 5: through a REST API. +[2025-06-01 19:13:26] Speaker 5: You have to just say +[2025-06-01 19:13:28] Speaker 5: when you want to run it +[2025-06-01 19:13:30] Speaker 5: and you on-demand run it. + +2025-06-01 19:16:30,881 - INFO - Processing audio with duration 5.320 seconds +2025-06-01 19:16:31,054 - INFO - Processing audio with duration 00:05.320 +2025-06-01 19:16:31,085 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:31,086 - INFO - Got result for chunk: output_test5/chunks/chunk_257.wav +2025-06-01 19:16:31,086 - INFO - +====== Streaming Chunk 258 ====== +[2025-06-01 19:13:25] Speaker 5: Because we don't know when your GPUs are idle, then they're not. + +2025-06-01 19:16:31,086 - INFO - Processing audio with duration 5.900 seconds +2025-06-01 19:16:31,255 - INFO - Processing audio with duration 00:05.900 +2025-06-01 19:16:31,257 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:31,384 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:31,417 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:31,417 - INFO - Got result for chunk: output_test5/chunks/chunk_258.wav +2025-06-01 19:16:31,417 - INFO - +====== Streaming Chunk 259 ====== +[2025-06-01 19:13:25] Speaker 1: Interesting. I was thinking like you might be able to figure that out like from a program. + +2025-06-01 19:16:31,417 - INFO - Processing audio with duration 8.360 seconds +2025-06-01 19:16:31,589 - INFO - Processing audio with duration 00:08.360 +2025-06-01 19:16:31,622 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:31,622 - INFO - Got result for chunk: output_test5/chunks/chunk_259.wav +2025-06-01 19:16:31,623 - INFO - +====== Streaming Chunk 260 ====== +[2025-06-01 19:13:25] Speaker 1: programmatic standpoint, you might be able to get that data and whenever they are idle. + +2025-06-01 19:16:31,623 - INFO - Processing audio with duration 5.000 seconds +2025-06-01 19:16:31,792 - INFO - Processing audio with duration 00:05.000 +2025-06-01 19:16:31,794 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:31,920 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:31,955 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:16:31,955 - INFO - Got result for chunk: output_test5/chunks/chunk_260.wav +2025-06-01 19:16:31,955 - INFO - +====== Streaming Chunk 261 ====== +[2025-06-01 19:13:25] Speaker 1: kick off like a workload that runs the Huff check. +[2025-06-01 19:13:30] Speaker 1: So the intuition behind creating + +2025-06-01 19:16:31,955 - INFO - Processing audio with duration 6.560 seconds +2025-06-01 19:16:32,128 - INFO - Processing audio with duration 00:06.560 +2025-06-01 19:16:32,161 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 1}) +2025-06-01 19:16:32,161 - INFO - Got result for chunk: output_test5/chunks/chunk_261.wav +2025-06-01 19:16:32,161 - INFO - +====== Streaming Chunk 262 ====== +[2025-06-01 19:13:25] Speaker 3: this health check recipe is imagine a machine learning engineer or like you have a team that's + +2025-06-01 19:16:32,161 - INFO - Processing audio with duration 5.640 seconds +2025-06-01 19:16:32,330 - INFO - Processing audio with duration 00:05.640 +2025-06-01 19:16:32,333 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:32,460 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:32,506 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 2}) +2025-06-01 19:16:32,507 - INFO - Got result for chunk: output_test5/chunks/chunk_262.wav +2025-06-01 19:16:32,507 - INFO - +====== Streaming Chunk 263 ====== +[2025-06-01 19:13:25] Speaker 3: going to do a multi-node training or like multi-node fine tuning, right? +[2025-06-01 19:13:29] Speaker 3: Before you go into doing... + +2025-06-01 19:16:32,507 - INFO - Processing audio with duration 5.920 seconds +2025-06-01 19:16:32,679 - INFO - Processing audio with duration 00:05.920 +2025-06-01 19:16:32,943 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 1}) +2025-06-01 19:16:32,943 - INFO - Got result for chunk: output_test5/chunks/chunk_263.wav +2025-06-01 19:16:32,943 - INFO - +====== Streaming Chunk 264 ====== +[2025-06-01 19:13:25] Speaker 3: a large-scale training operation or like any kind of like workload that's going to take too much + +2025-06-01 19:16:32,943 - INFO - Processing audio with duration 5.060 seconds +2025-06-01 19:16:33,114 - INFO - Processing audio with duration 00:05.060 +2025-06-01 19:16:33,116 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:33,242 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:33,279 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 2}) +2025-06-01 19:16:33,280 - INFO - Got result for chunk: output_test5/chunks/chunk_264.wav +2025-06-01 19:16:33,280 - INFO - +====== Streaming Chunk 265 ====== +[2025-06-01 19:13:25] Speaker 3: demand of the GPU itself. +[2025-06-01 19:13:27] Speaker 3: You'd want to run this health check before that to understand that + +2025-06-01 19:16:33,280 - INFO - Processing audio with duration 6.580 seconds +2025-06-01 19:16:33,452 - INFO - Processing audio with duration 00:06.580 +2025-06-01 19:16:33,485 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 1}) +2025-06-01 19:16:33,485 - INFO - Got result for chunk: output_test5/chunks/chunk_265.wav +2025-06-01 19:16:33,486 - INFO - +====== Streaming Chunk 266 ====== +[2025-06-01 19:13:25] Speaker 3: health of your infrastructure right for example the way this health check is designed is it's based + +2025-06-01 19:16:33,486 - INFO - Processing audio with duration 4.740 seconds +2025-06-01 19:16:33,654 - INFO - Processing audio with duration 00:04.740 +2025-06-01 19:16:33,657 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:33,783 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:33,811 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 1}) +2025-06-01 19:16:33,812 - INFO - Got result for chunk: output_test5/chunks/chunk_266.wav +2025-06-01 19:16:33,812 - INFO - +====== Streaming Chunk 267 ====== +[2025-06-01 19:13:25] Speaker 3: on just 10 0 matrix multiplication right so depending upon the matrix size you probably be + +2025-06-01 19:16:33,812 - INFO - Processing audio with duration 4.480 seconds +2025-06-01 19:16:33,985 - INFO - Processing audio with duration 00:04.480 +2025-06-01 19:16:34,018 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 1}) +2025-06-01 19:16:34,018 - INFO - Got result for chunk: output_test5/chunks/chunk_267.wav +2025-06-01 19:16:34,018 - INFO - +====== Streaming Chunk 268 ====== +[2025-06-01 19:13:25] Speaker 3: loading like 818 like imagine like a matrix size of like 8192 by 818 + +2025-06-01 19:16:34,018 - INFO - Processing audio with duration 5.560 seconds +2025-06-01 19:16:34,186 - INFO - Processing audio with duration 00:05.560 +2025-06-01 19:16:34,189 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:34,315 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:34,351 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 1}) +2025-06-01 19:16:34,351 - INFO - Got result for chunk: output_test5/chunks/chunk_268.wav +2025-06-01 19:16:34,351 - INFO - +====== Streaming Chunk 269 ====== +[2025-06-01 19:13:25] Speaker 3: too right you're going to be loading all these matrix into the gpu memory and we are going to + +2025-06-01 19:16:34,351 - INFO - Processing audio with duration 6.300 seconds +2025-06-01 19:16:34,530 - INFO - Processing audio with duration 00:06.300 +2025-06-01 19:16:34,564 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 1}) +2025-06-01 19:16:34,565 - INFO - Got result for chunk: output_test5/chunks/chunk_269.wav +2025-06-01 19:16:34,565 - INFO - +====== Streaming Chunk 270 ====== +[2025-06-01 19:13:25] Speaker 3: pressurize and see how much your machine is able to take the computation throughput or like throat + +2025-06-01 19:16:34,565 - INFO - Processing audio with duration 5.320 seconds +2025-06-01 19:16:34,751 - INFO - Processing audio with duration 00:05.320 +2025-06-01 19:16:34,754 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:34,879 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:34,916 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 1}) +2025-06-01 19:16:34,917 - INFO - Got result for chunk: output_test5/chunks/chunk_270.wav +2025-06-01 19:16:34,917 - INFO - +====== Streaming Chunk 271 ====== +[2025-06-01 19:13:25] Speaker 3: the power and things like that. So ideally you would want to do it when your machine is idle and to + +2025-06-01 19:16:34,917 - INFO - Processing audio with duration 4.500 seconds +2025-06-01 19:16:35,100 - INFO - Processing audio with duration 00:04.500 +2025-06-01 19:16:35,372 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 1}) +2025-06-01 19:16:35,372 - INFO - Got result for chunk: output_test5/chunks/chunk_271.wav +2025-06-01 19:16:35,373 - INFO - +====== Streaming Chunk 272 ====== +[2025-06-01 19:13:25] Speaker 3: address your point yeah we could actually schedule and understand when the jobs are not running + +2025-06-01 19:16:35,373 - INFO - Processing audio with duration 6.260 seconds +2025-06-01 19:16:35,546 - INFO - Processing audio with duration 00:06.260 +2025-06-01 19:16:35,549 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:35,675 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:35,715 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 2}) +2025-06-01 19:16:35,716 - INFO - Got result for chunk: output_test5/chunks/chunk_272.wav +2025-06-01 19:16:35,716 - INFO - +====== Streaming Chunk 273 ====== +[2025-06-01 19:13:25] Speaker 3: and we could just run the health check at that point. +[2025-06-01 19:13:27] Speaker 3: And we can detect when the machines are... + +2025-06-01 19:16:35,716 - INFO - Processing audio with duration 5.940 seconds +2025-06-01 19:16:35,889 - INFO - Processing audio with duration 00:05.940 +2025-06-01 19:16:35,927 - INFO - Speaker assignment breakdown: Counter({'Speaker 3': 1}) +2025-06-01 19:16:35,927 - INFO - Got result for chunk: output_test5/chunks/chunk_273.wav +2025-06-01 19:16:35,928 - INFO - +====== Streaming Chunk 274 ====== +[2025-06-01 19:13:25] Speaker 3: Yeah, I think it'd be interesting to have the option to schedule it on demand, like as a... + +2025-06-01 19:16:35,928 - INFO - Processing audio with duration 4.840 seconds +2025-06-01 19:16:36,096 - INFO - Processing audio with duration 00:04.840 +2025-06-01 19:16:36,099 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:36,225 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:36,260 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:36,261 - INFO - Got result for chunk: output_test5/chunks/chunk_274.wav +2025-06-01 19:16:36,261 - INFO - +====== Streaming Chunk 275 ====== +[2025-06-01 19:13:25] Speaker 1: we hook like to the jobs like you said um but i don't i need to think more about this + +2025-06-01 19:16:36,261 - INFO - Processing audio with duration 5.660 seconds +2025-06-01 19:16:36,432 - INFO - Processing audio with duration 00:05.660 +2025-06-01 19:16:36,472 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:36,472 - INFO - Got result for chunk: output_test5/chunks/chunk_275.wav +2025-06-01 19:16:36,473 - INFO - +====== Streaming Chunk 276 ====== +[2025-06-01 19:13:25] Speaker 1: But I think, you know, my intuition is why not both, you know, like having a periodic check. + +2025-06-01 19:16:36,473 - INFO - Processing audio with duration 4.940 seconds +2025-06-01 19:16:36,641 - INFO - Processing audio with duration 00:04.940 +2025-06-01 19:16:36,644 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:36,770 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:36,811 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:16:36,811 - INFO - Got result for chunk: output_test5/chunks/chunk_276.wav +2025-06-01 19:16:36,811 - INFO - +====== Streaming Chunk 277 ====== +[2025-06-01 19:13:25] Speaker 1: that best effort runs when the node is sitting there doing nothing. +[2025-06-01 19:13:29] Speaker 1: So that if a node does go unhealthy, + +2025-06-01 19:16:36,811 - INFO - Processing audio with duration 4.920 seconds +2025-06-01 19:16:36,983 - INFO - Processing audio with duration 00:04.920 +2025-06-01 19:16:37,020 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:37,021 - INFO - Got result for chunk: output_test5/chunks/chunk_277.wav +2025-06-01 19:16:37,021 - INFO - Processing audio with duration 3.240 seconds +2025-06-01 19:16:37,021 - INFO - +====== Streaming Chunk 278 ====== +[2025-06-01 19:13:25] Speaker 1: we can find out about it early and not wait for a job to get scheduled there to find out. + +2025-06-01 19:16:37,187 - INFO - Processing audio with duration 00:03.240 +2025-06-01 19:16:37,190 - INFO - Detected language 'en' with probability 0.99 +2025-06-01 19:16:37,316 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:37,359 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:37,359 - INFO - Got result for chunk: output_test5/chunks/chunk_278.wav +2025-06-01 19:16:37,360 - INFO - +====== Streaming Chunk 279 ====== +[2025-06-01 19:13:25] Speaker 5: Yeah, no, we'll take it as a good feedback. +[2025-06-01 19:13:28] Speaker 5: We have still, because we have to think. + +2025-06-01 19:16:37,360 - INFO - Processing audio with duration 5.480 seconds +2025-06-01 19:16:37,533 - INFO - Processing audio with duration 00:05.480 +2025-06-01 19:16:37,806 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:37,807 - INFO - Got result for chunk: output_test5/chunks/chunk_279.wav +2025-06-01 19:16:37,807 - INFO - +====== Streaming Chunk 280 ====== +[2025-06-01 19:13:25] Speaker 5: through a little bit on scheduling because it takes roughly five minutes for this head check + +2025-06-01 19:16:37,807 - INFO - Processing audio with duration 3.640 seconds +2025-06-01 19:16:37,978 - INFO - Processing audio with duration 00:03.640 +2025-06-01 19:16:37,981 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:38,107 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:38,141 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:38,142 - INFO - Got result for chunk: output_test5/chunks/chunk_280.wav +2025-06-01 19:16:38,142 - INFO - +====== Streaming Chunk 281 ====== +[2025-06-01 19:13:25] Speaker 5: to finish and in mix of this if your queue ends up scheduling another job uh they'll they'll + +2025-06-01 19:16:38,142 - INFO - Processing audio with duration 4.660 seconds +2025-06-01 19:16:38,313 - INFO - Processing audio with duration 00:04.660 +2025-06-01 19:16:38,353 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:38,353 - INFO - Got result for chunk: output_test5/chunks/chunk_281.wav +2025-06-01 19:16:38,353 - INFO - +====== Streaming Chunk 282 ====== +[2025-06-01 19:13:25] Speaker 5: it can't find an ideal GPU. So they go on pending. So we have to just think through this. + +2025-06-01 19:16:38,353 - INFO - Processing audio with duration 3.500 seconds +2025-06-01 19:16:38,521 - INFO - Processing audio with duration 00:03.500 +2025-06-01 19:16:38,523 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:38,649 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:38,680 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:38,680 - INFO - Got result for chunk: output_test5/chunks/chunk_282.wav +2025-06-01 19:16:38,680 - INFO - +====== Streaming Chunk 283 ====== +[2025-06-01 19:13:25] Speaker 5: Yeah, no, this would definitely, yes, this would definitely have to be an... + +2025-06-01 19:16:38,681 - INFO - Processing audio with duration 3.620 seconds +2025-06-01 19:16:38,851 - INFO - Processing audio with duration 00:03.620 +2025-06-01 19:16:38,877 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:38,878 - INFO - Got result for chunk: output_test5/chunks/chunk_283.wav +2025-06-01 19:16:38,878 - INFO - +====== Streaming Chunk 284 ====== +[2025-06-01 19:13:25] Speaker 1: interruptible workload so that our workloads can always schedule the priority. + +2025-06-01 19:16:38,878 - INFO - Processing audio with duration 4.440 seconds +2025-06-01 19:16:39,048 - INFO - Processing audio with duration 00:04.440 +2025-06-01 19:16:39,051 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:39,177 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:39,203 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:39,203 - INFO - Got result for chunk: output_test5/chunks/chunk_284.wav +2025-06-01 19:16:39,203 - INFO - +====== Streaming Chunk 285 ====== +[2025-06-01 19:13:25] Speaker 5: Exactly right you cannot boot this workload because this is running at the system + +2025-06-01 19:16:39,203 - INFO - Processing audio with duration 4.820 seconds +2025-06-01 19:16:39,374 - INFO - Processing audio with duration 00:04.820 +2025-06-01 19:16:39,403 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:39,403 - INFO - Got result for chunk: output_test5/chunks/chunk_285.wav +2025-06-01 19:16:39,403 - INFO - +====== Streaming Chunk 286 ====== +[2025-06-01 19:13:25] Speaker 5: level, right? And Kubernetes layers are at a much higher layer too. + +2025-06-01 19:16:39,404 - INFO - Processing audio with duration 2.920 seconds +2025-06-01 19:16:39,573 - INFO - Processing audio with duration 00:02.920 +2025-06-01 19:16:39,575 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:39,702 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:39,737 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:39,737 - INFO - Got result for chunk: output_test5/chunks/chunk_286.wav +2025-06-01 19:16:39,738 - INFO - +====== Streaming Chunk 287 ====== +[2025-06-01 19:13:25] Speaker 5: and we need to think of this a little bit but i think this is really good feedback um thank you + +2025-06-01 19:16:39,738 - INFO - Processing audio with duration 6.780 seconds +2025-06-01 19:16:39,911 - INFO - Processing audio with duration 00:06.780 +2025-06-01 19:16:40,219 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:16:40,219 - INFO - Got result for chunk: output_test5/chunks/chunk_287.wav +2025-06-01 19:16:40,219 - INFO - Processing audio with duration 4.500 seconds +2025-06-01 19:16:40,220 - INFO - +====== Streaming Chunk 288 ====== +[2025-06-01 19:13:25] Speaker 1: Yeah, no, thank you. +[2025-06-01 19:13:26] Speaker 1: Louise Ace, you have anything? + +2025-06-01 19:16:40,395 - INFO - Processing audio with duration 00:04.500 +2025-06-01 19:16:40,398 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:40,524 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:40,554 - INFO - Speaker assignment breakdown: Counter({'Speaker 4': 1}) +2025-06-01 19:16:40,554 - INFO - Got result for chunk: output_test5/chunks/chunk_288.wav +2025-06-01 19:16:40,554 - INFO - +====== Streaming Chunk 289 ====== +[2025-06-01 19:13:25] Speaker 4: Mostly just echoing your thoughts. I mean, when we talk about node lifecycle, + +2025-06-01 19:16:40,555 - INFO - Processing audio with duration 3.580 seconds +2025-06-01 19:16:40,733 - INFO - Processing audio with duration 00:03.580 +2025-06-01 19:16:40,782 - INFO - Speaker assignment breakdown: Counter({'Speaker 4': 3}) +2025-06-01 19:16:40,783 - INFO - Got result for chunk: output_test5/chunks/chunk_289.wav +2025-06-01 19:16:40,783 - INFO - +====== Streaming Chunk 290 ====== +[2025-06-01 19:13:25] Speaker 4: It's very much like an ongoing thing. +[2025-06-01 19:13:26] Speaker 4: It's not sort of a one off thing. +[2025-06-01 19:13:27] Speaker 4: So just, yeah. + +2025-06-01 19:16:40,783 - INFO - Processing audio with duration 6.180 seconds +2025-06-01 19:16:40,960 - INFO - Processing audio with duration 00:06.180 +2025-06-01 19:16:40,962 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:41,090 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:41,114 - INFO - Speaker assignment breakdown: Counter({'Speaker 4': 1}) +2025-06-01 19:16:41,114 - INFO - Got result for chunk: output_test5/chunks/chunk_290.wav +2025-06-01 19:16:41,114 - INFO - +====== Streaming Chunk 291 ====== +[2025-06-01 19:13:25] Speaker 4: like having both elements and thinking about how nodes like proceed through. + +2025-06-01 19:16:41,114 - INFO - Processing audio with duration 7.740 seconds +2025-06-01 19:16:41,290 - INFO - Processing audio with duration 00:07.740 +2025-06-01 19:16:41,332 - INFO - Speaker assignment breakdown: Counter({'Speaker 4': 1}) +2025-06-01 19:16:41,333 - INFO - Got result for chunk: output_test5/chunks/chunk_291.wav +2025-06-01 19:16:41,333 - INFO - +====== Streaming Chunk 292 ====== +[2025-06-01 19:13:25] Speaker 4: I don't want to say this process exactly, but like, I don't know, mostly just echoing Cecile's + +2025-06-01 19:16:41,333 - INFO - Processing audio with duration 5.560 seconds +2025-06-01 19:16:41,506 - INFO - Processing audio with duration 00:05.560 +2025-06-01 19:16:41,508 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:41,635 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:41,675 - INFO - Speaker assignment breakdown: Counter({'Speaker 4': 1}) +2025-06-01 19:16:41,675 - INFO - Got result for chunk: output_test5/chunks/chunk_292.wav +2025-06-01 19:16:41,675 - INFO - +====== Streaming Chunk 293 ====== +[2025-06-01 19:13:25] Speaker 4: thoughts. Okay. Yeah. Thank you. So what, what we are trying to also avoid is before you shut this + +2025-06-01 19:16:41,675 - INFO - Processing audio with duration 5.800 seconds +2025-06-01 19:16:41,846 - INFO - Processing audio with duration 00:05.800 +2025-06-01 19:16:41,884 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:41,885 - INFO - Got result for chunk: output_test5/chunks/chunk_293.wav +2025-06-01 19:16:41,885 - INFO - +====== Streaming Chunk 294 ====== +[2025-06-01 19:13:25] Speaker 5: machine and turn it back into OCI to repair. We want to see what is the issue, right? + +2025-06-01 19:16:41,885 - INFO - Processing audio with duration 5.220 seconds +2025-06-01 19:16:42,058 - INFO - Processing audio with duration 00:05.220 +2025-06-01 19:16:42,060 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:42,187 - INFO - Detected language 'en' with probability 0.99 +2025-06-01 19:16:42,220 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:42,220 - INFO - Got result for chunk: output_test5/chunks/chunk_294.wav +2025-06-01 19:16:42,220 - INFO - +====== Streaming Chunk 295 ====== +[2025-06-01 19:13:25] Speaker 5: deeper into what where is the concern is it a single gpu always coming back with a lower + +2025-06-01 19:16:42,220 - INFO - Processing audio with duration 6.560 seconds +2025-06-01 19:16:42,415 - INFO - Processing audio with duration 00:06.560 +2025-06-01 19:16:42,661 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:42,661 - INFO - Got result for chunk: output_test5/chunks/chunk_295.wav +2025-06-01 19:16:42,661 - INFO - +====== Streaming Chunk 296 ====== +[2025-06-01 19:13:25] Speaker 5: performance compared to the rest seven of them in the same node or is it consistent uh no + +2025-06-01 19:16:42,661 - INFO - Processing audio with duration 4.900 seconds +2025-06-01 19:16:42,848 - INFO - Processing audio with duration 00:04.900 +2025-06-01 19:16:42,850 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:42,978 - INFO - Detected language 'en' with probability 0.99 +2025-06-01 19:16:43,027 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:16:43,028 - INFO - Got result for chunk: output_test5/chunks/chunk_296.wav +2025-06-01 19:16:43,028 - INFO - +====== Streaming Chunk 297 ====== +[2025-06-01 19:13:25] Speaker 5: RDMA nick flapping issues, errors that show up. +[2025-06-01 19:13:28] Speaker 5: And this is a start. +[2025-06-01 19:13:30] Speaker 5: And where we want to go. + +2025-06-01 19:16:43,028 - INFO - Processing audio with duration 4.880 seconds +2025-06-01 19:16:43,207 - INFO - Processing audio with duration 00:04.880 +2025-06-01 19:16:43,242 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:43,242 - INFO - Got result for chunk: output_test5/chunks/chunk_297.wav +2025-06-01 19:16:43,242 - INFO - +====== Streaming Chunk 298 ====== +[2025-06-01 19:13:25] Speaker 5: is if we start seeing a pattern of things which are very consistent with a lot of other customers + +2025-06-01 19:16:43,243 - INFO - Processing audio with duration 6.600 seconds +2025-06-01 19:16:43,417 - INFO - Processing audio with duration 00:06.600 +2025-06-01 19:16:43,420 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:43,548 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:43,596 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:16:43,596 - INFO - Got result for chunk: output_test5/chunks/chunk_298.wav +2025-06-01 19:16:43,596 - INFO - +====== Streaming Chunk 299 ====== +[2025-06-01 19:13:25] Speaker 5: and a lot of type of GPUs, +[2025-06-01 19:13:26] Speaker 5: we will come back with recommendations +[2025-06-01 19:13:28] Speaker 5: through an agentic AI type of app. + +2025-06-01 19:16:43,596 - INFO - Processing audio with duration 6.040 seconds +2025-06-01 19:16:43,768 - INFO - Processing audio with duration 00:06.040 +2025-06-01 19:16:43,803 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:43,803 - INFO - Got result for chunk: output_test5/chunks/chunk_299.wav +2025-06-01 19:16:43,803 - INFO - +====== Streaming Chunk 300 ====== +[2025-06-01 19:13:25] Speaker 5: flow where probably just a reboot may fix the things or it could be specific to a driver. + +2025-06-01 19:16:43,803 - INFO - Processing audio with duration 5.440 seconds +2025-06-01 19:16:43,978 - INFO - Processing audio with duration 00:05.440 +2025-06-01 19:16:43,980 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:44,108 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:44,146 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:44,147 - INFO - Got result for chunk: output_test5/chunks/chunk_300.wav +2025-06-01 19:16:44,147 - INFO - +====== Streaming Chunk 301 ====== +[2025-06-01 19:13:25] Speaker 5: the GPU driver you have, which has seen this incompatibility with the Mellanox drivers we have. + +2025-06-01 19:16:44,147 - INFO - Processing audio with duration 5.600 seconds +2025-06-01 19:16:44,320 - INFO - Processing audio with duration 00:05.600 +2025-06-01 19:16:44,356 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:44,356 - INFO - Got result for chunk: output_test5/chunks/chunk_301.wav +2025-06-01 19:16:44,357 - INFO - +====== Streaming Chunk 302 ====== +[2025-06-01 19:13:25] Speaker 5: example. So we are trying to learn from using the data as well as all the issues customers. + +2025-06-01 19:16:44,357 - INFO - Processing audio with duration 6.520 seconds +2025-06-01 19:16:44,530 - INFO - Processing audio with duration 00:06.520 +2025-06-01 19:16:44,532 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:44,659 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:44,710 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:16:44,710 - INFO - Got result for chunk: output_test5/chunks/chunk_302.wav +2025-06-01 19:16:44,711 - INFO - +====== Streaming Chunk 303 ====== +[2025-06-01 19:13:25] Speaker 5: give us to start recommending, right? +[2025-06-01 19:13:27] Speaker 5: So this is just a start for us to start collecting, +[2025-06-01 19:13:29] Speaker 5: but next approach for us is + +2025-06-01 19:16:44,711 - INFO - Processing audio with duration 4.820 seconds +2025-06-01 19:16:44,904 - INFO - Processing audio with duration 00:04.820 +2025-06-01 19:16:45,104 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:45,104 - INFO - Got result for chunk: output_test5/chunks/chunk_303.wav +2025-06-01 19:16:45,104 - INFO - Processing audio with duration 5.360 seconds +2025-06-01 19:16:45,104 - INFO - +====== Streaming Chunk 304 ====== +[2025-06-01 19:13:25] Speaker 5: remove the noise and tell if there is really a issue with the host or if this is a transient issue. + +2025-06-01 19:16:45,276 - INFO - Processing audio with duration 00:05.360 +2025-06-01 19:16:45,279 - INFO - Detected language 'en' with probability 0.99 +2025-06-01 19:16:45,406 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:45,434 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:45,434 - INFO - Got result for chunk: output_test5/chunks/chunk_304.wav +2025-06-01 19:16:45,435 - INFO - +====== Streaming Chunk 305 ====== +[2025-06-01 19:13:25] Speaker 5: or not an issue with something related to an experiment that has been set up. + +2025-06-01 19:16:45,435 - INFO - Processing audio with duration 2.640 seconds +2025-06-01 19:16:45,606 - INFO - Processing audio with duration 00:02.640 +2025-06-01 19:16:45,655 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:45,655 - INFO - Got result for chunk: output_test5/chunks/chunk_305.wav +2025-06-01 19:16:45,655 - INFO - +====== Streaming Chunk 306 ====== +[2025-06-01 19:13:25] Speaker 5: workload is being scheduled on this or the topology that it's been deployed with. +[2025-06-01 19:13:29] Speaker 5: Any of those questions? + +2025-06-01 19:16:45,655 - INFO - Processing audio with duration 4.940 seconds +2025-06-01 19:16:45,830 - INFO - Processing audio with duration 00:04.940 +2025-06-01 19:16:45,833 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:45,960 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:45,983 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:45,983 - INFO - Got result for chunk: output_test5/chunks/chunk_306.wav +2025-06-01 19:16:45,984 - INFO - +====== Streaming Chunk 307 ====== +[2025-06-01 19:13:25] Speaker 5: Right, so that's where we want to get to. + +2025-06-01 19:16:45,984 - INFO - Processing audio with duration 5.900 seconds +2025-06-01 19:16:46,156 - INFO - Processing audio with duration 00:05.900 +2025-06-01 19:16:46,204 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:46,204 - INFO - Got result for chunk: output_test5/chunks/chunk_307.wav +2025-06-01 19:16:46,204 - INFO - +====== Streaming Chunk 308 ====== +[2025-06-01 19:13:25] Speaker 5: Awesome. So here's the log as well. +[2025-06-01 19:13:28] Speaker 5: This is this we'll build up more and more. + +2025-06-01 19:16:46,204 - INFO - Processing audio with duration 8.340 seconds +2025-06-01 19:16:46,285 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:46,401 - INFO - Processing audio with duration 00:08.340 +2025-06-01 19:16:46,448 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:46,448 - INFO - Got result for chunk: output_test5/chunks/chunk_308.wav +2025-06-01 19:16:46,449 - INFO - +====== Streaming Chunk 309 ====== +[2025-06-01 19:13:25] Speaker 5: stuff and it's all accessible either through portal or an api where you don't have to assess + +2025-06-01 19:16:46,449 - INFO - Processing audio with duration 5.700 seconds +2025-06-01 19:16:46,621 - INFO - Processing audio with duration 00:05.700 +2025-06-01 19:16:46,624 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:46,752 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:46,781 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:46,782 - INFO - Got result for chunk: output_test5/chunks/chunk_309.wav +2025-06-01 19:16:46,782 - INFO - +====== Streaming Chunk 310 ====== +[2025-06-01 19:13:25] Speaker 5: an instance. Because the scripts are going to be standardized that what you see against + +2025-06-01 19:16:46,782 - INFO - Processing audio with duration 7.420 seconds +2025-06-01 19:16:46,960 - INFO - Processing audio with duration 00:07.420 +2025-06-01 19:16:46,994 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:46,995 - INFO - Got result for chunk: output_test5/chunks/chunk_310.wav +2025-06-01 19:16:46,995 - INFO - +====== Streaming Chunk 311 ====== +[2025-06-01 19:13:25] Speaker 5: what OCI support sees everything will be consistent and maybe in the future once we have support team + +2025-06-01 19:16:46,995 - INFO - Processing audio with duration 6.200 seconds +2025-06-01 19:16:47,172 - INFO - Processing audio with duration 00:06.200 +2025-06-01 19:16:47,483 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:47,610 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:47,645 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:47,645 - INFO - Got result for chunk: output_test5/chunks/chunk_311.wav +2025-06-01 19:16:47,645 - INFO - +====== Streaming Chunk 312 ====== +[2025-06-01 19:13:25] Speaker 5: running full diagnosis they know where to narrow down the issue to like exact GPU or the NICs or + +2025-06-01 19:16:47,646 - INFO - Processing audio with duration 5.820 seconds +2025-06-01 19:16:47,819 - INFO - Processing audio with duration 00:05.820 +2025-06-01 19:16:47,849 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:47,849 - INFO - Got result for chunk: output_test5/chunks/chunk_312.wav +2025-06-01 19:16:47,850 - INFO - +====== Streaming Chunk 313 ====== +[2025-06-01 19:13:25] Speaker 5: anything instead of running this full you know six hours uh diagnosis on the host right + +2025-06-01 19:16:47,850 - INFO - Processing audio with duration 4.800 seconds +2025-06-01 19:16:48,024 - INFO - Processing audio with duration 00:04.800 +2025-06-01 19:16:48,026 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:48,154 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:48,197 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:48,198 - INFO - Got result for chunk: output_test5/chunks/chunk_313.wav +2025-06-01 19:16:48,198 - INFO - +====== Streaming Chunk 314 ====== +[2025-06-01 19:13:25] Speaker 5: So we're just trying to narrow down that time it takes to fix these machines too. +[2025-06-01 19:13:30] Speaker 5: So that will help. + +2025-06-01 19:16:48,198 - INFO - Processing audio with duration 4.100 seconds +2025-06-01 19:16:48,371 - INFO - Processing audio with duration 00:04.100 +2025-06-01 19:16:48,407 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:48,408 - INFO - Got result for chunk: output_test5/chunks/chunk_314.wav +2025-06-01 19:16:48,408 - INFO - +====== Streaming Chunk 315 ====== +[2025-06-01 19:13:25] Speaker 1: When you say consistent, is the idea that you give us a set of scripts and we run these? + +2025-06-01 19:16:48,408 - INFO - Processing audio with duration 6.540 seconds +2025-06-01 19:16:48,578 - INFO - Processing audio with duration 00:06.540 +2025-06-01 19:16:48,581 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:48,709 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:48,729 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:48,730 - INFO - Got result for chunk: output_test5/chunks/chunk_315.wav +2025-06-01 19:16:48,730 - INFO - +====== Streaming Chunk 316 ====== +[2025-06-01 19:13:25] Speaker 1: or they see the results of our health checks. + +2025-06-01 19:16:48,730 - INFO - Processing audio with duration 5.300 seconds +2025-06-01 19:16:48,904 - INFO - Processing audio with duration 00:05.300 +2025-06-01 19:16:48,947 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:48,948 - INFO - Got result for chunk: output_test5/chunks/chunk_316.wav +2025-06-01 19:16:48,948 - INFO - +====== Streaming Chunk 317 ====== +[2025-06-01 19:13:25] Speaker 1: So right now with the way it is integrated, the OCI support does not see the results, but you could just take... + +2025-06-01 19:16:48,948 - INFO - Processing audio with duration 5.840 seconds +2025-06-01 19:16:49,121 - INFO - Processing audio with duration 00:05.840 +2025-06-01 19:16:49,124 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:49,252 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:49,283 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:49,284 - INFO - Got result for chunk: output_test5/chunks/chunk_317.wav +2025-06-01 19:16:49,284 - INFO - +====== Streaming Chunk 318 ====== +[2025-06-01 19:13:25] Speaker 5: these results copy paste in your support ticket if you need to OCI support will have access + +2025-06-01 19:16:49,284 - INFO - Processing audio with duration 5.400 seconds +2025-06-01 19:16:49,455 - INFO - Processing audio with duration 00:05.400 +2025-06-01 19:16:49,494 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:49,494 - INFO - Got result for chunk: output_test5/chunks/chunk_318.wav +2025-06-01 19:16:49,494 - INFO - +====== Streaming Chunk 319 ====== +[2025-06-01 19:13:25] Speaker 5: the same health check tools and scripts that you are running. +[2025-06-01 19:13:29] Speaker 5: So it's a consistent tool. + +2025-06-01 19:16:49,494 - INFO - Processing audio with duration 6.140 seconds +2025-06-01 19:16:49,666 - INFO - Processing audio with duration 00:06.140 +2025-06-01 19:16:49,920 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:50,038 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:50,089 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:16:50,089 - INFO - Got result for chunk: output_test5/chunks/chunk_319.wav +2025-06-01 19:16:50,089 - INFO - +====== Streaming Chunk 320 ====== +[2025-06-01 19:13:25] Speaker 5: that you both are running, +[2025-06-01 19:13:26] Speaker 5: where what you have seen as the results, +[2025-06-01 19:13:29] Speaker 5: for example, the score, + +2025-06-01 19:16:50,089 - INFO - Processing audio with duration 5.160 seconds +2025-06-01 19:16:50,278 - INFO - Processing audio with duration 00:05.160 +2025-06-01 19:16:50,325 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:16:50,325 - INFO - Got result for chunk: output_test5/chunks/chunk_320.wav +2025-06-01 19:16:50,325 - INFO - Processing audio with duration 5.980 seconds +2025-06-01 19:16:50,326 - INFO - +====== Streaming Chunk 321 ====== +[2025-06-01 19:13:25] Speaker 5: So here's the flops, for example. +[2025-06-01 19:13:28] Speaker 5: This is the duration, right? +[2025-06-01 19:13:30] Speaker 5: What you are seeing... + +2025-06-01 19:16:50,516 - INFO - Processing audio with duration 00:05.980 +2025-06-01 19:16:50,519 - INFO - Detected language 'en' with probability 0.99 +2025-06-01 19:16:50,647 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:50,686 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:50,686 - INFO - Got result for chunk: output_test5/chunks/chunk_321.wav +2025-06-01 19:16:50,686 - INFO - +====== Streaming Chunk 322 ====== +[2025-06-01 19:13:25] Speaker 5: when OCI support is trying to run this on their own. +[2025-06-01 19:13:28] Speaker 5: There is a lot of consistency. + +2025-06-01 19:16:50,686 - INFO - Processing audio with duration 5.620 seconds +2025-06-01 19:16:50,864 - INFO - Processing audio with duration 00:05.620 +2025-06-01 19:16:50,900 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:50,900 - INFO - Got result for chunk: output_test5/chunks/chunk_322.wav +2025-06-01 19:16:50,901 - INFO - +====== Streaming Chunk 323 ====== +[2025-06-01 19:13:25] Speaker 5: Right. And if you see this GPU consistently coming back with like, okay, it takes 31 seconds. + +2025-06-01 19:16:50,901 - INFO - Processing audio with duration 0.840 seconds +2025-06-01 19:16:51,061 - INFO - Processing audio with duration 00:00.840 +2025-06-01 19:16:51,064 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:51,188 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:51,214 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:51,215 - INFO - Got result for chunk: output_test5/chunks/chunk_323.wav +2025-06-01 19:16:51,215 - INFO - +====== Streaming Chunk 324 ====== +[2025-06-01 19:13:25] Speaker 5: It's easy to narrow down saying that GQ7 is something. + +2025-06-01 19:16:51,215 - INFO - Processing audio with duration 4.060 seconds +2025-06-01 19:16:51,386 - INFO - Processing audio with duration 00:04.060 +2025-06-01 19:16:51,402 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:51,402 - INFO - Got result for chunk: output_test5/chunks/chunk_324.wav +2025-06-01 19:16:51,403 - INFO - +====== Streaming Chunk 325 ====== +[2025-06-01 19:13:25] Speaker 5: that we need to request. + +2025-06-01 19:16:51,403 - INFO - Processing audio with duration 4.080 seconds +2025-06-01 19:16:51,573 - INFO - Processing audio with duration 00:04.080 +2025-06-01 19:16:51,575 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:51,702 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:51,739 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:51,739 - INFO - Got result for chunk: output_test5/chunks/chunk_325.wav +2025-06-01 19:16:51,740 - INFO - +====== Streaming Chunk 326 ====== +[2025-06-01 19:13:25] Speaker 1: I see. Yeah, I think it would be even better if we can get to a stage where they... + +2025-06-01 19:16:51,740 - INFO - Processing audio with duration 0.100 seconds +2025-06-01 19:16:51,900 - INFO - Processing audio with duration 00:00.100 +2025-06-01 19:16:51,939 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:16:51,940 - INFO - Got result for chunk: output_test5/chunks/chunk_326.wav +2025-06-01 19:16:51,940 - INFO - +====== Streaming Chunk 327 ====== +[2025-06-01 19:13:25] Speaker 1: like have access to historical results and they can just see things that are +[2025-06-01 19:13:28] Speaker 1: ZeeBrand. + +2025-06-01 19:16:51,940 - INFO - Processing audio with duration 2.900 seconds +2025-06-01 19:16:52,110 - INFO - Processing audio with duration 00:02.900 +2025-06-01 19:16:52,348 - INFO - Detected language 'en' with probability 0.97 +2025-06-01 19:16:52,472 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:52,482 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:52,482 - INFO - Got result for chunk: output_test5/chunks/chunk_327.wav +2025-06-01 19:16:52,482 - INFO - +====== Streaming Chunk 328 ====== +[2025-06-01 19:13:25] Speaker 1: Yeah. + +2025-06-01 19:16:52,482 - INFO - Processing audio with duration 4.240 seconds +2025-06-01 19:16:52,655 - INFO - Processing audio with duration 00:04.240 +2025-06-01 19:16:52,678 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:52,678 - INFO - Got result for chunk: output_test5/chunks/chunk_328.wav +2025-06-01 19:16:52,678 - INFO - +====== Streaming Chunk 329 ====== +[2025-06-01 19:13:25] Speaker 1: instead of having to reproduce and copy pasting things. + +2025-06-01 19:16:52,678 - INFO - Processing audio with duration 5.380 seconds +2025-06-01 19:16:52,849 - INFO - Processing audio with duration 00:05.380 +2025-06-01 19:16:52,851 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:52,979 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:53,011 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:53,011 - INFO - Got result for chunk: output_test5/chunks/chunk_329.wav +2025-06-01 19:16:53,011 - INFO - +====== Streaming Chunk 330 ====== +[2025-06-01 19:13:25] Speaker 1: i fully agree i think that's our goal but we are not a oci service yet + +2025-06-01 19:16:53,012 - INFO - Processing audio with duration 3.680 seconds +2025-06-01 19:16:53,184 - INFO - Processing audio with duration 00:03.680 +2025-06-01 19:16:53,215 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:53,216 - INFO - Got result for chunk: output_test5/chunks/chunk_330.wav +2025-06-01 19:16:53,216 - INFO - +====== Streaming Chunk 331 ====== +[2025-06-01 19:13:25] Speaker 5: right and uh because it's all running the customer's tenancy uh there's obviously + +2025-06-01 19:16:53,216 - INFO - Processing audio with duration 5.780 seconds +2025-06-01 19:16:53,387 - INFO - Processing audio with duration 00:05.780 +2025-06-01 19:16:53,389 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:53,518 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:53,555 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:53,555 - INFO - Got result for chunk: output_test5/chunks/chunk_331.wav +2025-06-01 19:16:53,555 - INFO - +====== Streaming Chunk 332 ====== +[2025-06-01 19:13:25] Speaker 5: little bit of things about what we can collect, what we cannot. And we had to go through those. + +2025-06-01 19:16:53,555 - INFO - Processing audio with duration 5.140 seconds +2025-06-01 19:16:53,729 - INFO - Processing audio with duration 00:05.140 +2025-06-01 19:16:53,768 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:16:53,768 - INFO - Got result for chunk: output_test5/chunks/chunk_332.wav +2025-06-01 19:16:53,768 - INFO - +====== Streaming Chunk 333 ====== +[2025-06-01 19:13:25] Speaker 5: product approval phases. +[2025-06-01 19:13:28] Speaker 5: And then we will probably +[2025-06-01 19:13:29] Speaker 5: support and have access to this. + +2025-06-01 19:16:53,769 - INFO - Processing audio with duration 5.920 seconds +2025-06-01 19:16:53,946 - INFO - Processing audio with duration 00:05.920 +2025-06-01 19:16:53,948 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:54,077 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:54,111 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:54,112 - INFO - Got result for chunk: output_test5/chunks/chunk_333.wav +2025-06-01 19:16:54,112 - INFO - +====== Streaming Chunk 334 ====== +[2025-06-01 19:13:25] Speaker 1: Okay, makes sense. One other question for Prometheus. You mentioned at some point in the + +2025-06-01 19:16:54,112 - INFO - Processing audio with duration 2.180 seconds +2025-06-01 19:16:54,282 - INFO - Processing audio with duration 00:02.180 +2025-06-01 19:16:54,315 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:54,315 - INFO - Got result for chunk: output_test5/chunks/chunk_334.wav +2025-06-01 19:16:54,315 - INFO - +====== Streaming Chunk 335 ====== +[2025-06-01 19:13:25] Speaker 1: this slide something about setting up a CPU cluster or gate cluster to run Prometheus. + +2025-06-01 19:16:54,315 - INFO - Processing audio with duration 5.260 seconds +2025-06-01 19:16:54,491 - INFO - Processing audio with duration 00:05.260 +2025-06-01 19:16:54,775 - INFO - Detected language 'en' with probability 0.99 +2025-06-01 19:16:54,899 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:54,919 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:54,920 - INFO - Got result for chunk: output_test5/chunks/chunk_335.wav +2025-06-01 19:16:54,920 - INFO - +====== Streaming Chunk 336 ====== +[2025-06-01 19:13:25] Speaker 1: can we reuse our existing primukes instead? + +2025-06-01 19:16:54,920 - INFO - Processing audio with duration 5.840 seconds +2025-06-01 19:16:55,093 - INFO - Processing audio with duration 00:05.840 +2025-06-01 19:16:55,132 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:55,132 - INFO - Got result for chunk: output_test5/chunks/chunk_336.wav +2025-06-01 19:16:55,132 - INFO - +====== Streaming Chunk 337 ====== +[2025-06-01 19:13:25] Speaker 5: Yes, yes, absolutely. You can use your existing. You need to just enable push gateway and add. + +2025-06-01 19:16:55,132 - INFO - Processing audio with duration 5.500 seconds +2025-06-01 19:16:55,310 - INFO - Processing audio with duration 00:05.500 +2025-06-01 19:16:55,312 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:55,441 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:55,471 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:55,471 - INFO - Got result for chunk: output_test5/chunks/chunk_337.wav +2025-06-01 19:16:55,472 - INFO - +====== Streaming Chunk 338 ====== +[2025-06-01 19:13:25] Speaker 5: push gateway configurations because that's how all of the scripts push their results and metrics. + +2025-06-01 19:16:55,472 - INFO - Processing audio with duration 2.420 seconds +2025-06-01 19:16:55,642 - INFO - Processing audio with duration 00:02.420 +2025-06-01 19:16:55,669 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:55,669 - INFO - Got result for chunk: output_test5/chunks/chunk_338.wav +2025-06-01 19:16:55,670 - INFO - +====== Streaming Chunk 339 ====== +[2025-06-01 19:13:25] Speaker 5: Okay, can you give a bit more detail about how this works? + +2025-06-01 19:16:55,670 - INFO - Processing audio with duration 3.120 seconds +2025-06-01 19:16:55,840 - INFO - Processing audio with duration 00:03.120 +2025-06-01 19:16:55,843 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:55,967 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:55,985 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:55,986 - INFO - Got result for chunk: output_test5/chunks/chunk_339.wav +2025-06-01 19:16:55,986 - INFO - Processing audio with duration 1.840 seconds +2025-06-01 19:16:55,986 - INFO - +====== Streaming Chunk 340 ====== +[2025-06-01 19:13:25] Speaker 1: scripts push metrics or how does that work + +2025-06-01 19:16:56,156 - INFO - Processing audio with duration 00:01.840 +2025-06-01 19:16:56,182 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:56,182 - INFO - Got result for chunk: output_test5/chunks/chunk_340.wav +2025-06-01 19:16:56,182 - INFO - +====== Streaming Chunk 341 ====== +[2025-06-01 19:13:25] Speaker 5: Yeah, Hrithika, can you elaborate on this? + +2025-06-01 19:16:56,182 - INFO - Processing audio with duration 1.280 seconds +2025-06-01 19:16:56,346 - INFO - Processing audio with duration 00:01.280 +2025-06-01 19:16:56,349 - INFO - Detected language 'en' with probability 0.96 +2025-06-01 19:16:56,474 - INFO - Detected language 'en' with probability 0.95 +2025-06-01 19:16:56,619 - INFO - Speaker assignment breakdown: Counter({'Speaker 2': 1}) +2025-06-01 19:16:56,619 - INFO - Got result for chunk: output_test5/chunks/chunk_342.wav +2025-06-01 19:16:56,619 - INFO - +====== Streaming Chunk 343 ====== +[2025-06-01 19:13:25] Speaker 2: Hello? Can you hear me? + +2025-06-01 19:16:56,619 - INFO - Processing audio with duration 0.220 seconds +2025-06-01 19:16:56,782 - INFO - Processing audio with duration 00:00.220 +2025-06-01 19:16:56,782 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:56,783 - INFO - Got result for chunk: output_test5/chunks/chunk_341.wav +2025-06-01 19:16:56,783 - INFO - Processing audio with duration 2.260 seconds +2025-06-01 19:16:56,783 - INFO - +====== Streaming Chunk 342 ====== +[2025-06-01 19:13:25] Speaker 5: Is VTK wrong? + +2025-06-01 19:16:56,954 - INFO - Processing audio with duration 00:02.260 +2025-06-01 19:16:57,234 - INFO - Detected language 'en' with probability 0.95 +2025-06-01 19:16:57,348 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:57,400 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:16:57,400 - INFO - Got result for chunk: output_test5/chunks/chunk_343.wav +2025-06-01 19:16:57,401 - INFO - +====== Streaming Chunk 344 ====== +[2025-06-01 19:13:25] Speaker 5: Yeah. + +2025-06-01 19:16:57,401 - INFO - Processing audio with duration 4.980 seconds +2025-06-01 19:16:57,579 - INFO - Processing audio with duration 00:04.980 +2025-06-01 19:16:57,606 - INFO - Speaker assignment breakdown: Counter({'Speaker 2': 1}) +2025-06-01 19:16:57,606 - INFO - Got result for chunk: output_test5/chunks/chunk_344.wav +2025-06-01 19:16:57,607 - INFO - +====== Streaming Chunk 345 ====== +[2025-06-01 19:13:25] Speaker 2: All right. Could you give that question once again? + +2025-06-01 19:16:57,607 - INFO - Processing audio with duration 4.180 seconds +2025-06-01 19:16:57,796 - INFO - Processing audio with duration 00:04.180 +2025-06-01 19:16:57,798 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:57,924 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:57,952 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:57,952 - INFO - Got result for chunk: output_test5/chunks/chunk_345.wav +2025-06-01 19:16:57,952 - INFO - +====== Streaming Chunk 346 ====== +[2025-06-01 19:13:25] Speaker 1: It was just having a bit more detail about how the metrics get emitted. + +2025-06-01 19:16:57,952 - INFO - Processing audio with duration 1.020 seconds +2025-06-01 19:16:58,131 - INFO - Processing audio with duration 00:01.020 +2025-06-01 19:16:58,167 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:58,167 - INFO - Got result for chunk: output_test5/chunks/chunk_346.wav +2025-06-01 19:16:58,167 - INFO - +====== Streaming Chunk 347 ====== +[2025-06-01 19:13:25] Speaker 1: Because I understand there are scripts and then I'm just trying to figure out like how we could. + +2025-06-01 19:16:58,168 - INFO - Processing audio with duration 9.260 seconds +2025-06-01 19:16:58,351 - INFO - Processing audio with duration 00:09.260 +2025-06-01 19:16:58,353 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:58,481 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:58,494 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:58,494 - INFO - Got result for chunk: output_test5/chunks/chunk_347.wav +2025-06-01 19:16:58,494 - INFO - +====== Streaming Chunk 348 ====== +[2025-06-01 19:13:25] Speaker 1: scrape our own metrics. + +2025-06-01 19:16:58,494 - INFO - Processing audio with duration 9.320 seconds +2025-06-01 19:16:58,673 - INFO - Processing audio with duration 00:09.320 +2025-06-01 19:16:58,721 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:16:58,722 - INFO - Got result for chunk: output_test5/chunks/chunk_348.wav +2025-06-01 19:16:58,722 - INFO - +====== Streaming Chunk 349 ====== +[2025-06-01 19:13:25] Speaker 1: Oh yeah, so we basically build the node exporter, DCGM, AMD exporter, all of that depending on the GPU. + +2025-06-01 19:16:58,722 - INFO - Processing audio with duration 6.940 seconds +2025-06-01 19:16:58,894 - INFO - Processing audio with duration 00:06.940 +2025-06-01 19:16:58,897 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:59,024 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:59,056 - INFO - Speaker assignment breakdown: Counter({'Speaker 2': 1}) +2025-06-01 19:16:59,057 - INFO - Got result for chunk: output_test5/chunks/chunk_349.wav +2025-06-01 19:16:59,057 - INFO - +====== Streaming Chunk 350 ====== +[2025-06-01 19:13:25] Speaker 2: it is and bring those up and there are also some metrics which are OCI specific which are + +2025-06-01 19:16:59,057 - INFO - Processing audio with duration 3.880 seconds +2025-06-01 19:16:59,236 - INFO - Processing audio with duration 00:03.880 +2025-06-01 19:16:59,284 - INFO - Speaker assignment breakdown: Counter({'Speaker 2': 2}) +2025-06-01 19:16:59,284 - INFO - Got result for chunk: output_test5/chunks/chunk_350.wav +2025-06-01 19:16:59,284 - INFO - +====== Streaming Chunk 351 ====== +[2025-06-01 19:13:25] Speaker 2: return in a Go plugin and that plugin +[2025-06-01 19:13:29] Speaker 2: fits metrics to the push gateway at a regular interval. + +2025-06-01 19:16:59,285 - INFO - Processing audio with duration 4.640 seconds +2025-06-01 19:16:59,456 - INFO - Processing audio with duration 00:04.640 +2025-06-01 19:16:59,688 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:59,803 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:16:59,844 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:16:59,844 - INFO - Got result for chunk: output_test5/chunks/chunk_351.wav +2025-06-01 19:16:59,844 - INFO - +====== Streaming Chunk 352 ====== +[2025-06-01 19:13:25] Speaker 5: Yeah, all of the scripts here are... +[2025-06-01 19:13:27] Speaker 5: Does that make sense? + +2025-06-01 19:16:59,844 - INFO - Processing audio with duration 5.540 seconds +2025-06-01 19:17:00,019 - INFO - Processing audio with duration 00:05.540 +2025-06-01 19:17:00,042 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:17:00,042 - INFO - Got result for chunk: output_test5/chunks/chunk_352.wav +2025-06-01 19:17:00,042 - INFO - +====== Streaming Chunk 353 ====== +[2025-06-01 19:13:25] Speaker 5: push gateway if you see it here and + +2025-06-01 19:17:00,042 - INFO - Processing audio with duration 6.980 seconds +2025-06-01 19:17:00,215 - INFO - Processing audio with duration 00:06.980 +2025-06-01 19:17:00,218 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:00,344 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:00,386 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:17:00,386 - INFO - Got result for chunk: output_test5/chunks/chunk_353.wav +2025-06-01 19:17:00,386 - INFO - +====== Streaming Chunk 354 ====== +[2025-06-01 19:13:25] Speaker 5: And these are scraped by Prometheus +[2025-06-01 19:13:27] Speaker 5: based on Prometheus script config. +[2025-06-01 19:13:29] Speaker 5: And- + +2025-06-01 19:17:00,386 - INFO - Processing audio with duration 5.920 seconds +2025-06-01 19:17:00,562 - INFO - Processing audio with duration 00:05.920 +2025-06-01 19:17:00,613 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:17:00,613 - INFO - Got result for chunk: output_test5/chunks/chunk_354.wav +2025-06-01 19:17:00,613 - INFO - +====== Streaming Chunk 355 ====== +[2025-06-01 19:13:25] Speaker 5: Push gateway is part of a service that is default in Prometheus server install. +[2025-06-01 19:13:30] Speaker 5: And it's the same. + +2025-06-01 19:17:00,613 - INFO - Processing audio with duration 4.940 seconds +2025-06-01 19:17:00,786 - INFO - Processing audio with duration 00:04.940 +2025-06-01 19:17:00,789 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:00,916 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:00,971 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 3, 'Speaker 5': 2}) +2025-06-01 19:17:00,972 - INFO - Got result for chunk: output_test5/chunks/chunk_355.wav +2025-06-01 19:17:00,972 - INFO - +====== Streaming Chunk 356 ====== +[2025-06-01 19:13:25] Speaker 5: mechanism. +[2025-06-01 19:13:27] Speaker 5: Gotcha. +[2025-06-01 19:13:28] Speaker 1: Okay. +[2025-06-01 19:13:28] Speaker 1: I'll look into this a bit more. +[2025-06-01 19:13:29] Speaker 1: I haven't used push gateway before. + +2025-06-01 19:17:00,972 - INFO - Processing audio with duration 6.040 seconds +2025-06-01 19:17:01,146 - INFO - Processing audio with duration 00:06.040 +2025-06-01 19:17:01,203 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2, 'Speaker 5': 1}) +2025-06-01 19:17:01,203 - INFO - Got result for chunk: output_test5/chunks/chunk_356.wav +2025-06-01 19:17:01,204 - INFO - +====== Streaming Chunk 357 ====== +[2025-06-01 19:13:25] Speaker 1: So that's probably where my gap comes from. +[2025-06-01 19:13:27] Speaker 1: Okay. +[2025-06-01 19:13:27] Speaker 5: Yeah, I think it's part of the setup. + +2025-06-01 19:17:01,204 - INFO - Processing audio with duration 5.140 seconds +2025-06-01 19:17:01,379 - INFO - Processing audio with duration 00:05.140 +2025-06-01 19:17:01,381 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:01,507 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:01,550 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:17:01,550 - INFO - Got result for chunk: output_test5/chunks/chunk_357.wav +2025-06-01 19:17:01,551 - INFO - +====== Streaming Chunk 358 ====== +[2025-06-01 19:13:25] Speaker 5: And if you go, if you run Prometheus through operator, +[2025-06-01 19:13:28] Speaker 5: Push Gateway is usually installed as a service. + +2025-06-01 19:17:01,551 - INFO - Processing audio with duration 7.700 seconds +2025-06-01 19:17:01,731 - INFO - Processing audio with duration 00:07.700 +2025-06-01 19:17:02,157 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:17:02,158 - INFO - Got result for chunk: output_test5/chunks/chunk_358.wav +2025-06-01 19:17:02,158 - INFO - +====== Streaming Chunk 359 ====== +[2025-06-01 19:13:25] Speaker 5: It should have its own service endpoint, +[2025-06-01 19:13:28] Speaker 5: and you have to just enable a few things. + +2025-06-01 19:17:02,158 - INFO - Processing audio with duration 4.280 seconds +2025-06-01 19:17:02,167 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:02,331 - INFO - Processing audio with duration 00:04.280 +2025-06-01 19:17:02,369 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:17:02,369 - INFO - Got result for chunk: output_test5/chunks/chunk_359.wav +2025-06-01 19:17:02,369 - INFO - +====== Streaming Chunk 360 ====== +[2025-06-01 19:13:25] Speaker 5: and you're good to go. Yeah. So, this is for instead of you going and reaching out. + +2025-06-01 19:17:02,370 - INFO - Processing audio with duration 5.580 seconds +2025-06-01 19:17:02,540 - INFO - Processing audio with duration 00:05.580 +2025-06-01 19:17:02,543 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:02,672 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:02,700 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:17:02,701 - INFO - Got result for chunk: output_test5/chunks/chunk_360.wav +2025-06-01 19:17:02,701 - INFO - +====== Streaming Chunk 361 ====== +[2025-06-01 19:13:25] Speaker 5: going and reaching out and pulling the metrics out which is where you need firewall exception + +2025-06-01 19:17:02,701 - INFO - Processing audio with duration 5.480 seconds +2025-06-01 19:17:02,876 - INFO - Processing audio with duration 00:05.480 +2025-06-01 19:17:02,917 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:17:02,917 - INFO - Got result for chunk: output_test5/chunks/chunk_361.wav +2025-06-01 19:17:02,918 - INFO - +====== Streaming Chunk 362 ====== +[2025-06-01 19:13:25] Speaker 5: and opening the ports and all of that. +[2025-06-01 19:13:27] Speaker 5: Here, any metrics you are generating, including DCGM, + +2025-06-01 19:17:02,918 - INFO - Processing audio with duration 5.860 seconds +2025-06-01 19:17:03,098 - INFO - Processing audio with duration 00:05.860 +2025-06-01 19:17:03,101 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:03,229 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:03,265 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:17:03,265 - INFO - Got result for chunk: output_test5/chunks/chunk_362.wav +2025-06-01 19:17:03,265 - INFO - +====== Streaming Chunk 363 ====== +[2025-06-01 19:13:25] Speaker 5: all of those are pool metrics, right? +[2025-06-01 19:13:27] Speaker 5: We push it directly to the push page. + +2025-06-01 19:17:03,266 - INFO - Processing audio with duration 5.800 seconds +2025-06-01 19:17:03,442 - INFO - Processing audio with duration 00:05.800 +2025-06-01 19:17:03,480 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:17:03,481 - INFO - Got result for chunk: output_test5/chunks/chunk_363.wav +2025-06-01 19:17:03,481 - INFO - +====== Streaming Chunk 364 ====== +[2025-06-01 19:13:25] Speaker 5: and your Prometheus is scraping push gateway basically. +[2025-06-01 19:13:29] Speaker 5: It's like an intermediate data store. + +2025-06-01 19:17:03,481 - INFO - Processing audio with duration 5.420 seconds +2025-06-01 19:17:03,660 - INFO - Processing audio with duration 00:05.420 +2025-06-01 19:17:03,662 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:03,792 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:03,824 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:17:03,824 - INFO - Got result for chunk: output_test5/chunks/chunk_364.wav +2025-06-01 19:17:03,825 - INFO - +====== Streaming Chunk 365 ====== +[2025-06-01 19:13:25] Speaker 5: All right. Any further questions? And we are also eager if you have any feedback. + +2025-06-01 19:17:03,825 - INFO - Processing audio with duration 6.740 seconds +2025-06-01 19:17:04,002 - INFO - Processing audio with duration 00:06.740 +2025-06-01 19:17:04,041 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:17:04,041 - INFO - Got result for chunk: output_test5/chunks/chunk_365.wav +2025-06-01 19:17:04,042 - INFO - +====== Streaming Chunk 366 ====== +[2025-06-01 19:13:25] Speaker 5: can think about we are thinking in next roughly two weeks you can you can have access to all of + +2025-06-01 19:17:04,042 - INFO - Processing audio with duration 5.940 seconds +2025-06-01 19:17:04,225 - INFO - Processing audio with duration 00:05.940 +2025-06-01 19:17:04,227 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:04,637 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 3}) +2025-06-01 19:17:04,637 - INFO - Got result for chunk: output_test5/chunks/chunk_366.wav +2025-06-01 19:17:04,637 - INFO - +====== Streaming Chunk 367 ====== +[2025-06-01 19:13:25] Speaker 5: the health check scripts. +[2025-06-01 19:13:27] Speaker 5: We'll get access to your repo +[2025-06-01 19:13:29] Speaker 5: once you share your GitHub IDs. + +2025-06-01 19:17:04,637 - INFO - Processing audio with duration 6.820 seconds +2025-06-01 19:17:04,814 - INFO - Processing audio with duration 00:06.820 +2025-06-01 19:17:04,816 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:04,944 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:04,975 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:17:04,975 - INFO - Got result for chunk: output_test5/chunks/chunk_367.wav +2025-06-01 19:17:04,976 - INFO - +====== Streaming Chunk 368 ====== +[2025-06-01 19:13:25] Speaker 5: can poke around and really we are here to listen and and build something that that makes uh + +2025-06-01 19:17:04,976 - INFO - Processing audio with duration 5.760 seconds +2025-06-01 19:17:05,154 - INFO - Processing audio with duration 00:05.760 +2025-06-01 19:17:05,188 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:17:05,189 - INFO - Got result for chunk: output_test5/chunks/chunk_368.wav +2025-06-01 19:17:05,189 - INFO - +====== Streaming Chunk 369 ====== +[2025-06-01 19:13:25] Speaker 5: life easy right so we are here to listen and you know continue to iterate as fast as we + +2025-06-01 19:17:05,189 - INFO - Processing audio with duration 4.520 seconds +2025-06-01 19:17:05,366 - INFO - Processing audio with duration 00:04.520 +2025-06-01 19:17:05,369 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:05,496 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:05,552 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2, 'Speaker 1': 2}) +2025-06-01 19:17:05,552 - INFO - Got result for chunk: output_test5/chunks/chunk_369.wav +2025-06-01 19:17:05,552 - INFO - +====== Streaming Chunk 370 ====== +[2025-06-01 19:13:25] Speaker 5: can to give you what you're looking for. +[2025-06-01 19:13:27] Speaker 5: So. +[2025-06-01 19:13:28] Speaker 1: Thanks, Samar. +[2025-06-01 19:13:29] Speaker 1: I think overall this is, you know. + +2025-06-01 19:17:05,552 - INFO - Processing audio with duration 5.040 seconds +2025-06-01 19:17:05,748 - INFO - Processing audio with duration 00:05.040 +2025-06-01 19:17:05,785 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:17:05,786 - INFO - Got result for chunk: output_test5/chunks/chunk_370.wav +2025-06-01 19:17:05,786 - INFO - +====== Streaming Chunk 371 ====== +[2025-06-01 19:13:25] Speaker 1: something that we've been asking for. So really, really happy it's, it's getting there. + +2025-06-01 19:17:05,786 - INFO - Processing audio with duration 5.360 seconds +2025-06-01 19:17:05,974 - INFO - Processing audio with duration 00:05.360 +2025-06-01 19:17:05,976 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:06,104 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:06,138 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:17:06,139 - INFO - Got result for chunk: output_test5/chunks/chunk_371.wav +2025-06-01 19:17:06,139 - INFO - +====== Streaming Chunk 372 ====== +[2025-06-01 19:13:25] Speaker 1: and you all are focusing your efforts there. +[2025-06-01 19:13:26] Speaker 1: It's definitely heading the right direction. + +2025-06-01 19:17:06,139 - INFO - Processing audio with duration 5.800 seconds +2025-06-01 19:17:06,314 - INFO - Processing audio with duration 00:05.800 +2025-06-01 19:17:06,358 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:17:06,358 - INFO - Got result for chunk: output_test5/chunks/chunk_372.wav +2025-06-01 19:17:06,359 - INFO - +====== Streaming Chunk 373 ====== +[2025-06-01 19:13:25] Speaker 1: I think looking at what we have today that we built ourselves a best over the past year, +[2025-06-01 19:13:29] Speaker 1: it's definitely... + +2025-06-01 19:17:06,359 - INFO - Processing audio with duration 4.080 seconds +2025-06-01 19:17:06,531 - INFO - Processing audio with duration 00:04.080 +2025-06-01 19:17:06,534 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:07,022 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:07,055 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:17:07,055 - INFO - Got result for chunk: output_test5/chunks/chunk_373.wav +2025-06-01 19:17:07,056 - INFO - +====== Streaming Chunk 374 ====== +[2025-06-01 19:13:25] Speaker 1: not there yet. I think what we have right now is a bit more advanced just because we've built + +2025-06-01 19:17:07,056 - INFO - Processing audio with duration 5.540 seconds +2025-06-01 19:17:07,231 - INFO - Processing audio with duration 00:05.540 +2025-06-01 19:17:07,266 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:17:07,267 - INFO - Got result for chunk: output_test5/chunks/chunk_374.wav +2025-06-01 19:17:07,267 - INFO - +====== Streaming Chunk 375 ====== +[2025-06-01 19:13:25] Speaker 1: our own set of scripts and like we've iterated on them and we have all that automation. + +2025-06-01 19:17:07,267 - INFO - Processing audio with duration 7.440 seconds +2025-06-01 19:17:07,447 - INFO - Processing audio with duration 00:07.440 +2025-06-01 19:17:07,450 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:07,577 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:07,614 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 2}) +2025-06-01 19:17:07,614 - INFO - Got result for chunk: output_test5/chunks/chunk_375.wav +2025-06-01 19:17:07,615 - INFO - +====== Streaming Chunk 376 ====== +[2025-06-01 19:13:25] Speaker 1: and integration with Kubernetes already. +[2025-06-01 19:13:27] Speaker 1: But I like where this is going and yeah, excited too. + +2025-06-01 19:17:07,615 - INFO - Processing audio with duration 4.740 seconds +2025-06-01 19:17:07,795 - INFO - Processing audio with duration 00:04.740 +2025-06-01 19:17:07,864 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 5, 'Speaker 1': 2}) +2025-06-01 19:17:07,865 - INFO - Got result for chunk: output_test5/chunks/chunk_376.wav +2025-06-01 19:17:07,865 - INFO - +====== Streaming Chunk 377 ====== +[2025-06-01 19:13:25] Speaker 1: work with you all too. +[2025-06-01 19:13:26] Speaker 1: It's great. +[2025-06-01 19:13:27] Speaker 5: Okay. +[2025-06-01 19:13:28] Speaker 5: Yeah. +[2025-06-01 19:13:28] Speaker 5: We'll try to, +[2025-06-01 19:13:30] Speaker 5: right now, +[2025-06-01 19:13:31] Speaker 5: we are not overly + +2025-06-01 19:17:07,865 - INFO - Processing audio with duration 4.580 seconds +2025-06-01 19:17:07,924 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:08,047 - INFO - Processing audio with duration 00:04.580 +2025-06-01 19:17:08,175 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:08,202 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:17:08,202 - INFO - Got result for chunk: output_test5/chunks/chunk_377.wav +2025-06-01 19:17:08,202 - INFO - +====== Streaming Chunk 378 ====== +[2025-06-01 19:13:25] Speaker 5: Kubernetes heavy features and if we will probably in the next iteration or a milestone + +2025-06-01 19:17:08,202 - INFO - Processing audio with duration 6.180 seconds +2025-06-01 19:17:08,376 - INFO - Processing audio with duration 00:06.180 +2025-06-01 19:17:08,404 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 1}) +2025-06-01 19:17:08,404 - INFO - Got result for chunk: output_test5/chunks/chunk_378.wav +2025-06-01 19:17:08,404 - INFO - +====== Streaming Chunk 379 ====== +[2025-06-01 19:13:25] Speaker 5: add all of those capabilities to bring Kubernetes metrics and that experience together. + +2025-06-01 19:17:08,405 - INFO - Processing audio with duration 9.460 seconds +2025-06-01 19:17:08,576 - INFO - Processing audio with duration 00:09.460 +2025-06-01 19:17:08,578 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:08,706 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:08,751 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2}) +2025-06-01 19:17:08,752 - INFO - Got result for chunk: output_test5/chunks/chunk_379.wav +2025-06-01 19:17:08,752 - INFO - +====== Streaming Chunk 380 ====== +[2025-06-01 19:13:25] Speaker 5: And hopefully, Sisal, I'll post that. +[2025-06-01 19:13:27] Speaker 5: You would be open to trying it out and giving us the feedback. + +2025-06-01 19:17:08,752 - INFO - Processing audio with duration 5.440 seconds +2025-06-01 19:17:08,929 - INFO - Processing audio with duration 00:05.440 +2025-06-01 19:17:08,973 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2, 'Speaker 1': 1}) +2025-06-01 19:17:08,974 - INFO - Got result for chunk: output_test5/chunks/chunk_380.wav +2025-06-01 19:17:08,974 - INFO - Processing audio with duration 0.340 seconds +2025-06-01 19:17:08,974 - INFO - +====== Streaming Chunk 381 ====== +[2025-06-01 19:13:25] Speaker 5: and iterating like that. +[2025-06-01 19:13:29] Speaker 1: Sounds good. +[2025-06-01 19:13:32] Speaker 5: Thanks for making the time to give us a demo. + +2025-06-01 19:17:09,139 - INFO - Processing audio with duration 00:00.340 +2025-06-01 19:17:09,447 - INFO - Detected language 'en' with probability 1.00 +2025-06-01 19:17:09,573 - INFO - Detected language 'en' with probability 0.19 +2025-06-01 19:17:09,616 - INFO - Speaker assignment breakdown: Counter({'Speaker 5': 2, 'Speaker 1': 1}) +2025-06-01 19:17:09,616 - INFO - Got result for chunk: output_test5/chunks/chunk_381.wav +2025-06-01 19:17:09,616 - INFO - +====== Streaming Chunk 382 ====== +[2025-06-01 19:13:25] Speaker 1: Appreciate it. +[2025-06-01 19:13:25] Speaker 5: Thanks for spending your time with us as well. +[2025-06-01 19:13:28] Speaker 5: So anyone else, any questions? + +2025-06-01 19:17:09,617 - INFO - Processing audio with duration 0.160 seconds +2025-06-01 19:17:09,779 - INFO - Processing audio with duration 00:00.160 +2025-06-01 19:17:09,792 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:17:09,792 - INFO - Got result for chunk: output_test5/chunks/chunk_382.wav +2025-06-01 19:17:09,792 - INFO - +====== Streaming Chunk 383 ====== +[2025-06-01 19:13:25] Speaker 1: Thank you. + +2025-06-01 19:17:09,792 - INFO - Processing audio with duration 0.340 seconds +2025-06-01 19:17:09,953 - INFO - Processing audio with duration 00:00.340 +2025-06-01 19:17:09,956 - INFO - Detected language 'en' with probability 0.29 +2025-06-01 19:17:10,082 - INFO - Detected language 'en' with probability 0.21 +2025-06-01 19:17:10,093 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:17:10,093 - INFO - Got result for chunk: output_test5/chunks/chunk_383.wav +2025-06-01 19:17:10,093 - INFO - +====== Streaming Chunk 384 ====== +[2025-06-01 19:13:25] Speaker 1: Thank you. + +2025-06-01 19:17:10,093 - INFO - Processing audio with duration 0.240 seconds +2025-06-01 19:17:10,255 - INFO - Processing audio with duration 00:00.240 +2025-06-01 19:17:10,268 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:17:10,269 - INFO - Got result for chunk: output_test5/chunks/chunk_384.wav +2025-06-01 19:17:10,269 - INFO - +====== Streaming Chunk 385 ====== +[2025-06-01 19:13:25] Speaker 1: Thank you. + +2025-06-01 19:17:10,381 - INFO - Detected language 'en' with probability 0.26 +2025-06-01 19:17:10,520 - INFO - Speaker assignment breakdown: Counter({'Speaker 1': 1}) +2025-06-01 19:17:10,520 - INFO - Got result for chunk: output_test5/chunks/chunk_385.wav +2025-06-01 19:17:10,520 - INFO - +====== Streaming Chunk 386 ====== +[2025-06-01 19:13:25] Speaker 1: Thank you. + +2025-06-01 19:17:10,593 - INFO - Transcript saved to: output_test5/audio1788670787_all_transcripts_20250601_191710.txt +2025-06-01 19:17:10,593 - INFO - CUDA available before summarization: True +2025-06-01 19:17:10,593 - INFO - Loading summarizer model from HuggingFace: mistralai/Mistral-7B-Instruct-v0.1 +2025-06-01 19:17:27,226 - INFO - Summarizing chunk 1/8 +2025-06-01 19:17:38,366 - INFO - Summarizing chunk 2/8 +2025-06-01 19:17:48,951 - INFO - Summarizing chunk 3/8 +2025-06-01 19:18:05,688 - INFO - Summarizing chunk 4/8 +2025-06-01 19:18:30,083 - INFO - Summarizing chunk 5/8 +2025-06-01 19:18:39,819 - INFO - Summarizing chunk 6/8 +2025-06-01 19:18:53,281 - INFO - Summarizing chunk 7/8 +2025-06-01 19:19:09,144 - INFO - Summarizing chunk 8/8 +2025-06-01 19:19:22,050 - INFO - Summary successfully appended to transcript TXT. +2025-06-01 19:19:22,056 - INFO - Final JSON saved to: output_test5/audio1788670787_all_transcripts_20250601_191710.json diff --git a/docs/whisper_transcription/README.md b/docs/whisper_transcription/README.md new file mode 100644 index 0000000..b404853 --- /dev/null +++ b/docs/whisper_transcription/README.md @@ -0,0 +1,138 @@ +# Whisper Transcription API + +### Transcription + Summarization + Diarization Pipeline (FastAPI-powered) + +This blueprint provides a complete solution for running **audio/video transcription**, **speaker diarization**, and **summarization** via a RESTful API. It integrates [Faster-Whisper](https://github.com/guillaumekln/faster-whisper) for efficient transcription, [pyannote.audio](https://github.com/pyannote/pyannote-audio) for diarization, and Hugging Face instruction-tuned LLMs (e.g., Mistral-7B) for summarization. It supports multi-GPU acceleration, real-time streaming logs, and JSON/text output formats. + +--- + +## Key Features + +| Capability | Description | +|------------------------|-----------------------------------------------------------------------------------------------| +| Transcription | Fast, multi-GPU inference with Faster-Whisper | +| Summarization | Uses Mistral-7B (or other HF models) to create summaries of long transcripts | +| Speaker Diarization | Global speaker labeling via pyannote.audio | +| Denoising | Hybrid removal of background noise using Demucs and noisereduce | +| Real-Time Streaming | Logs stream live via HTTP if enabled | +| Format Compatibility | Supports `.mp3`, `.wav`, `.flac`, `.aac`, `.m4a`, `.mp4`, `.webm`, `.mov`, `.mkv`, `.avi`, etc. | + +--- + +## Deployment on OCI Blueprint + +### Sample Recipe (Service Mode) +```json +{ + "recipe_id": "whisper_transcription", + "recipe_mode": "service", + "deployment_name": "whisper-transcription-a10", + "recipe_image_uri": "iad.ocir.io/iduyx1qnmway/corrino-devops-repository:whisper_transcription_v8", + "recipe_node_shape": "VM.GPU.A10.2", + "recipe_replica_count": 1, + "recipe_container_port": "8000", + "recipe_nvidia_gpu_count": 2, + "recipe_node_pool_size": 1, + "recipe_node_boot_volume_size_in_gbs": 200, + "recipe_ephemeral_storage_size": 100, + "recipe_shared_memory_volume_size_limit_in_mb": 200 +} +``` + +### Endpoint +``` +POST https://.nip.io/transcribe +``` +**Example:** +`https://whisper-transcription-a10-6666.130-162-199-33.nip.io/transcribe` + +--- + +## API Parameters + +| Name | Type | Description | +|-------------------|-----------|-----------------------------------------------------------------------------------------------------------------------| +| `audio_url` | string | URL to audio file in OCI Object Storage (requires PAR) | +| `model` | string | Whisper model to use: `base`, `medium`, `large`, `turbo`, etc. | +| `summary` | bool | Whether to generate a summary (default: false). Requires `hf_token` if model path not provided | +| `speaker` | bool | Whether to run diarization (default: false). Requires `hf_token` | +| `max_speakers` | int | (Optional) Maximum number of speakers expected for diarization | +| `denoise` | bool | Whether to apply noise reduction | +| `streaming` | bool | Enables real-time logs via /stream_log endpoint | +| `hf_token` | string | Hugging Face access token (required for diarization or HF-hosted summarizers) | +| `prop_decrease` | float | (Optional) Controls level of noise suppression. Range: 0.0–1.0 (default: 0.7) | +| `summarized_model`| string | (Optional) Path or HF model ID for summarizer. Default: `mistralai/Mistral-7B-Instruct-v0.1` | +| `ground_truth` | string | (Optional) Path to reference transcript file to compute WER | + +--- + +## Example cURL Command +```bash +curl -k -N -L -X POST https://.nip.io/transcribe \ + -F "audio_url=" \ + -F "model=turbo" \ + -F "summary=true" \ + -F "speaker=true" \ + -F "streaming=true" \ + -F "denoise=false" \ + -F "hf_token=hf_xxxxxxx" \ + -F "max_speakers=2" +``` + +--- + +## Output Files + +Each processed audio generates the following: + +- `*.txt` – Human-readable transcript with speaker turns and timestamps +- `*.json` – Full structured metadata: transcript, summary, diarization +- `*.log` – Detailed processing log (useful for debugging or auditing) + +--- + +## Streaming Logs + +If `streaming=true`, the response will contain a log filename: +```json +{ + "meta": "logfile_name", + "logfile": "transcription_log_remote_audio_.log" +} +``` +To stream logs in real-time: +```bash +curl -N https://.nip.io/stream_log/ +``` + +--- + +## Hugging Face Access + +To enable diarization, accept model terms at: +https://huggingface.co/pyannote/segmentation + +Generate token at: +https://huggingface.co/settings/tokens + +--- + +## Dependencies + +| Package | Purpose | +|---------------------|----------------------------------| +| `faster-whisper` | Core transcription engine | +| `transformers` | Summarization via Hugging Face | +| `pyannote.audio` | Speaker diarization | +| `pydub`, `librosa` | Audio chunking and processing | +| `demucs` | Vocal separation / denoising | +| `fastapi`, `uvicorn`| REST API server | +| `jiwer` | WER evaluation | + +--- + +## Final Notes + +- Whisper model is GPU-cached per thread for performance. +- Diarization runs globally, not chunk-by-chunk. +- Denoising is optional but improves quality on noisy files. diff --git a/docs/whisper_transcription/docs/Whisper_Architecture.pdf b/docs/whisper_transcription/docs/Whisper_Architecture.pdf new file mode 100644 index 0000000..da32ce1 Binary files /dev/null and b/docs/whisper_transcription/docs/Whisper_Architecture.pdf differ diff --git a/docs/whisper_transcription/requirements.txt b/docs/whisper_transcription/requirements.txt new file mode 100644 index 0000000..2352f28 --- /dev/null +++ b/docs/whisper_transcription/requirements.txt @@ -0,0 +1,37 @@ +# Core frameworks +fastapi==0.115.12 +uvicorn==0.34.2 + +# Whisper + speech processing +faster-whisper==1.1.1 +librosa==0.11.0 +pydub==0.25.1 +soundfile==0.13.1 +noisereduce==3.0.3 +demucs==4.0.1 +ffmpeg-python==0.2.0 + +# Diarization (PyAnnote) +pyannote.audio==3.3.2 +pyannote.core==5.0.0 +pyannote.database==5.1.3 +pyannote.metrics==3.2.1 +pyannote.pipeline==3.0.1 + +# Transformers + Summarization +transformers==4.51.3 +huggingface-hub==0.31.4 +sentencepiece==0.2.0 + +# Evaluation +jiwer==3.1.0 + +# Utilities +numpy==1.26.4 +scikit-learn==1.6.1 +requests==2.32.3 +tqdm==4.67.1 +typing_extensions==4.13.2 +pydantic==2.11.4 +python-multipart==0.0.20 +accelerate==1.7.0 \ No newline at end of file diff --git a/docs/whisper_transcription/whisper_api_server.py b/docs/whisper_transcription/whisper_api_server.py new file mode 100644 index 0000000..dd5a671 --- /dev/null +++ b/docs/whisper_transcription/whisper_api_server.py @@ -0,0 +1,173 @@ +import os +import tempfile +import json +import threading +import glob +from queue import Queue +from threading import Lock +from fastapi import FastAPI, UploadFile, File, Form +from fastapi.responses import StreamingResponse, JSONResponse, FileResponse +from whisper_code import transcribe_and_summarize, load_whisper_model_faster +import logging +import time + +app = FastAPI() + +# Global model cache and lock +model_cache = {} +model_lock = Lock() + +@app.post("/transcribe") +async def transcribe_audio_api( + audio_file: UploadFile = File(...), + model: str = Form("base"), + summarized_model: str = Form("mistralai/Mistral-7B-Instruct-v0.1"), + denoise: bool = Form(False), + prop_decrease: float = Form(0.7), + summary: bool = Form(True), + speaker: bool = Form(False), + hf_token: str = Form(None), + max_speakers: int = Form(None), + streaming: bool = Form(False) +): + + # ✅ Setup dynamic logging per request + from datetime import datetime + basename = os.path.splitext(os.path.basename(audio_file.filename))[0] + timestamp = datetime.now().strftime('%Y%m%d_%H%M%S') + log_file = f"transcription_log_{basename}_{timestamp}.log" + + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(levelname)s - %(message)s", + filename=log_file, + filemode="w", + force=True # ensures logging resets each request + ) + + temp_audio_path = tempfile.mktemp(suffix=f"_{audio_file.filename}") + with open(temp_audio_path, "wb") as f: + f.write(await audio_file.read()) + + output_dir = tempfile.mkdtemp() + + # Ensure model is loaded once + if model not in model_cache: + model_cache[model] = load_whisper_model_faster(model) + + whisper_model = model_cache[model] + + if streaming: + def generator(): + q = Queue() + q.put(f"data: {json.dumps({'meta': 'logfile_name', 'logfile': log_file})}\n\n") + + + def api_callback(result): + if result.get("chunk_id", "").startswith("chunk_"): + logging.info(f"\n====== Streaming {result['chunk_id']} ======\n{result.get('text', '').strip()}\n") + elif result.get("chunk_id") == "summary": + logging.info("\n====== Final Summary ======\n") + logging.info(result.get("summary", "").strip()) + result["logfile"] = log_file + + q.put(f"data: {json.dumps(result)}\n\n") + + def run_pipeline(): + try: + with model_lock: + transcribe_and_summarize( + path=temp_audio_path, + model_name=model, + output_dir=output_dir, + summarized_model_id=summarized_model, + denoise=denoise, + prop_decrease=prop_decrease, + summary=summary, + speaker=speaker, + hf_token=hf_token, + max_speakers=max_speakers, + streaming=True, + api_callback=api_callback, + model_instance=whisper_model + ) + except Exception as e: + import traceback + traceback.print_exc() + q.put(json.dumps({"error": f"Streaming transcription failed: {str(e)}"})) + finally: + q.put(None) + + threading.Thread(target=run_pipeline).start() + + while True: + chunk = q.get() + if chunk is None: + break + yield chunk + + return StreamingResponse(generator(), media_type="text/event-stream") + + else: + try: + with model_lock: + transcribe_and_summarize( + path=temp_audio_path, + model_name=model, + output_dir=output_dir, + summarized_model_id=summarized_model, + denoise=denoise, + prop_decrease=prop_decrease, + summary=summary, + speaker=speaker, + hf_token=hf_token, + max_speakers=max_speakers, + streaming=False, + api_callback=None, + model_instance=whisper_model + ) + except Exception as e: + import traceback + traceback.print_exc() + return JSONResponse( + content={"error": f"Transcription failed: {str(e)}"}, + status_code=500 + ) + + try: + json_files = sorted( + glob.glob(os.path.join(output_dir, "*.json")), + key=os.path.getmtime, + reverse=True + ) + if not json_files: + raise FileNotFoundError("No output JSON file found.") + + with open(json_files[0]) as f: + return JSONResponse(content=json.load(f)) + + except Exception as e: + import traceback + traceback.print_exc() + return JSONResponse( + content={"error": f"Failed to read output JSON: {str(e)}"}, + status_code=500 + ) + +@app.get("/stream_log/{filename}") +def stream_log(filename: str): + log_path = os.path.join(".", filename) + if not os.path.exists(log_path): + return JSONResponse(content={"error": "Log file not found."}, status_code=404) + + def stream_lines(): + with open(log_path, "r") as f: + f.seek(0, 2) # Move to end of file + while True: + line = f.readline() + if line: + yield f"data: {line.strip()}\n\n" + else: + time.sleep(0.2) # Poll every 200ms + + return StreamingResponse(stream_lines(), media_type="text/event-stream") \ No newline at end of file diff --git a/docs/whisper_transcription/whisper_code.py b/docs/whisper_transcription/whisper_code.py new file mode 100644 index 0000000..b3d1108 --- /dev/null +++ b/docs/whisper_transcription/whisper_code.py @@ -0,0 +1,914 @@ +from datetime import datetime +import re +import os +import torch +#import whisper +import tempfile +import wave +import json +import librosa +import noisereduce as nr +import argparse +from pydub import AudioSegment, silence +from transformers import AutoTokenizer, AutoModelForCausalLM +from datetime import datetime +import logging +from collections import OrderedDict +from jiwer import wer +from pyannote.audio import Pipeline +import soundfile as sf +import shutil +import subprocess +from datetime import datetime, timedelta +from collections import Counter +from faster_whisper import WhisperModel +from pydub.utils import make_chunks +from concurrent.futures import ThreadPoolExecutor, as_completed +from difflib import get_close_matches +import threading +thread_local = threading.local() + +SUPPORTED_EXTENSIONS = ('.flac', '.wav', '.mp3', '.m4a', '.aac', '.ogg', '.webm', '.opus', '.mp4', '.mov', '.mkv', '.avi') +CHUNK_FOLDER = "chunks" + +# Detects GPU type and suggests best Whisper model and token limit accordingly +def detect_gpu_and_optimize(): + """ + Detect the current GPU and return Whisper model + token limit. + Returns: (gpu_name, whisper_model, token_limit) + """ + if not torch.cuda.is_available(): + return "cpu", "small", 30 + device = torch.cuda.current_device() + name = torch.cuda.get_device_name(device).lower() + if "a10" in name: + return "a10", "medium", 60 + elif "a100" in name: + return "a100", "large", 120 + elif "h100" in name: + return "h100", "turbo", 180 + return "unknown_gpu", "base", 60 + +def load_whisper_model_faster(model_name, gpu_id=0): + """ + Loads a WhisperModel from faster-whisper for a specified GPU. + + Args: + model_name (str): Name of the Whisper model to load. + gpu_id (int): The GPU index to load the model on. + + Returns: + WhisperModel: An instance of the loaded WhisperModel ready for transcription. + """ + logging.info(f"[load_whisper_model_faster] Loading WhisperModel '{model_name}' on GPU {gpu_id}") + return WhisperModel( + model_name, + device="cuda", + device_index=gpu_id, + compute_type="float16", + cpu_threads=8 + ) + + +# Converts input audio/video file to 16kHz, mono WAV format (Whisper compatible) +def convert_to_wav(input_path): + """ + Converts any audio/video file into 16kHz, mono-channel WAV format using ffmpeg. + + Args: + input_path (str): Path to the input file. + + Returns: + str: Path to the converted WAV file or None on failure. + """ + output_path = tempfile.mktemp(suffix=".wav") + try: + cmd = [ + "ffmpeg", "-y", "-threads", "4", + "-i", input_path, + "-ac", "1", + "-ar", "16000", + "-sample_fmt", "s16", + output_path + ] + subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=True) + except Exception as e: + logging.error(f"Chunk processing failed: {e}") + return None + logging.error(f"ffmpeg conversion failed on {input_path}: {e}") + raise + return output_path + + +def assign_speakers_to_segments_from_global(chunk_path, segments, diarized_segments): + """ + Assign speakers using previously diarized segments. + + Args: + chunk_path (str): Path to current chunk. + segments (list): List of Whisper segments. + diarized_segments (list): List of diarized (start, end, speaker) triples. + + Returns: + list[str]: Speaker label per segment. + """ + + assigned_speakers = [] + for seg in segments: + midpoint = (seg['start'] + seg['end']) / 2 + match = None + for track, _, label in diarized_segments: + normalized_label = str(label).strip().upper() # Normalize case and spacing + if track.start <= midpoint <= track.end: + match = normalized_label + break + + if not match: + match = "Speaker_0" + assigned_speakers.append(match) + logging.info(f"Speaker assignment breakdown: {Counter(assigned_speakers)}") + return assigned_speakers + + +# Applies noise reduction using noisereduce (conservative settings) +def denoise_audio(input_file, output_file, prop_decrease=0.7): + """ + Applies denoising using Demucs and noise reduction techniques. + + Args: + input_file (str): Path to the input noisy audio. + output_file (str): Output path for the denoised audio. + prop_decrease (float): Aggressiveness of denoising. + + Returns: + None + """ + temp_dir = tempfile.mkdtemp() + try: + logging.info(f"Running Demucs on: {input_file}") + demucs_cmd = f"python3 -m demucs.separate -d cuda -n htdemucs --two-stems=vocals -o \"{temp_dir}\" \"{input_file}\"" + subprocess.run(demucs_cmd, shell=True, check=True) + + base = os.path.splitext(os.path.basename(input_file))[0] + vocals_path = os.path.join(temp_dir, "htdemucs", base, "vocals.wav") + + if not os.path.exists(vocals_path): + raise FileNotFoundError(f"Demucs output not found: {vocals_path}") + + y, sr = librosa.load(vocals_path, sr=None) + y_denoised = nr.reduce_noise(y=y, sr=sr, prop_decrease=prop_decrease, stationary=False) + sf.write(output_file, y_denoised, sr) + logging.info(f"Denoised file saved to: {output_file}") + except Exception as e: + logging.error(f"Denoising failed on {input_file}: {e}. Falling back to original.") + shutil.copy(input_file, output_file) # FALLBACK + finally: + shutil.rmtree(temp_dir, ignore_errors=True) + + + +# Calculates duration (in seconds) of the audio file using wave or pydub +def get_audio_duration(path): + """ + Calculates duration of a WAV or general audio file. + + Args: + path (str): File path to audio. + + Returns: + float: Duration in seconds. + """ + try: + if path.endswith('.wav'): + with wave.open(path, 'r') as f: + frames = f.getnframes() + rate = f.getframerate() + return frames / float(rate) + else: + audio = AudioSegment.from_file(path) + return len(audio) / 1000.0 # seconds + except Exception as e: + logging.error(f"Chunk processing failed: {e}") + return None + logging.warning(f"Failed to get duration of {path}: {e}") + return 0 + + +def fallback_fixed_chunks(audio, chunk_length_ms=30000): + """ + Splits audio into fixed-length chunks (30s default). + + Args: + audio (AudioSegment): Loaded audio object. + chunk_length_ms (int): Length in milliseconds. + + Returns: + list[AudioSegment]: List of chunks. + """ + return make_chunks(audio, chunk_length_ms) + +def smart_chunk_audio(input_path, min_silence_len=1000, silence_thresh=-40, chunk_padding=1000): + """ + Splits audio based on silence; falls back to fixed chunks. + + Args: + input_path (str): Path to audio file. + min_silence_len (int): Minimum silence to consider for split. + silence_thresh (int): Silence threshold in dB. + chunk_padding (int): Padding around detected speech chunks. + + Returns: + list[str]: List of chunked audio paths. + """ + audio = AudioSegment.from_file(input_path) + os.makedirs(CHUNK_FOLDER, exist_ok=True) + logging.info(f"Smart chunking using silence detection on: {input_path}") + chunks = silence.split_on_silence(audio, min_silence_len=min_silence_len, silence_thresh=silence_thresh, keep_silence=chunk_padding) + if not chunks or any(len(chunk) > 90000 for chunk in chunks): # fallback if no chunks or too long + logging.warning("Fallback to fixed chunking due to silence issues or long chunks.") + chunks = fallback_fixed_chunks(audio) + chunk_paths = [] + for idx, chunk in enumerate(chunks): + chunk_path = os.path.join(CHUNK_FOLDER, f"chunk_{idx:03d}.wav") + chunk.export(chunk_path, format="wav") + chunk_paths.append(chunk_path) + logging.info(f"Exported chunk: {chunk_path}") + return chunk_paths + +# Converts and transcribes an audio file using Whisper model +def transcribe_file(model, audio_path): + """ + Transcribes a WAV file using a Whisper model. + + Args: + model (WhisperModel): Loaded Whisper model. + audio_path (str): Path to WAV file. + beam_size (int): Beam size for decoding. + language (str): Optional forced language. + + Returns: + tuple[list[dict], str]: List of segments and detected language. + """ + try: + wav_path = convert_to_wav(audio_path) + + segments_generator, info = model.transcribe(wav_path, beam_size=1, language=None) + segments = [] + text = "" + for seg in segments_generator: + segments.append({ + "start": round(seg.start, 2), + "end": round(seg.end, 2), + "text": seg.text.strip() + }) + text += seg.text.strip() + " " + + os.remove(wav_path) + + detected_lang = info.language + return text.strip(), detected_lang, segments + except Exception as e: + logging.error(f"Chunk processing failed: {e}") + return None + logging.error(f"Failed: {audio_path} — {e}") + return "", "unknown", [] + + +# Loads the summarization model (default: Mistral) from HuggingFace +def load_mistral_model(model_id, hf_token): + """ + Loads a HuggingFace LLM model and tokenizer. + + Args: + model_name (str): Name of model. + + Returns: + tuple: (model, tokenizer) + """ + if os.path.isdir(model_id): + logging.info(f"Loading summarizer model from local path: {model_id}") + tokenizer = AutoTokenizer.from_pretrained(model_id, use_auth_token=hf_token) + model = AutoModelForCausalLM.from_pretrained( + model_id, + torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32, + device_map="auto", + use_auth_token=hf_token + ) + else: + logging.info(f"Loading summarizer model from HuggingFace: {model_id}") + tokenizer = AutoTokenizer.from_pretrained(model_id, use_auth_token=hf_token) + model = AutoModelForCausalLM.from_pretrained( + model_id, + torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32, + device_map="auto", + use_auth_token=hf_token + ) + return model, tokenizer + + +# Splits long transcript into smaller LLM-compatible chunks based on token count +def split_into_chunks(text, tokenizer, max_tokens=3000): + """ + Splits long text into overlapping token-aware chunks. + + Args: + text (str): Input string. + tokenizer: HuggingFace tokenizer. + max_tokens (int): Max tokens per chunk. + stride (int): Overlap for continuity. + + Returns: + list[str]: List of text chunks. + """ + words = text.split() + chunks = [] + current = [] + for word in words: + current.append(word) + if len(tokenizer.encode(" ".join(current))) > max_tokens: + chunks.append(" ".join(current[:-1])) + current = [word] + if current: + chunks.append(" ".join(current)) + return chunks + + +# Summarizes a chunk of transcript using the loaded summarizer model +def summarize_with_mistral(model, tokenizer, text): + """ + Summarizes text using an LLM like Mistral. + + Args: + model: HuggingFace model. + tokenizer: Corresponding tokenizer. + text (str): Text to summarize. + + Returns: + str: Summary string. + """ + prompt = f"[INST] Summarize the following meeting transcript into a concise list of key points, decisions, and action items:\n\n{text} [/INST]" + inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=4096).to(model.device) + with torch.no_grad(): + output = model.generate(**inputs, max_new_tokens=512) + + result = tokenizer.decode(output[0], skip_special_tokens=True) + + # Strip echoed prompt (everything before [/INST]) + if "[/INST]" in result: + result = result.split("[/INST]", 1)[-1].strip() + + return result + + +# Main pipeline for one file: validate → denoise → chunk → transcribe → summarize → save outputs +def assign_speakers_to_segments(full_audio_path, segments, hf_token, max_speakers=None): + """ + Performs diarization and aligns speakers with transcript. + + Args: + full_audio_path (str): Path to audio. + segments (list): List of Whisper segments. + hf_token (str): HuggingFace token. + max_speakers (int): Optional speaker limit. + + Returns: + list[str]: Speaker label per segment. + """ + + logging.info(f"Running speaker diarization on: {full_audio_path}") + try: + pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1", use_auth_token=hf_token) + if torch.cuda.is_available(): + pipeline.to(torch.device("cuda")) + # Run diarization once per full audio (global instead of per chunk) + diarization = pipeline({"audio": converted_path}) + global_diarized_segments = list(diarization.itertracks(yield_label=True)) if diarization else [] + logging.info(f"Global diarization complete. {len(global_diarized_segments)} segments.") + except Exception as e: + logging.error(f"Chunk processing failed: {e}") + return None + logging.error(f"Could not load pyannote pipeline: {e}") + return ["Speaker 1"] * len(segments) + + try: + if max_speakers: + diarization = pipeline({"audio": full_audio_path}, num_speakers=max_speakers) + else: + diarization = pipeline(full_audio_path) + except Exception as e: + logging.error(f"Chunk processing failed: {e}") + return None + logging.error(f"Diarization pipeline failed: {e}") + return ["Speaker 1"] * len(segments) + + diarized_segments = list(diarization.itertracks(yield_label=True)) + if not diarized_segments: + logging.warning("No diarized speaker segments were found.") + return ["Speaker 1"] * len(segments) + + logging.info(f"Total diarized segments: {len(diarized_segments)}") + for turn, _, speaker in diarized_segments: + logging.info(f"Diarized: {speaker} | {turn.start:.2f} - {turn.end:.2f}") + + speaker_map = [] + for segment in segments: + mid_point = (segment["start"] + segment["end"]) / 2.0 + assigned_speaker = "Speaker 1" + for turn, _, speaker in diarized_segments: + if turn.start <= mid_point <= turn.end: + assigned_speaker = speaker + break + speaker_map.append(assigned_speaker) + logging.info(f"Segment midpoint={mid_point:.2f} assigned to: {assigned_speaker} — {segment['text'][:30]}") + + logging.info(f"Speaker assignment breakdown: {Counter(speaker_map)}") + return speaker_map + +def transcribe_and_summarize(path, model_name="base", output_dir="outputs", summarized_model_id="mistralai/Mistral-7B-Instruct-v0.1", ground_truth_path=None, denoise=False, prop_decrease=0.7, summary=True, speaker=False, hf_token=None, session_timestamp=None, max_speakers=None, streaming=False, api_callback=None, model_instance=None): + """ + End-to-end pipeline: denoise, chunk, transcribe, diarize, summarize, and save. + + Args: + path (str): Path to input audio file. + model_name (str): Whisper model name. + output_dir (str): Directory to save outputs. + denoise (bool): Whether to apply denoising. + summary (bool): Whether to generate summaries. + speaker (bool): Whether to run speaker diarization. + streaming (bool): Whether to stream results. + hf_token (str): HuggingFace token. + max_speakers (int): Max speakers for diarization. + + Returns: + None + """ + os.makedirs(output_dir, exist_ok=True) + + if not isinstance(session_timestamp, str) or not re.match(r"^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}$", session_timestamp): + session_timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + base_time = datetime.strptime(session_timestamp, "%Y-%m-%d %H:%M:%S") + + + if denoise: + denoised_path = tempfile.mktemp(suffix=".wav") + logging.info(f"Applying noise reduction to: {path}") + denoise_audio(path, denoised_path, prop_decrease=prop_decrease) + path = denoised_path + + converted_path = convert_to_wav(path) + if not converted_path or not os.path.exists(converted_path): + logging.error("Conversion failed — aborting transcription.") + return + + + global_diarized_segments = [] + if speaker: + try: + logging.info("Running global speaker diarization...") + pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization", use_auth_token=hf_token) + if torch.cuda.is_available(): + pipeline.to(torch.device("cuda")) + diarization = pipeline({"audio": converted_path}) + global_diarized_segments = list(diarization.itertracks(yield_label=True)) if diarization else [] + logging.info(f"Global diarization complete. {len(global_diarized_segments)} segments.") + except Exception as e: + logging.warning(f"Global diarization failed on {converted_path}: {e}") + global_diarized_segments = [] + + chunk_paths = smart_chunk_audio(converted_path) + base = os.path.splitext(os.path.basename(path))[0] + structured_chunks = {} + temp_transcript_path = os.path.join(output_dir, f"{base}_partial_transcript.txt") + if os.path.exists(temp_transcript_path): + os.remove(temp_transcript_path) + logging.info(f"Total chunks: {len(chunk_paths)}") + + num_gpus = torch.cuda.device_count() + if num_gpus == 0: + raise RuntimeError("No CUDA devices available") + gpus = list(range(num_gpus)) + if model_instance is not None: + whisper_models = {0: model_instance} + else: + whisper_models = { + gpu_id: WhisperModel( + model_name, + device="cuda", + device_index=gpu_id, + compute_type="float16", + cpu_threads=4 + ) for gpu_id in gpus + + } + + + failed_chunks = [] + + def clean_audio_with_ffmpeg(input_path, cleaned_path): + """ + Cleans audio using ffmpeg (mono, 16-bit PCM, 16kHz). + + Args: + input_path (str): Path to input audio. + cleaned_path (str): Output path for cleaned audio. + + Returns: + None + """ + try: + cmd = [ + "ffmpeg", "-y", "-i", input_path, + "-ar", "16000", "-ac", "1", "-c:a", "pcm_s16le", + cleaned_path + ] + subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + return cleaned_path + except subprocess.CalledProcessError: + logging.error(f"FFmpeg failed to clean {input_path}") + return None + + + + def transcribe_on_gpu(gpu_id, chunk_path, global_diarized_segments, model_name): + """ + Transcribes a single chunk using a specified GPU, optionally performing speaker diarization. + + Args: + gpu_id (int): GPU index to use for transcription. + chunk_path (str): Path to audio chunk file. + global_diarized_segments (list): List of diarized segments (start, end, label). + model_name (str): Name of Whisper model to load. + + Returns: + dict: Dictionary containing transcription, chunk path, and speaker labels. + """ + try: + torch.cuda.set_device(gpu_id) + + # Lazily initialize model per thread + if not hasattr(thread_local, "model"): + thread_local.model = WhisperModel( + model_name, + device="cuda", + device_index=gpu_id, + compute_type="float16" + ) + logging.info(f"WhisperModel '{model_name}' loaded on GPU {gpu_id} for thread {threading.get_ident()}") + + model = thread_local.model + + logging.debug(f"Processing chunk: {chunk_path}") + duration = get_audio_duration(chunk_path) + if not duration or duration < 1.0: + logging.warning(f"Skipping chunk {chunk_path}: invalid or too short ({duration}s)") + return None + + result = transcribe_file(model, chunk_path) + + if not result or not isinstance(result, tuple) or len(result) != 3: + raise ValueError("Invalid result returned from transcribe_file()") + + chunk_text, lang, segments = result + + # Speaker assignment + if speaker: + try: + speakers = assign_speakers_to_segments_from_global(chunk_path, segments, global_diarized_segments) + except Exception as e: + logging.warning(f"Diarization failed for {chunk_path}: {e}") + speakers = ["Speaker 1"] * len(segments) + else: + speakers = ["Speaker 1"] * len(segments) + + for i, s in enumerate(segments): + s["speaker"] = speakers[i] if i < len(speakers) else "Speaker 1" + + # Format output + text_with_speakers = "" + for seg in segments: + raw_speaker = str(seg.get("speaker", "Speaker 1")) + if raw_speaker.lower().startswith("speaker_"): + speaker_number = int(raw_speaker.split("_")[1]) + speaker_label = f"Speaker {speaker_number + 1}" + else: + speaker_label = raw_speaker + + timestamp_full = ( + base_time + timedelta(seconds=seg["start"]) + ).strftime("%Y-%m-%d %H:%M:%S") + + text_with_speakers += f"[{timestamp_full}] {speaker_label}: {seg['text'].strip()}\n" + + chunk_filename = os.path.basename(chunk_path) + return chunk_filename, text_with_speakers, lang, segments + + except Exception as e: + logging.error(f"Chunk processing failed on {chunk_path}: {e}") + failed_chunks.append(chunk_path) + return None + + + + if speaker: + try: + speakers = assign_speakers_to_segments_from_global(chunk_path, segments, global_diarized_segments) + except Exception as e: + logging.warning(f"Diarization failed for chunk file {chunk_path}: {e}") + speakers = ["Speaker 1"] * len(segments) + else: + speakers = ["Speaker 1"] * len(segments) + + for i, s in enumerate(segments): + s["speaker"] = speakers[i] if i < len(speakers) else "Speaker 1" + + text_with_speakers = "" + for seg in segments: + raw_speaker = str(seg.get("speaker", "Speaker 1")) + if raw_speaker.lower().startswith("speaker_"): + speaker_number = int(raw_speaker.split("_")[1]) + speaker_label = f"Speaker {speaker_number + 1}" + else: + speaker_label = raw_speaker + + timestamp_full = ( + base_time + timedelta(seconds=seg["start"]) + ).strftime("%Y-%m-%d %H:%M:%S") + text_with_speakers += f"[{timestamp_full}] {speaker_label}: {seg['text'].strip()}\n" + + chunk_filename = os.path.basename(chunk_path) + return chunk_filename, text_with_speakers, lang, segments + + except Exception as e: + logging.error(f"Chunk processing failed: {e}") + failed_chunks.append(chunk_path) + return None + + + + structured_chunks = OrderedDict() + full_transcript = "" + + with ThreadPoolExecutor(max_workers=len(gpus)) as executor: + future_to_idx = {} + for i, chunk in enumerate(chunk_paths): + gpu_id = gpus[i % len(gpus)] + future = executor.submit(transcribe_on_gpu, gpu_id, chunk, global_diarized_segments, model_name) + future_to_idx[future] = (i, chunk) + + for future in as_completed(future_to_idx): + idx, chunk = future_to_idx[future] + result = future.result() + if result is None: + continue + + chunk_filename, text_with_speakers, lang, segments = result + chunk_id = f"{idx+1:03d}" + + # this is new — gather full transcript across all chunks + full_transcript += text_with_speakers.strip() + "\n" + + structured_chunks[chunk_filename] = { + "chunk_id": chunk_id, + "transcript": text_with_speakers.strip(), + "segments": [ + { + "text": seg["text"].strip(), + "timestamp": ( + datetime.strptime(session_timestamp, "%Y-%m-%d %H:%M:%S") + timedelta(seconds=seg["start"]) + ).strftime("%Y-%m-%d %H:%M:%S"), + "speaker": seg["speaker"] + } + for seg in segments + ], + "language": lang + } + + if streaming: + logging.info(f"\n====== Streaming Chunk {chunk_id} ======\n{text_with_speakers.strip()}\n") + + + # Save transcript to temp file + try: + with open(temp_transcript_path, "w") as f: + f.write(full_transcript.strip()) + except Exception as e: + logging.error(f"Failed to write temp transcript: {e}") + return + + # Ensure temp transcript file exists and re-read + if not os.path.exists(temp_transcript_path): + logging.error(f"Transcript file not found: {temp_transcript_path}") + return + + with open(temp_transcript_path, "r") as tf: + full_transcript = tf.read() + + if not full_transcript.strip(): + logging.warning(" Skipping save: empty transcript detected.") + return + + # Define final output paths + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + txt_path = os.path.join(output_dir, f"{base}_all_transcripts_{timestamp}.txt") + json_path = os.path.join(output_dir, f"{base}_all_transcripts_{timestamp}.json") + + # Write transcript to final TXT file + with open(txt_path, "w") as f: + f.write(full_transcript) + logging.info(f"Transcript saved to: {txt_path}") + + try: + logging.info(f"CUDA available before summarization: {torch.cuda.is_available()}") + model_llm, tokenizer = load_mistral_model(summarized_model_id, hf_token=hf_token) + except Exception as e: + logging.error(f"Failed to load summarization model: {e}") + model_llm = tokenizer = None + + # Generate summary if requested + if not model_llm: + combined_summary = "(Summarization failed due to model load issue.)" + elif len(full_transcript.split()) < 50: + logging.info("Skipping summarization: transcript too short.") + combined_summary = "(Transcript too short for meaningful summarization.)" + else: + summary_chunks = split_into_chunks(full_transcript, tokenizer) + summaries = [] + for i, chunk in enumerate(summary_chunks): + logging.info(f"Summarizing chunk {i+1}/{len(summary_chunks)}") + summary = summarize_with_mistral(model_llm, tokenizer, chunk) + summary = summary.replace(chunk.strip(), '').strip() + summaries.append(summary) + combined_summary = "\n\n---\n\n".join(summaries) + # Safeguard against None summary + if not isinstance(combined_summary, str) or not combined_summary.strip(): + combined_summary = "(Summary generation failed or was skipped.)" + + # Write summary to TXT file + try: + with open(txt_path, "a") as f: + f.write("\n\n====== Summary ======\n\n") + f.write(combined_summary.strip()) + logging.info(" Summary successfully appended to transcript TXT.") + except Exception as e: + logging.error(f" Failed to write summary to TXT: {e}") + + # Write final JSON + try: + ordered_output = OrderedDict() + ordered_output["chunks"] = structured_chunks + ordered_output["transcript"] = full_transcript.strip() + ordered_output["summary"] = combined_summary.strip() + with open(json_path, "w") as f: + json.dump(ordered_output, f, indent=2) + logging.info(f"Final JSON saved to: {json_path}") + except Exception as e: + logging.error(f" Failed to save JSON: {e}") + + if streaming and api_callback: + logging.info(f"\n====== Streaming Chunk {chunk_id} ======\n{text_with_speakers.strip()}\n") + api_callback({ + "chunk_id": "summary", + "transcript": full_transcript.strip(), + "summary": combined_summary.strip(), + }) + + if ground_truth_path and os.path.isfile(ground_truth_path): + with open(ground_truth_path, 'r') as gt: + reference = gt.read().strip() + error = wer(reference, full_transcript.strip()) + logging.info(f"WER (Word Error Rate) against ground truth: {error:.2%}") + elif ground_truth_path: + logging.warning(f"Ground truth file not found at: {ground_truth_path}") + + if failed_chunks: + failed_path = os.path.join(output_dir, f"{base}_failed_chunks.txt") + with open(failed_path, "w") as f: + for path in failed_chunks: + f.write(path + "\n") + logging.warning(f"{len(failed_chunks)} chunks failed. Paths saved to: {failed_path}") + + + + +local_session_timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + +# Handles single file or directory input, validates each audio, and runs full pipeline +def process_audio_batch(input_path, model_name, output_dir, summarized_model_id, ground_truth_path=None, denoise=False, prop_decrease=0.7, summary=True, speaker=False, hf_token=None, max_speakers=None, streaming=False, api_callback=None): + """ + Processes a file or a folder of audio through the full pipeline. + + Args: + input_path (str): File or directory path. + model (str): Whisper model name. + output_dir (str): Where to write output. + summary (bool): Enable summarization. + speaker (bool): Enable speaker diarization. + denoise (bool): Enable denoising. + streaming (bool): Enable chunk-wise streaming. + hf_token (str): HuggingFace token. + max_speakers (int): Maximum number of speakers. + + Returns: + None + """ + session_timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + timestamp = datetime.now().strftime('%Y%m%d_%H%M%S') + + if not output_dir: + output_dir = os.path.join("outputs", f"run_{timestamp}") + os.makedirs(output_dir, exist_ok=True) + + # Initialize logging + log_file = os.path.join(output_dir, f"transcription_log_{timestamp}.log") + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(levelname)s - %(message)s", + handlers=[ + logging.FileHandler(log_file), + logging.StreamHandler() + ] + ) + + # Get list of files + if os.path.isdir(input_path): + audio_files = [ + os.path.join(dp, f) + for dp, dn, filenames in os.walk(input_path) + for f in filenames if f.endswith(SUPPORTED_EXTENSIONS) + ] + else: + audio_files = [input_path] if input_path.endswith(SUPPORTED_EXTENSIONS) else [] + + if not audio_files: + logging.warning(f"No valid audio files found in input: {input_path}") + return + + logging.info(f"Total audio files to process: {len(audio_files)}") + + for audio_file in audio_files: + logging.info(f"Validating: {audio_file}") + if not os.path.exists(audio_file): + logging.warning(f"File does not exist: {audio_file}") + continue + + duration = get_audio_duration(audio_file) + logging.info(f"Duration of {audio_file}: {duration:.2f}s") + if duration < 1: + logging.warning(f"Skipping {audio_file}: too short (<1s)") + continue + elif duration > 3600: + logging.warning(f"{audio_file} is too long (>60 min), may require chunking") + + logging.info(f"Processing: {audio_file}") + temp_transcript_path = os.path.join(output_dir, f"{os.path.splitext(os.path.basename(audio_file))[0]}_partial_transcript.txt") + try: + transcribe_and_summarize( + audio_file, + model_name, + output_dir, + summarized_model_id, + model_instance=None, + ground_truth_path=ground_truth_path, + denoise=denoise, + prop_decrease=prop_decrease, + summary=summary, + speaker=speaker, + hf_token=hf_token, + session_timestamp=session_timestamp, + max_speakers=max_speakers, + streaming=streaming, + api_callback=api_callback + ) + except Exception as e: + logging.error(f"Failed on {audio_file}: {e}") + continue # or pass if this is not inside a loop + + +# Script entry: parse CLI args and run processing pipeline +if __name__ == "__main__": + # Entry point: parse args and run batch + parser = argparse.ArgumentParser() + parser.add_argument("--input", required=True, help="Path to audio file or directory") + parser.add_argument("--model", help="Override model selection", default=None) + parser.add_argument("--output-dir", help="Directory to store outputs", default=None) + parser.add_argument("--summarized-model", help="HuggingFace model for summarization or provide the path of you own model", default="mistralai/Mistral-7B-Instruct-v0.1") + parser.add_argument("--ground-truth", help="Path to ground truth transcript file for WER evaluation", default=None) + parser.add_argument("--denoise", action="store_true", help="Apply noise reduction to audio before transcription") + parser.add_argument("--prop-decrease", type=float, default=0.7, help="Noise reduction aggressiveness (0.0 to 1.0)") + parser.add_argument("--summary", action="store_true", help="Include summarization after transcription") + parser.add_argument("--speaker", action="store_true", help="Enable speaker diarization using pyannote.audio") + parser.add_argument("--hf-token", help="Hugging Face token for diarization", default=None) + parser.add_argument("--max-speakers", type=int, help="Maximum number of speakers to force diarization into", default=None) + parser.add_argument("--streaming", action="store_true", help="Stream transcript results chunk by chunk") + + + args = parser.parse_args() + + gpu, suggested_model, _ = detect_gpu_and_optimize() + model_name = args.model if args.model else suggested_model + print(f"GPU Detected: {gpu.upper()} — Using model: {model_name}") + process_audio_batch( + args.input, model_name, args.output_dir, args.summarized_model, + args.ground_truth, args.denoise, args.prop_decrease, args.summary, + args.speaker, args.hf_token , args.max_speakers ,args.streaming, api_callback=None +) \ No newline at end of file