Skip to content

codiphile/EdgeVision

BranchesTags

Repository files navigation

EdgeVision - Real-Time Edge Detection Viewer

A minimal Android application that captures camera frames, processes them using OpenCV in C++ (via JNI), displays the processed output using OpenGL ES, and includes a TypeScript-based web viewer for real-time frame display.

Table of Contents

Tech Stack

Core Requirements

  • Android SDK - Kotlin
  • NDK (Native Development Kit) - C++17
  • OpenGL ES 2.0+ - Hardware-accelerated rendering
  • OpenCV 4.12.0 - C++ computer vision library
  • JNI - Java to C++ communication bridge
  • TypeScript - Web-based frame viewer
  • CMake 3.22.1 - Native build system

Additional Technologies

  • Camera2 API - Advanced camera control
  • Jetpack Compose - Modern Android UI
  • WebSocket (Java-WebSocket 1.5.4) - Real-time frame streaming
  • GLSL ES 2.0 - OpenGL shaders

Features Implemented

Camera Feed Integration

  • Camera2 API with TextureView and SurfaceTexture
  • Real-time frame streaming at 30 FPS
  • Frame buffer queue with automatic overflow handling

OpenCV C++ Processing

  • JNI Bridge for Java to C++ communication
  • Canny Edge Detection algorithm
  • Grayscale filter mode
  • YUV to Grayscale conversion
  • Gaussian blur preprocessing
  • Efficient buffer reuse for memory optimization

OpenGL ES Rendering

  • OpenGL ES 2.0 texture rendering
  • Custom GLSL vertex and fragment shaders
  • Hardware-accelerated rendering
  • 10-15 FPS real-time performance
  • Dynamic texture updates
  • Portrait orientation support

TypeScript Web Viewer

  • TypeScript + HTML5 Canvas viewer
  • Real-time WebSocket connection to Android
  • Live frame streaming with base64 decoding
  • Frame statistics overlay (resolution, FPS, processing mode, size, timestamp)
  • Connection management with auto-reconnect
  • IP validation and localStorage persistence
  • Modern gradient UI design

Additional Features

  • Toggle between Canny edge detection and grayscale modes
  • Real-time FPS counter
  • Frame capture to device gallery
  • Full WebSocket server implementation
  • Network utilities for automatic IP detection

Project Structure

EdgeVision/
├── app/                              # Android Application
│   ├── src/main/
│   │   ├── java/com/example/edgevision/
│   │   │   ├── camera/              # Camera2 API integration
│   │   │   │   ├── CameraController.kt
│   │   │   │   ├── CameraCaptureManager.kt
│   │   │   │   ├── FrameReader.kt
│   │   │   │   ├── FrameBufferQueue.kt
│   │   │   │   └── PreviewSurface.kt
│   │   │   ├── gl/                  # OpenGL ES renderer
│   │   │   │   └── EdgeVisionRenderer.kt
│   │   │   ├── native/              # JNI interface
│   │   │   │   └── NativeProcessor.kt
│   │   │   ├── websocket/           # WebSocket server
│   │   │   │   ├── FrameWebSocketServer.kt
│   │   │   │   ├── FrameMessage.kt
│   │   │   │   └── WebSocketManager.kt
│   │   │   ├── utils/               # Utilities
│   │   │   │   └── NetworkUtils.kt
│   │   │   └── MainActivity.kt      # Main UI
│   │   ├── cpp/                     # Native C++ OpenCV processing
│   │   │   ├── edge_processor.cpp   # Canny edge detection
│   │   │   ├── frame_converter.cpp  # YUV to grayscale
│   │   │   ├── native-lib.cpp       # JNI entry points
│   │   │   └── CMakeLists.txt       # CMake build config
│   │   └── res/                     # Android resources
│   └── build.gradle.kts
├── web/                             # TypeScript Web Viewer
│   ├── src/
│   │   ├── index.ts                 # Entry point
│   │   ├── viewer.ts                # Frame viewer class
│   │   └── websocket.ts             # WebSocket client
│   ├── dist/                        # Compiled JavaScript
│   ├── index.html                   # Web UI
│   ├── styles.css                   # Styling
│   ├── package.json                 # NPM config
│   └── tsconfig.json                # TypeScript config
├── OpenCV/                          # OpenCV Android SDK
├── README.md                        # This file
└── build.gradle.kts

Modular Architecture

  • /app - Java/Kotlin code (camera access, UI setup)
  • /cpp - C++ OpenCV processing (all CV logic in native)
  • /gl - OpenGL renderer classes
  • /web - TypeScript web viewer (clean, modular, buildable via tsc)

Setup Instructions

Prerequisites

  • Android Studio Ladybug or later
  • Android SDK API 24+ (Android 7.0+)
  • NDK r25c or later
  • CMake 3.22.1+
  • Node.js 18+ (for TypeScript web viewer)
  • Git for version control

1. Clone the Repository

git clone <repository-url>
cd EdgeVision

2. Setup OpenCV Android SDK

  1. Download OpenCV Android SDK from: https://opencv.org/releases/

    • Version: 4.12.0 or later
    • File: opencv-4.12.0-android-sdk.zip

  2. Extract and copy the sdk folder:

    unzip opencv-4.12.0-android-sdk.zip
cp -r OpenCV-android-sdk/sdk ./OpenCV

  3. Verify the OpenCV directory exists at: EdgeVision/OpenCV/

3. Configure NDK

Install NDK via Android Studio:

  • Tools → SDK Manager → SDK Tools → NDK (Side by side)

4. Build the Android App

# Clean and build
./gradlew clean build

# Or directly install to device
./gradlew installDebug

5. Run on Android Device

Option 1: Using Gradle

./gradlew installDebug

Option 2: Using Android Studio

  1. Open project in Android Studio
  2. Connect Android device with USB debugging enabled
  3. Run → Run 'app'

6. Setup TypeScript Web Viewer

cd web
npm install
npm run build

Serve the web viewer:

# Using Python HTTP server
python3 -m http.server 8080

# Then open http://localhost:8080

7. Connect Web Viewer to Android

  1. Open EdgeVision app on Android device
  2. Ensure device is connected to WiFi
  3. Tap "START WS" button
  4. Note the WebSocket URL displayed (e.g., ws://192.168.1.100:8888)
  5. Open web viewer at http://localhost:8080
  6. Enter Android device IP (e.g., 192.168.1.100)
  7. Port defaults to 8888
  8. Click CONNECT
  9. Watch real-time frames stream from Android

Requirements:

  • Both devices on same WiFi network
  • WebSocket server running on Android (START WS pressed)
  • Valid IP address (shown in Android app)

Finding Android Device IP

Method 1: Check EdgeVision app (displayed when server starts)

Method 2: Android Settings

Settings → Network & Internet → WiFi → [Your Network] → Advanced → IP address

Method 3: ADB

adb shell ip addr show wlan0

Architecture

High-Level Architecture

EdgeVision implements a multi-layered architecture with clear separation of concerns:

High-Level Architecture

Data Flow Pipeline

Data Flow Pipeline

JNI Bridge

Java Side (NativeProcessor.kt):

external fun processFrameCanny(
    data: ByteArray,
    width: Int,
    height: Int
): ByteArray?

external fun processFrameGrayscale(
    data: ByteArray,
    width: Int,
    height: Int
): ByteArray?

C++ Side (native-lib.cpp):

JNIEXPORT jbyteArray JNICALL
Java_com_example_edgevision_native_NativeProcessor_processFrameCanny(
    JNIEnv* env,
    jobject /* this */,
    jbyteArray data,
    jint width,
    jint height
) {
    // Convert jbyteArray to cv::Mat
    // Apply Canny edge detection
    // Return processed cv::Mat as jbyteArray
}

WebSocket Communication

The app includes a full WebSocket server that streams processed frames to web clients in real-time.

Message Format:

{
  "timestamp": "2025-10-08T13:45:23.123Z",
  "width": 1088,
  "height": 1088,
  "format": "Grayscale",
  "processingMode": "Canny Edge Detection",
  "fps": 12.5,
  "frameData": "<base64-encoded-image-data>",
  "frameSize": 1183744
}

Server Features:

  • Listen on port 8888
  • Broadcast frames at ~10 FPS (throttled for network efficiency)
  • Send frames in JSON format with base64-encoded image data
  • Display connected client count in real-time
  • Handle multiple simultaneous connections

Frame Flow Timing

Stage Time Notes
Camera Capture ~33ms 30 FPS input
JNI Transfer <1ms Direct ByteBuffer, zero-copy
OpenCV Processing 30-50ms Canny: ~50ms, Grayscale: ~30ms
OpenGL Rendering ~16ms 60 FPS capable, vsync limited
Total Pipeline ~80-100ms 10-12 FPS output

Demo

EdgeVision Demo

Real-time edge detection with WebSocket streaming from Android to web viewer.

Performance

Achieved Performance Metrics

Metric Target Achieved
Rendering FPS 10-15 FPS 12-15 FPS
Frame Processing Smooth 30-50ms/frame
WebSocket Streaming Real-time ~10 FPS
Memory Usage Efficient Buffer reuse
Camera Input 30 FPS 30 FPS

Optimizations Implemented

  • Buffer Reuse: OpenCV Mat objects recycled (reduces GC pressure by ~30%)
  • Frame Dropping: Skip processing if renderer/WebSocket busy
  • Memory Pool: Fixed ArrayBlockingQueue (size: 3, O(1) operations)
  • NEON SIMD: ARM vectorization enabled (-mfpu=neon flag)
  • GPU Textures: Direct glTexSubImage2D uploads
  • Thread Isolation: Separate threads for camera/processing/render
  • Zero-Copy JNI: Direct ByteBuffer access (eliminates memcpy)
  • WebSocket Throttling: 100ms minimum between frames (~10 FPS max)

Quick Start Guide

For Evaluators

  1. Clone the repository

    git clone <repository-url>
cd EdgeVision

  2. Setup OpenCV (see Setup Instructions)

  3. Build and run Android app

    ./gradlew installDebug

  4. Start WebSocket server in the app (tap "START WS")

  5. Open web viewer

    cd web
python3 -m http.server 8080
# Open http://localhost:8080

  6. Connect web viewer to Android device

    • Enter IP shown in app
    • Click CONNECT
    • Watch real-time frames

Author

Mudit Sharma

About

Real-Time Edge Detection Viewer

Topics

android kotlin opencv cmake opengl websocket jni typescipt

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages