Dynamic Chess Board-Pieces Extraction

Computer vision system that automatically extracts chess board positions and piece placements from images or real-time video, converting them into standard FEN (Forsyth-Edwards Notation) format. This project uses YOLO and RF-DETR object detection with image processing techniques to recognize chess pieces and their positions on the board.

Mobile App

Chesspector is available on Google Play. Scan physical boards, analyze positions with Stockfish, and practice from digitized games. This project started with local image analysis, then evolved into a published mobile app that runs the same core pipeline using Flutter, OpenCV, RF-DETR, and AWS-hosted services. Chesspector on Google Play

Project Versions

There are two versions of this project:

1. Extract the chessboard and pieces from an image (.jpeg, .png, etc.).
2. Extract the chessboard and pieces using an OAK-D Lite Camera (utilizing DepthAI for real-time video and inference).

C++ version is also available : cpp-version/

I had a blog post about this project on my personal website. You can read it here: https://visionbrick.com/how-to-extract-chess-positions-from-any-photo/

Version 1: Conversion from image

Version 2: real-time conversion with OAK D Lite camera (depthai)

There are two files for converting images to FEN format: the first uses a square-filling algorithm, and the second uses perspective transformation.

• The square-filling algorithm works better with non-angled (straight) images.
  
• Perspective transformation works better with images taken from different angles.

Perspective transformation Method

Square Filling Method

Installation

CPU Version (Recommended for most users)

pip install -r requirements.txt

GPU Version (For CUDA-compatible systems)

There is an additional step for installing PyTorch with GPU support. Please check the end of the requirements-gpu.txt file.

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements-gpu.txt

RF-DETR Version

Use this environment when you want RF-DETR support. Inside the code, you need to change YOLO to RF-DETR for inference.

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements-rfdetr.txt

Note: The GPU version requires NVIDIA CUDA toolkit to be installed on your system. If you're unsure, start with the CPU version.

Docker Setup

Running with Docker

For a containerized environment, you can use Docker to run the project without installing dependencies directly on your system.

Note: The Docker setup currently uses the CPU version (requirements.txt) only. GPU support is not included in the Docker configuration.

Build the Docker Image

docker build -t dynamic-chess-board-demo:latest .

Run with Jupyter Lab

docker run --rm -it -p 8888:8888 -v ${PWD}:/app -w /app dynamic-chess-board-demo:latest jupyter lab --ip=0.0.0.0 --no-browser --allow-root

Run Interactive Container

docker run --rm -it -v ${PWD}:/app -w /app dynamic-chess-board-demo:latest

Files

• square_filling.py: Script for conversion using the square-filling algorithm.

• square_filling-step-by-step.ipynb: Jupyter notebook for visualizing the entire process step by step.

• perspective_transformation.py: Script for conversion using perspective transformation.

• perspective_transformation-step-by-step.ipynb: Jupyter notebook for visualizing the entire process step by step.

• chess-model-yolov8m.pt --> Trained YOLOv8 model for chess piece detection.

• chess-model-rfdetr.pth --> Trained RF-DETR model for chess piece detection.

• rfdetr-inference-demo.ipynb --> Notebook for RF-DETR inference and matplotlib detection visualization on one test image.
  

Folders

• extracted-data --> It contains result (converted image), and all the information (coordinates,board ..)

• test-images --> Collection of images for testing purposes.

• example-results --> Contains various images along with their corresponding results.

• Dephtai-chess (Folder) --> It contains real-life camera version with depthai library , It is not updated but it can still be used. I have different and better algorithms but this depthai version is not using them , it is old . I will update it (not updated)
  
  

Important Note: I didn't train the model enough because the first phase of this project was extracting the board and pieces dynamically with changing positions and different boards. As a result, the model cannot predict all the pieces correctly, but the positions are nearly perfect. You can train better models and use them with this code. In the future, I will train better models.

Example Images

for more example, check example-results folder

Example of image using the square-filling algorithm.

Example of conversion using perspective transformation.

Example of square-filling algorithm on an image.