feat: unreal engine mcp

[jeevanpillay]

Unreal Engine MCP Server

Overview

Implement MCP server for Unreal Engine to enable AI-driven game development, level design, and interactive content creation workflows.

Technical Approach

• Connection Method: Direct Python API integration via unreal module
• Dependencies: Unreal Engine Python API, socket communication
• Architecture: Follow Blender server pattern with UE-specific adaptations

Priority

• Phase: 1 (High Priority)
• Category: Game Development
• Connection Method: Python API
• Type: Feature

Key Features

• Scene management (create/delete actors, query scene hierarchy)
• Asset operations (import/export assets, manage content browser)
• Transform controls (position, rotation, scale manipulation)
• Blueprint integration (create/modify blueprint classes)
• Rendering (camera controls, lighting setup, render pipeline)
• Level editing (terrain modification, landscape tools)

Implementation Requirements

• Follow BaseServer architecture pattern
• Integrate with existing lightfast-mcp ecosystem
• Support both stdio and streamable-http transports
• Comprehensive error handling and logging
• Security validation for all operations

Related Issues

Part of the broader creative applications integration roadmap for 40+ applications.

Labels

• enhancement
• game-development
• phase-1

[jeevanpillay]

Unreal Engine MCP Server Implementation Plan

Technical Approach

• Connection Method: Direct Python API integration via unreal module
• Dependencies: Unreal Engine Python API, socket communication
• Architecture: Follow Blender server pattern with UE-specific adaptations

Directory Structure

src/lightfast_mcp/servers/unreal/
├── server.py          # Main UnrealMCPServer implementation
├── __init__.py        # Package initialization
├── tools/             # Unreal Engine-specific tools
│   ├── __init__.py
│   ├── actors.py      # Actor management
│   ├── assets.py      # Asset operations
│   ├── blueprints.py  # Blueprint integration
│   ├── levels.py      # Level editing
│   ├── rendering.py   # Rendering operations
│   └── materials.py   # Material system
└── utils/             # Unreal utilities
    ├── __init__.py
    └── unreal_helpers.py # Helper functions

Key Features to Implement

Core Tools

@mcp.tool()
async def create_actor(actor_class: str, location: list[float] = None, name: str = None) -> str:
    """Create a new actor in the level."""

@mcp.tool()
async def delete_actor(actor_name: str) -> str:
    """Delete an actor from the level."""

@mcp.tool()
async def import_asset(file_path: str, destination_path: str = "/Game/") -> str:
    """Import asset into the content browser."""

@mcp.tool()
async def create_blueprint(blueprint_name: str, parent_class: str = "Actor") -> str:
    """Create a new Blueprint class."""

@mcp.tool()
async def set_actor_transform(actor_name: str, location: list[float] = None, 
                            rotation: list[float] = None, scale: list[float] = None) -> str:
    """Set actor transform (location, rotation, scale)."""

@mcp.tool()
async def create_material(material_name: str, base_color: list[float] = None) -> str:
    """Create a new material."""

@mcp.tool()
async def render_screenshot(output_path: str, resolution: list[int] = None) -> str:
    """Render screenshot of current viewport."""

@mcp.tool()
async def build_lighting() -> str:
    """Build lighting for the current level."""

Implementation Example

Server Class

from lightfast_mcp.core.base_server import BaseServer
from lightfast_mcp.core.server_config import ServerConfig
import sys
import os
from typing import Dict, Any, List

class UnrealMCPServer(BaseServer):
    SERVER_TYPE = "unreal"
    REQUIRED_APPS = ["Unreal Engine"]
    
    def __init__(self, config: ServerConfig):
        super().__init__(config)
        self.unreal = None
        self._setup_unreal_api()
        
    def _setup_unreal_api(self):
        """Setup Unreal Engine Python API."""
        try:
            import unreal
            self.unreal = unreal
            self.editor_subsystem = unreal.get_editor_subsystem(unreal.EditorActorSubsystem)
            self.asset_tools = unreal.AssetToolsHelpers.get_asset_tools()
            self.editor_util = unreal.EditorUtilityLibrary()
        except ImportError:
            self.logger.error("Unreal Engine Python API not available")
            self.unreal = None
    
    async def _check_application_availability(self) -> bool:
        """Check if Unreal Engine is running and accessible."""
        try:
            return self.unreal is not None and self.editor_subsystem is not None
        except Exception:
            return False
    
    def _register_tools(self):
        """Register all Unreal Engine-specific tools."""
        self.mcp.tool()(self.create_actor)
        self.mcp.tool()(self.delete_actor)
        self.mcp.tool()(self.import_asset)
        self.mcp.tool()(self.create_blueprint)
        self.mcp.tool()(self.set_actor_transform)
        self.mcp.tool()(self.create_material)
        self.mcp.tool()(self.render_screenshot)
        self.mcp.tool()(self.build_lighting)
        self.mcp.tool()(self.get_level_info)
        self.mcp.tool()(self.list_actors)
        self.mcp.tool()(self.duplicate_actor)
    
    async def create_actor(self, actor_class: str, location: List[float] = None, name: str = None) -> str:
        """Create a new actor in the level."""
        try:
            if not self.unreal:
                raise Exception("Unreal Engine API not available")
            
            # Get actor class
            actor_class_obj = self.unreal.EditorAssetLibrary.load_asset(f"/Script/Engine.{actor_class}")
            if not actor_class_obj:
                # Try common actor classes
                class_map = {
                    "StaticMeshActor": self.unreal.StaticMeshActor,
                    "PointLight": self.unreal.PointLight,
                    "Camera": self.unreal.CameraActor,
                    "Pawn": self.unreal.Pawn,
                    "Character": self.unreal.Character
                }
                actor_class_obj = class_map.get(actor_class)
                
            if not actor_class_obj:
                raise ValueError(f"Unknown actor class: {actor_class}")
            
            # Set spawn location
            spawn_location = self.unreal.Vector(0, 0, 0)
            if location and len(location) >= 3:
                spawn_location = self.unreal.Vector(location[0], location[1], location[2])
            
            # Spawn actor
            actor = self.editor_subsystem.spawn_actor_from_class(
                actor_class_obj, 
                spawn_location,
                self.unreal.Rotator(0, 0, 0)
            )
            
            if actor and name:
                actor.set_actor_label(name)
            
            actor_name = actor.get_actor_label() if actor else "Unknown"
            return f"Created {actor_class} actor: {actor_name}"
            
        except Exception as e:
            return f"Error creating actor: {str(e)}"
    
    async def delete_actor(self, actor_name: str) -> str:
        """Delete an actor from the level."""
        try:
            if not self.unreal:
                raise Exception("Unreal Engine API not available")
            
            # Find actor by name
            all_actors = self.editor_subsystem.get_all_level_actors()
            target_actor = None
            
            for actor in all_actors:
                if actor.get_actor_label() == actor_name:
                    target_actor = actor
                    break
            
            if not target_actor:
                raise ValueError(f"Actor '{actor_name}' not found")
            
            # Delete actor
            success = self.editor_subsystem.destroy_actor(target_actor)
            
            if success:
                return f"Deleted actor: {actor_name}"
            else:
                raise Exception("Failed to delete actor")
                
        except Exception as e:
            return f"Error deleting actor: {str(e)}"
    
    async def import_asset(self, file_path: str, destination_path: str = "/Game/") -> str:
        """Import asset into the content browser."""
        try:
            if not self.unreal:
                raise Exception("Unreal Engine API not available")
            
            if not os.path.exists(file_path):
                raise ValueError(f"File not found: {file_path}")
            
            # Create import task
            import_task = self.unreal.AssetImportTask()
            import_task.filename = file_path
            import_task.destination_path = destination_path
            import_task.automated = True
            import_task.save = True
            
            # Import asset
            self.asset_tools.import_asset_tasks([import_task])
            
            # Get imported asset name
            asset_name = os.path.splitext(os.path.basename(file_path))[0]
            
            return f"Imported asset: {asset_name} to {destination_path}"
            
        except Exception as e:
            return f"Error importing asset: {str(e)}"
    
    async def create_blueprint(self, blueprint_name: str, parent_class: str = "Actor") -> str:
        """Create a new Blueprint class."""
        try:
            if not self.unreal:
                raise Exception("Unreal Engine API not available")
            
            # Get parent class
            parent_class_obj = getattr(self.unreal, parent_class, None)
            if not parent_class_obj:
                raise ValueError(f"Unknown parent class: {parent_class}")
            
            # Create blueprint
            blueprint_path = f"/Game/{blueprint_name}"
            blueprint = self.unreal.BlueprintFactory()
            blueprint.parent_class = parent_class_obj
            
            # Create the asset
            created_asset = self.asset_tools.create_asset(
                blueprint_name,
                "/Game/",
                self.unreal.Blueprint,
                blueprint
            )
            
            if created_asset:
                return f"Created Blueprint: {blueprint_name}"
            else:
                raise Exception("Failed to create Blueprint")
                
        except Exception as e:
            return f"Error creating Blueprint: {str(e)}"
    
    async def set_actor_transform(self, actor_name: str, location: List[float] = None,
                                rotation: List[float] = None, scale: List[float] = None) -> str:
        """Set actor transform (location, rotation, scale)."""
        try:
            if not self.unreal:
                raise Exception("Unreal Engine API not available")
            
            # Find actor
            all_actors = self.editor_subsystem.get_all_level_actors()
            target_actor = None
            
            for actor in all_actors:
                if actor.get_actor_label() == actor_name:
                    target_actor = actor
                    break
            
            if not target_actor:
                raise ValueError(f"Actor '{actor_name}' not found")
            
            # Get current transform
            current_transform = target_actor.get_actor_transform()
            
            # Update location
            if location and len(location) >= 3:
                new_location = self.unreal.Vector(location[0], location[1], location[2])
                current_transform.translation = new_location
            
            # Update rotation
            if rotation and len(rotation) >= 3:
                new_rotation = self.unreal.Rotator(rotation[0], rotation[1], rotation[2])
                current_transform.rotation = new_rotation.quaternion()
            
            # Update scale
            if scale and len(scale) >= 3:
                new_scale = self.unreal.Vector(scale[0], scale[1], scale[2])
                current_transform.scale3d = new_scale
            
            # Apply transform
            target_actor.set_actor_transform(current_transform, False, True)
            
            return f"Updated transform for actor: {actor_name}"
            
        except Exception as e:
            return f"Error setting actor transform: {str(e)}"
    
    async def create_material(self, material_name: str, base_color: List[float] = None) -> str:
        """Create a new material."""
        try:
            if not self.unreal:
                raise Exception("Unreal Engine API not available")
            
            # Create material factory
            material_factory = self.unreal.MaterialFactoryNew()
            
            # Create the material asset
            material = self.asset_tools.create_asset(
                material_name,
                "/Game/Materials/",
                self.unreal.Material,
                material_factory
            )
            
            if material and base_color and len(base_color) >= 3:
                # Set base color (this would require more complex material graph setup)
                # For now, just create the material
                pass
            
            if material:
                return f"Created material: {material_name}"
            else:
                raise Exception("Failed to create material")
                
        except Exception as e:
            return f"Error creating material: {str(e)}"
    
    async def render_screenshot(self, output_path: str, resolution: List[int] = None) -> str:
        """Render screenshot of current viewport."""
        try:
            if not self.unreal:
                raise Exception("Unreal Engine API not available")
            
            # Set default resolution
            width, height = resolution if resolution and len(resolution) >= 2 else [1920, 1080]
            
            # Take screenshot
            success = self.unreal.AutomationLibrary.take_high_res_screenshot(
                width, height, output_path
            )
            
            if success:
                return f"Screenshot saved to: {output_path}"
            else:
                raise Exception("Failed to take screenshot")
                
        except Exception as e:
            return f"Error taking screenshot: {str(e)}"
    
    async def build_lighting(self) -> str:
        """Build lighting for the current level."""
        try:
            if not self.unreal:
                raise Exception("Unreal Engine API not available")
            
            # Build lighting
            self.unreal.EditorLevelLibrary.build_lighting(quality=self.unreal.LightingBuildQuality.MEDIUM)
            
            return "Lighting build started"
            
        except Exception as e:
            return f"Error building lighting: {str(e)}"
    
    async def get_level_info(self) -> str:
        """Get information about the current level."""
        try:
            if not self.unreal:
                raise Exception("Unreal Engine API not available")
            
            # Get level info
            current_level = self.unreal.EditorLevelLibrary.get_editor_world()
            level_name = current_level.get_name() if current_level else "Unknown"
            
            all_actors = self.editor_subsystem.get_all_level_actors()
            actor_count = len(all_actors)
            
            # Count different actor types
            actor_types = {}
            for actor in all_actors:
                actor_type = actor.__class__.__name__
                actor_types[actor_type] = actor_types.get(actor_type, 0) + 1
            
            info = {
                "level_name": level_name,
                "total_actors": actor_count,
                "actor_types": actor_types
            }
            
            return str(info)
            
        except Exception as e:
            return f"Error getting level info: {str(e)}"
    
    async def list_actors(self, actor_type: str = None) -> str:
        """List all actors in the level, optionally filtered by type."""
        try:
            if not self.unreal:
                raise Exception("Unreal Engine API not available")
            
            all_actors = self.editor_subsystem.get_all_level_actors()
            
            actor_list = []
            for actor in all_actors:
                actor_class_name = actor.__class__.__name__
                
                # Filter by type if specified
                if actor_type and actor_type.lower() not in actor_class_name.lower():
                    continue
                
                actor_info = {
                    "name": actor.get_actor_label(),
                    "type": actor_class_name,
                    "location": str(actor.get_actor_location())
                }
                actor_list.append(actor_info)
            
            return str(actor_list)
            
        except Exception as e:
            return f"Error listing actors: {str(e)}"
    
    async def duplicate_actor(self, actor_name: str, new_name: str = None) -> str:
        """Duplicate an existing actor."""
        try:
            if not self.unreal:
                raise Exception("Unreal Engine API not available")
            
            # Find source actor
            all_actors = self.editor_subsystem.get_all_level_actors()
            source_actor = None
            
            for actor in all_actors:
                if actor.get_actor_label() == actor_name:
                    source_actor = actor
                    break
            
            if not source_actor:
                raise ValueError(f"Actor '{actor_name}' not found")
            
            # Duplicate actor
            duplicated_actor = self.editor_subsystem.duplicate_actor(source_actor)
            
            if duplicated_actor:
                if new_name:
                    duplicated_actor.set_actor_label(new_name)
                
                result_name = duplicated_actor.get_actor_label()
                return f"Duplicated actor '{actor_name}' as '{result_name}'"
            else:
                raise Exception("Failed to duplicate actor")
                
        except Exception as e:
            return f"Error duplicating actor: {str(e)}"

Configuration Example

# config/servers.yaml
unreal:
  enabled: true
  transport: stdio
  connection:
    timeout: 60
    python_path: "C:/Program Files/Epic Games/UE_5.3/Engine/Binaries/ThirdParty/Python3/Win64/python.exe"
  features:
    - actor_management
    - asset_operations
    - blueprint_integration
    - level_editing
    - rendering
    - material_system

Entry Point Script

# src/lightfast_mcp/servers/unreal_mcp_server.py
#!/usr/bin/env python3
"""Unreal Engine MCP Server entry point."""

import asyncio
from lightfast_mcp.servers.unreal.server import UnrealMCPServer
from lightfast_mcp.core.server_config import ServerConfig

async def main():
    config = ServerConfig.from_env("unreal")
    server = UnrealMCPServer(config)
    await server.run()

if __name__ == "__main__":
    asyncio.run(main())

Setup Requirements

1. Unreal Engine Installation: Ensure Unreal Engine is installed with Python support enabled
2. Python API: Enable Python scripting in Unreal Engine project settings
3. Editor Mode: Server must run while Unreal Editor is open
4. Project Setup: Configure Python paths and enable required plugins

This implementation provides comprehensive Unreal Engine automation through MCP, enabling AI-driven game development and interactive content creation workflows.