Website | Quick Start | User Guide

This is a monorepo that contains the Kubernetes Topology Autoscaler (KTA)

• Kubernetes Operator,
• Python SDK,
• quick start examples, and
• website.

🤔 What is Kubernetes Topology Autoscaler (KTA)?

TL;DR Kubernetes Topology Autoscaler (KTA) is a framework composed of a Kubernetes Operator and a Python SDK, specifically designed for research, development and deployment of custom autoscaling algorithms for stream processing applications running on Kubernetes.

💡 The Motivation Behind KTA

For a detailed comparison with other Kubernetes autoscaling solutions, pleaser refer to Comparison to Related Projects.

Stream processing applications are structured as directed acyclic graphs (also referred to as "streaming topology"), where nodes represent operators (e.g., filter, join, aggregation) and edges represent data flow. While this architecture enables scalable execution of data-intensive real-time workflows, it also introduces complex interdependencies between operators, making horizontal autoscaling challenging. To address these challenges, state-of-the-art horizontal autoscaling algorithms for stream processing applications¹ -- such as DS2 -- go beyond simple, threshold-based rules and judiciously determine the parallelism at the operator-level of the streaming topology.

While there are several existing horizontal autoscaling solutions for Kubernetes, they

• are limited in expressiveness, mostly relying on static scaling rules (HPA, KEDA)
• are tightly coupled to a specific framework (e.g., Flink Kubernetes Operator Autoscaler), or
• only support scaling a single resource (e.g., a single Deployment) in isolation (HPA, KEDA, Custom Pod Autoscaler).

KTA addresses these limitations by providing a generic framework for research, development and deployment of stream processing autoscaling algorithms. KTA enables fine-grained, horizontal scaling at the operator level², and allows users to define custom stream processing autoscaling algorithms using a general purpose programming language. Implement your autoscaling algorithm with the KTA Python SDK, and the KTA Kubernetes Operator handles orchestration and reconciliation.

At Dynatrace Research, we actively use KTA for research and development of stream processing autoscaling algorithms 🔬🧪

⭐ Key Features

For upcoming features, please refer to our Roadmap.

• Framework-/System-agnostic scaling: Supports operator-level scaling over streaming topologies. Deployment-level scaling is also supported, as it is just a special case of operator-level scaling (i.e., a topology with a single node).
• Research and prototyping: Focus on algorithm development, not on writing boiler plate code or worrying about orchestration.
• Support for complex algorithms: Algorithms can be implemented in Python using the KTA Python SDK³. This includes algorithms that are based on machine learning.
• Composable architecture: Built on the Monitor-Analyze-Plan-Execute over shared Knowledge (MAPE-K) feedback loop. Implement your autoscaling algorithm once and reuse it across different monitoring systems and stream processing frameworks by only customizing the Monitor step.
• Traceable decision-making: The result of each loop iteration ("Knowledge") is stored in the Kubernetes Operator, ensuring transparency and traceability of scaling decisions.

🗺️ System Overview

📜 License

This project is licensed under Apache License 2.0.

💻 Contributing

See Contributor Guide.

⚖️ Disclaimer

Created by Dynatrace Research.

For general questions or inquiries, please open a GitHub issue.

Caution

KTA is currently in alpha.

We recommend using KTA in a test environment. The API and behavior may change without prior notice. Use at your own risk.

ℹ️ Note
This product is not officially supported by Dynatrace.

Footnotes

1. Ranging from frameworks like Apache Flink and Apache Kafka Streams to native-Kubernetes streaming workflows that are organized as a directed acyclic graph of Kubernetes Resources that implement the Scale Subresource. ↩

2. If and how operator-level scaling can be supported depends on the respective system. Currently, we support operator-level scaling for Apache Kafka Streams, Apache Flink, and native-Kubernetes streaming worfklows that are composed of Kubernetes Deployments. ↩

3. Actually any programming language, as long as there is a way to handle HTTP requests. However, currently you would have to handle the low-level interactions with the KTA Kubernetes Operator for other programming languages than Python yourself. ↩