Elastic edge-based stream processing over Apache Storm

Abstract

Apache Storm is a distributed stream processing framework designed to handle continuous data flows, yet its elasticity remains limited when components are deployed across heterogeneous nodes and incoming data rates vary over time. Existing work on Apache Storm does not adequately address how elasticity can be achieved without manual intervention or system restarts. As a result, performance bottlenecks in CPU utilization and bandwidth often persist, increasing end-to-end latency. This thesis introduces an additional layer on top of Apache Storm to enhance its elasticity under distributed and dynamic conditions. The proposed solution addresses both the avoidance and resolution of bottlenecks in inbound bandwidth, outbound bandwidth, and CPU resources, aiming to minimize overall processing latency. Elasticity was achieved through two complementary mechanisms. The first, global adaptation, monitors system performance to decide when and where to create new replicas and when to safely remove underutilized replicas without harming throughput. The second, local adaptation, is implemented at the operator level as a resource-aware load distribution strategy, balancing data across downstream replicas. Together, these mechanisms helped ensure that bottlenecks are avoided as much as possible and effectively resolved when they occur. The system was implemented on Docker for deployment across heterogeneous nodes. Experimental results show that the system can create and remove replicas on the fly, without interrupting execution, while successfully detecting and mitigating CPU and bandwidth bottlenecks. These mechanisms significantly reduce average end-to-end latency and improve overall resource utilization compared to baseline configurations. Moreover, the findings showed that the proposed solution can scale reliably within the experimental setup. This research provides a practical approach to enhancing elasticity in distributed stream processing systems, enabling robust performance in dynamic and heterogeneous environments.

Date	8 Dec 2025
Original language	American English
Awarding Institution	École de technologie supérieure
Supervisor	Julien Gascon-Samson (Supervisor)