Decentralized coordination of autonomous agents in the Compute Continuum using consensus

1 Introduction

The growing availability of Internet of Things (IoT) devices, edge-based systems, cloud infrastructures, and high-performance computing (HPC) systems is reshaping how distributed applications are designed and orchestrated. These hyper-distributed applications require seamless integration across the Compute Continuum (Balouek-Thomert et al., 2019; Antoniu et al., 2021), where computation and data are distributed across heterogeneous and geographically dispersed resources interconnected through diverse networking technologies that can often be unreliable. Executing workloads closer to data sources can significantly reduce latency, improve energy efficiency, and enhance privacy by limiting unnecessary data transfers to centralized cloud infrastructures (Shi et al., 2016). However, these benefits come at the cost of increased complexity, as applications must operate across dynamic, multi-tenant environments with limited global visibility and unreliable connectivity.

Most existing orchestration and workload management solutions rely on centralized control planes to make deployment and scheduling decisions (Vaño et al., 2023). While effective in traditional cloud and HPC settings, centralized approaches face fundamental limitations in hyper-distributed environments. They introduce single points of failure, struggle to react to dynamic and volatile environments, and incur additional communication overhead that undermines the responsiveness of latency-sensitive applications. As the scale and heterogeneity of the Compute Continuum grow, these limitations increasingly motivate the exploration of decentralized control paradigms.

Multi-agent systems (MAS) provide a natural abstraction for decentralized environments. In this model, autonomous agents operate based on local observations, manage local resources, and interact with peers to achieve global objectives. Such systems are scalable and resilient, as decisions are distributed and do not depend on a centralized entity. However, decentralization introduces a critical challenge: coordination. Agents must agree on which actions to take and which functionalities (i.e., roles) to assume–commonly referred to in the literature as the role allocation problem (Campbell and Wu, 2011).

Consensus algorithms play a fundamental role in addressing this challenge. By allowing agents to reach agreement through coordinated interactions, consensus enables coherent collective behavior without centralized oversight. Consensus mechanisms have been extensively studied in distributed systems and applied to problems such as leader election, fault tolerance, and replicated state management (Howard et al., 2016; Hunt et al., 2010). In multi-agent systems, consensus has also been explored for distributed task allocation in multi-robot systems (Mahato et al., 2023; Choi et al., 2009) and job selection for achieving decentralized management of workflows (Thareja et al., 2025). Nevertheless, its application to the dynamic orchestration of hyper-distributed services in swarm-like environments across the Compute Continuum remains underexplored. These agents often communicate via wide area networks, making latency a challenge for consensus algorithms.

In this work, we investigate the use of consensus to manage dynamic service orchestration in decentralized multi-agent systems. Services are dynamically managed through the use of autonomous agents, which self-organize to decide which functionalities to execute based on their capabilities and real-time quality-of-service (QoS) indicators.

To explore this question, we built a prototype that leverages and extends COLMENA (Lordan et al., 2024), a swarm-based programming environment for developing and orchestrating services in the Compute Continuum. To enable decentralized coordination, we extended the consensus mechanism in Thareja et al. (2025) and integrated it within COLMENA. While the original algorithm was designed for job selection, in this work, we extend it to support decision-making in hyper-distributed services, allowing agents to autonomously select functionalities/roles based on their capabilities and real-time system conditions.

We demonstrate the applicability of this implementation through a use case in modern power systems. Electrical grids are undergoing a profound transformation driven by the large-scale integration of renewable energy sources. This shift from a few centralized, predictable power plants to thousands of distributed, intermittent resources introduces operational challenges (Fang et al., 2011). As a result, power systems increasingly require decentralized, adaptive control mechanisms capable of reacting quickly to local disturbances while preserving global stability. Using the ANDES toolbox (Cui et al., 2020), we simulated the NPCC 140-Bus network, which spans across the northeastern region of the United States and parts of Canada. COLMENA agents monitor frequency deviations, detect contingencies, and self-organize to execute distributed optimization algorithms that stabilize grid frequency. In this scenario, consensus is used to elect local leaders responsible for executing optimization roles, ensuring coordinated responses across geographically distributed areas. We tested this use case across six sites in the FABRIC testbed (Baldin et al., 2019), including up to 60 geographically distributed nodes. To facilitate reproducibility and future reuse, we provide our experimental setup in Kiso, an experiment management platform (Pegasus Project Contributors, 2025) for workloads in the Compute Continuum.

The main contributions of this paper are threefold. First, we propose a decentralized coordination approach based on consensus for managing dynamic service execution in multi-agent systems operating across the Compute Continuum. Second, we present the integration of this strategy into a middleware-based programming environment, which supports autonomous role selection and elastic service management. Third, we provide an experimental evaluation using a realistic power grid frequency optimization scenario deployed across geographically distributed resources, demonstrating scalability, responsiveness, and low operational overhead.

Section 2 gives an overview of the related work regarding consensus mechanisms, multi-agent systems, and service orchestration in the Continuum. Section 3 describes our contribution, specifying our methodology for implementing the consensus algorithm, the use case in modern power systems, and the experiment deployment using the Kiso resource provisioning system (Pegasus Project Contributors, 2025) and the FABRIC testbed. The experiments and results are described in Section 4, followed by the discussion in Section 5.

2 Related work

2.1 Consensus mechanisms

Consensus algorithms are a cornerstone of distributed systems, enabling multiple agents to reach agreement on shared decisions even in the presence of failures or unreliable communication. Classical consensus protocols, such as Practical Byzantine Fault Tolerance (PBFT) (Castro and Liskov, 2002), Raft (Ongaro and Ousterhout, 2014), and Paxos (Howard et al., 2016), have been extensively studied and applied to multiple challenges in distributed systems. PBFT is resilient to Byzantine failures, tolerating malicious or arbitrary agent behavior through a multi-phase agreement process, whereas Raft and Paxos are designed primarily for crash-failure tolerance, favoring simplicity and scalability over resilience to arbitrary faults.

Thareja et al. (2025) developed a PBFT-based algorithm for job selection in workflow management systems. The novelty of this approach lies in the integration of a cost function that explicitly maps agent capabilities to job requirements, allowing agents to collectively select assignments that optimize resource utilization. Each agent calculates a cost matrix, typically based on available resources such as CPU, RAM, and disk, which is shared among agents to determine the optimal job assignment. This contrasts with traditional workflow managers that rely on a centralized runtime, enabling fully distributed, decentralized job selection. In this algorithm, all agents participate in the selection process and assume a global pool of jobs.

Building on this approach, our work further enhances the consensus algorithm by incorporating dynamic role selection in hyper-distributed services, thereby facing new challenges. Unlike job selection, where tasks are globally visible, roles are locally triggered based on agent observations and QoS conditions. There is no global view of all required roles; consensus is reached only when a sufficient number of agents independently determine that a role is needed. This increases the complexity of coordination, as agents must continuously evaluate the QoS state, negotiate role assignments, and dynamically adapt to events and platform changes.

2.2 Multi-agent systems and service orchestration in the Continuum

Several programming environments and middleware platforms have been proposed to support distributed and adaptive applications across heterogeneous computing environments. Traditional agent-based frameworks, such as JADE (Bellifemine et al., 2000), provide abstractions for autonomous decision-making, coordination, and reasoning, typically relying on predefined mappings between agents and tasks or behaviors. However, in these frameworks, the assignment of roles to agents is often static or centrally orchestrated, and typically does not account for QoS requirements.

In parallel, microservice architectures, commonly deployed through containerization technologies, have become the dominant paradigm for building scalable and modular distributed systems. Containerized microservices enable lightweight deployment, isolation, and elasticity across cloud and edge infrastructures. However, microservice orchestration platforms such as Kubernetes (The Kubernetes Authors, 2024), Apache Mesos (Hindman et al., 2011), Nomad (HashiCorp, 2025), and KubeEdge (Li et al., 2021), usually rely on centralized controllers and predefined policies for service placement, scaling, and lifecycle management. As a result, they offer limited support for fine-grained, autonomous decision-making at the device level and for dynamic role assignment driven by real-time QoS conditions across the Compute Continuum.

COLMENA (Lordan et al., 2024; Lordan et al., 2026) differs from both traditional agent frameworks and containerized microservice platforms by introducing a role-based programming model in which services are decomposed into roles that can be dynamically executed or terminated by agents themselves. Roles encapsulate executable, application-specific functionalities that solve particular tasks or problems within a service. Each role includes the behavior needed to perform its task, along with abstractions for interacting with other roles and with the broader service. Roles are the primary units of execution in COLMENA, enabling agents to make fine-grained decisions about which functionalities to execute and when.

Services are defined using this programming model, allowing developers to declaratively specify roles, behaviors, and key performance indicators (KPIs) for QoS demands. A KPI represents the aggregation of one or more metrics over a specific context, involving multiple agents and operations. Each KPI is associated with a threshold that determines whether the desired QoS level is satisfied or violated. KPIs are formally specified using Prometheus Query Language (PromQL) (Prometheus Authors, 2026), which defines the metrics to be observed, the aggregation functions applied across agents or services, and the conditions under which a violation or completion is detected. Agents execute decision-making policies to start or stop roles based on these KPI levels.

The COLMENA agent stack integrates context awareness, decision-making, QoS monitoring, and data and compute sharing, enabling fully decentralized and adaptive behavior. Since COLMENA uses a microservice architecture, it integrates state-of-the-art technologies such as Prometheus (Prometheus Authors, 2026) for QoS evaluation, Zenoh (Eclipse Foundation, 2022) for name-centric networking, data sharing, and discovery, COMPSs (Lordan et al., 2014) for compute sharing, ZeroMQ (The ZeroMQ authors, 2025) for communication, and Grafana (Grafana Labs, 2026) for visualization and monitoring. A key novelty of COLMENA lies in the autonomous and dynamic mapping between roles and agents, which is dynamically adjusted at runtime according to QoS metrics in a decentralized way by agents themselves using decision-making policies. These policies range from simple rule-based strategies to more sophisticated mechanisms based on consensus mechanisms, as explored in this work.

3 Materials and methods

The objective of this work is to investigate decentralized decision-making for role selection in multi-agent systems. In this paradigm, services are composed as a set of roles that are autonomously selected and executed (or terminated) by agents based on their available resources and observed QoS. An analogous problem has been extensively studied in the context of traditional workflow systems, where decentralized job allocation is achieved through consensus mechanisms or distributed scheduling policies. Building on this line of work, our methodology extends an existing consensus-based coordination mechanism—originally introduced for decentralized job selection in workflow management systems (Thareja et al., 2025)—so that it can be applied to role-based service orchestrators. We describe this extension in Section 3.1.

To demonstrate the applicability of the proposed approach and to evaluate its effectiveness, we consider a use case in modern power systems (Section 3.2), in which agents collaboratively execute a distributed control algorithm to regulate grid frequency, which must be maintained within a narrow range around its nominal value. Distributing frequency control enables faster decision-making while reducing reliance on centralized resources. Using the COLMENA platform, this use case is expressed as a set of roles, abstractions, and KPIs, and deployed across a highly distributed infrastructure on the FABRIC testbed. Section 3.3 details the experimental deployment and the use of Kiso, an experiment management framework for workloads in the Compute Continuum.

3.1 Consensus-based role selection in COLMENA

In decentralized multi-agent systems, consensus algorithms enable agents to reach agreement on shared decisions without relying on a central coordinator. The fundamental challenge is ensuring that all participating agents converge on the same outcome, even when messages may arrive out of order, or some agents may fail. Our approach builds upon the Practical Byzantine Fault Tolerance (PBFT) protocol, adapting its core principles to the problem of role selection in swarm-based services.

3.1.1 Overview of the consensus mechanism

The consensus mechanism presented in this work enables agents to dynamically determine which agent should execute a given role in response to a KPI violation. When a KPI indicates that a role should be executed, the initiating agent does not act unilaterally. Instead, it triggers a distributed agreement process, in which participating agents compute cost values reflecting their suitability for the role based on available resources such as CPU, memory, and disk. These costs are shared, and through a series of message exchanges, the group collectively elects a leader—the agent with the lowest cost—to execute the role. Figure 1a shows the start of the consensus process, while Figure 1b shows its completion, from leader election to role execution by the selected agent.

To illustrate this process, consider our use case (further explained in Section 3.2), where agents monitor grid frequency and request that a frequency optimizer role be started whenever the frequency deviates beyond acceptable bounds, such as during contingencies like load changes or generator disconnections. Once the consensus process selects a leader, the role is executed with the goal of bringing the frequency back to stable values. While this use case focuses on grid frequency, the mechanism is general and can be applied to any KPI and role within a service.

Agreement is reached through a three-phase protocol: propose, prepare, and commit, following the PBFT protocol principles. In the propose phase, an agent broadcasts its candidacy along with its computed cost. During the prepare phase, agents evaluate incoming proposals, vote for the best candidate, and share their votes with peers. Once a sufficient number of agents (a quorum) have voted for the same candidate, the protocol enters the commit phase, where agents confirm their final agreement. The quorum requirement—typically a majority of participating agents—ensures that no two conflicting decisions can be reached simultaneously, even if some agents fail or messages are delayed. Upon successful commitment, only the elected leader proceeds to execute the role, while other agents record the outcome and continue monitoring the system.

Figure 2 illustrates the message exchange during the consensus process. In this example, Agent 1 has a lower cost function than the other two agents. Therefore, Agent 1 broadcasts a proposal to the others. The remaining agents then broadcast prepare messages. Subsequently, the agents enter the commit phase, where they reach quorum and agree on Agent 1 as the leader. Agent 1 then triggers role execution, while the other agents detect that consensus has been achieved based on the received commit messages.

The cost function used in this work is described in Equation 1. It quantifies the suitability of an agent to execute a given role by comparing the role’s resource demands d r to the agent’s available capacities c r. For each resource type (CPU, RAM, and disk), the usage ratio d r / c r is computed and summed to capture the overall resource load. To account for potential bottlenecks, the function also considers the maximum usage ratio across all resources and squares it. This penalizes roles that heavily stress a single resource, favoring agents with more balanced resource availability.

The cost function can be configured depending on the application needs. For example, more terms could be added such as network proximity, communication latency to neighboring agents, or reliance/trust metrics.

C = 100 ∑ r ∈ core, ram, disk d r c r + max r ∈ core, ram, disk d r c r 2 (1)

This approach ensures consistency across the distributed system: all agents agree on who should act, duplicate executions are avoided, and the most capable agent is selected based on objective resource metrics. The protocol is resilient to agent failures, as the dynamic quorum calculation adapts to the number of currently active agents. In the original implementation, this consensus mechanism was applied to job selection in workflow management systems, where agents continuously select jobs from a globally shared pool. In the following sections, we describe how we adapted this approach for role-based service orchestration in COLMENA, which introduces distinct challenges due to the event-driven and locally-triggered nature of role requests.

3.1.2 Adapting consensus for role selection

A fundamental difference arises when moving from job selection to role selection. In workflow systems, jobs are continuously selected from a globally visible pool, and the consensus mechanism operates proactively. In contrast, in role-based services, role requests are triggered asynchronously by agents in response to the status of the QoS, defined by KPIs. Therefore, the consensus process is no longer internally driven, but externally triggered, and it must ensure that a sufficient number of agents independently agree on the need to activate a role before a leader can be selected. This shift introduces additional complexity and requires a more flexible integration of consensus within the service runtime.

To support this execution model, we refactored the original codebase to decouple the consensus mechanism from job-specific assumptions and to generalize it for swarm-based coordination. The resulting architecture introduces two core abstractions. The Consensus Engine manages the exchange and processing of consensus messages (proposal, prepare, and commit) and enforces the agreement protocol. The Selection Engine computes the cost matrix and determines the leader for a given coordination object based on agent capabilities and resource availability.

An additional challenge stems from the semantic differences between jobs and roles. Jobs are uniquely identifiable and scheduled only once, whereas roles represent behavioral definitions that may be instantiated multiple times throughout the lifetime of a service and do not possess a unique identifier. To address this, we introduce a new abstraction, Role, which complements the original Job object used by the consensus algorithm. While both jobs and roles are characterized by a set of required capacities (e.g., CPU, memory, and disk), roles can be requested and executed repeatedly.

Integration of the consensus mechanism with COLMENA is achieved via the colmena_adapter, which bridges the COLMENA runtime and the consensus service through gRPC. This integration introduces several key differences compared to the original job-selection approach. First, the consensus process is now event-driven, triggered on demand by COLMENA when agents independently detect that a role needs to be instantiated due to QoS violations. Second, role reuse is supported: roles are identified by their role name and can be instantiated multiple times over the course of execution, unlike jobs, which are scheduled only once. Third, asynchronous execution ensures that, upon completion of the consensus process, only the elected leader sends a gRPC request back to the COLMENA agent to activate the corresponding role.

We extended COLMENA to allow asynchronous policy execution. This is necessary because agents may request consensus at different times, and only the elected leaders must trigger concrete actions for starting roles. Asynchronous execution avoids blocking the agent decision loop and enables scalable coordination. The started roles are terminated once the KPI is back to the desired range. Additionally, we introduced a timeout that prevents agents from repeatedly requesting the same role within a short time window. This design choice accounts for the delayed impact of role execution on the QoS and allows KPIs to stabilize before new decisions are triggered. The value for this timeout is configurable and depends on the application, as it is adapted to the frequency stabilization time.

3.2 Use case implementation: frequency optimization in modern power grids

Decentralized frequency control in modern power grids is a critical real-time operation. The grid frequency must remain within a narrow band around its nominal value (e.g., 50 Hz) to ensure stable operation. Sudden imbalances between generation and consumption, such as generator outages or load disconnections, can cause rapid frequency variations. If not corrected, these deviations may trigger protection mechanisms, potentially leading to cascading failures and large-scale blackouts. Traditional power systems include several frequency control schemes (primary, secondary, and tertiary) aimed at correcting frequency imbalances and bringing it back to its nominal value. This tends to be achieved via thermal power plants, although they follow centralized instructions from the grid operator, and are slow to react.

The objective of this use case is to validate that autonomous agents deployed across multiple electrical areas can (i) detect frequency contingencies, (ii) coordinate corrective actions through decentralized consensus mechanisms, and (iii) execute distributed optimization-based control without relying on a centralized controller. Each agent is responsible for a specific grid area and has access only to local measurements and information exchanged with neighboring areas.

3.2.1 The distributed model predictive control (DMPC) algorithm

To address frequency control, we adopt a Model Predictive Control (MPC) formulation. MPC is well suited for power system control problems as it explicitly incorporates system dynamics, predicts future behavior over a finite horizon, and enforces operational constraints on control actions (Negenborn, 2007; Tur and Bayindir, 2019).

In this work, the power grid is modeled as a set of interconnected areas. Each area i is characterized by two state variables: the system frequency f i and the voltage angle δ i. The evolution of these states follows a discretized version of the swing equation (Caliskan and Tabuada, 2015; Venkat et al., 2008), assuming known, as shown in Equation 2, inertia and a fixed control sampling time Δ t: where P i , t generated denotes the active power generated in area i, P t i , j is the active power exchanged with neighboring area j, N i is the set of neighboring areas, and f 0 is the nominal frequency.

f i , t + 1 = f i , t + Δ t P i , t generated − ∑ j ∈ N i P t i , j ,

δ i , t + 1 = δ i , t + 2 π Δ t f i , t − f 0 , (2)

The MPC objective is to minimize frequency deviations over a prediction horizon H. Equation 3 defines the cost function for a centralized formulation:

J = ∑ t = 0 H ‖ f t − f 0 ‖ 2 2 , (3)

Subject to system dynamics and operational constraints. Control inputs in this formulation correspond to generator setpoint adjustments and load curtailment actions, reflecting the strong coupling between active power and frequency (Tur and Bayindir, 2019).

A centralized MPC formulation requires full system observability and a global optimization step, which becomes impractical for large-scale or geographically distributed grids. To overcome this limitation, we adopt a Distributed Model Predictive Control (DMPC) approach, in which each area solves a local MPC problem while coordinating with neighboring areas to ensure global consistency.

In the distributed formulation, each area maintains local copies of shared variables, such as voltage angles at tie-lines. Consistency across areas is enforced through coupling constraints of the form (Equation 4)

δ j , i = δ i , i , (4)

Which ensures that the voltage angle of area i as perceived by area j matches its actual value.

These coupling constraints are enforced using the Alternating Direction Method of Multipliers (ADMM) (Boyd et al., 2010). Following the approach described in (Negenborn, 2007), the global optimization problem is decomposed into n local subproblems, where n is the number of areas. At each ADMM iteration, agents exchange local estimates of shared variables and associated dual variables with their neighbors until convergence is achieved. Once consensus is reached, the resulting control actions are applied locally.

3.2.2 Grid frequency optimization in COLMENA

The frequency optimization solution is implemented as a service within the COLMENA framework, which provides abstractions for decentralized coordination, role execution, and communication among autonomous agents. MPC is not deployed as a continuously active controller; instead, it is encapsulated as a COLMENA role that is activated when a frequency-related KPI is violated.

Due to the limited availability of real-world testing environments for electrical power grids, we simulate the NPCC 140-bus system using the open-source power system analysis tool ANDES (see Section 4 for the results). The NPCC is a widely used benchmark system representing the power grid of the northeastern United States, originally developed within the Power System Toolbox (Chow and Cheung, 1992). The system consists of 140 buses, 48 generators, and 92 loads. In this study, we partition the network into six distinct regions, each corresponding to a different Independent System Operator (ISO). These regions define the operational areas in which the COLMENA agents operate. The simulation starts from steady-state operating conditions, where power generation and demand remain balanced. We then introduce a set of disturbances in the form of device disconnections, which drive the system away from stable operation. During MPC execution, agents exchange state estimates and ADMM dual variables using communication channels provided by COLMENA, enabling parallel and coordinated optimization across areas. Once convergence is achieved, control signals—typically generator power setpoints or load curtailment commands—are dispatched locally.

For this use case, we define three roles for the COLMENA agents. The first is the Monitoring role, which continuously collects and publishes local frequency measurements from each area. The second is the DMPC role, which executes the control algorithm, performing local optimization while coordinating with neighboring areas to maintain global consistency. For interacting with the simulation and injecting the contingencies we use an additional role, the Simulation Manager. The KPI is defined as an aggregation of frequency deviations across all areas over a specified time window. When the KPI threshold is exceeded, agents collectively activate the DMPC role using the consensus-based role selection mechanism developed as a COLMENA policy. This role-based implementation allows distributed MPC to coexist with other control strategies, such as local proportional–integral (PI)-like power optimization roles or grid-forming converter control, each operating on different time scales and activation conditions.

The primary contribution of this work lies in demonstrating how advanced distributed optimization-based control can be integrated, activated, and coordinated in real time using COLMENA, alongside more traditional grid control mechanisms.

3.3 Experiment deployment with FABRIC and Kiso

The experiments in this work were conducted on the FABRIC testbed (Baldin et al., 2019), a national-scale, programmable experimental network infrastructure that interconnects heterogeneous compute and network resources across 29 geographically distributed sites, including national laboratories, university campuses, and commercial colocation facilities. FABRIC provides an extensible network service model Ruth et al. (2022) that enables the composition of advanced networking services to support diverse experimental workflows. The testbed offers programmable network elements with P4-capable switches, enabling custom packet processing and network experimentation. Each site provides compute nodes with varying configurations, allowing researchers to provision heterogeneous resources that mirror real-world Compute Continuum deployments.

Experiment configuration, orchestration, and interactive development were performed using two complementary approaches. First, we used the FABRIC Portal and its integrated JupyterHub environment to iteratively design and deploy experiments in an interactive manner, which allows immediate inspection and debugging of resource allocation and workflow execution. Second, we automated the same experiments using Kiso, an open-source framework that provides a lightweight orchestration layer for reproducible execution of complex workflows across heterogeneous edge-to-cloud infrastructures. Kiso enables experiments to be expressed as ordered sequences of labeled tasks with associated commands and target nodes, allowing repeated execution with consistent semantics while minimizing manual intervention.

Experiments in Kiso are described as YAML files, which allows researchers to focus on their experiments, while not worrying about learning the features, terminologies, APIs, and idiosyncrasies of the testbeds on which the resources are to be provisioned. Kiso does three things on behalf of the user, (i) provisions resources on testbeds, (ii) configures resources for the experiments, and (iii) triggers the experiments.

The YAML specification describes four elements: sites, software, deployment, and experiments. The sites section describes a sequence of testbeds and the resources to be provisioned on each of them. Currently, Kiso supports provisioning resources on: Chameleon (Keahey et al., 2020), Chameleon Edge, FABRIC testbeds, and local VMs using Vagrant (HashiCorp, 2025). The software section describes what software to install on some or all of the provisioned resources. Currently, Kiso supports the installation of Docker, Apptainer, and Ollama. The deployment section describes what workload management system to install and how to configure it on the provisioned resources. Out-of-the-box Kiso installs and configures an HTCondor cluster. The experiments section describes a sequence of experiments to run on the provisioned and configured resources. Currently, Kiso supports the running of Shell and the Pegasus (Deelman et al., 2015) workflow management experiments.

The COLMENA experiment is implemented as a Kiso shell experiment as a sequence of labeled tasks, where each task specifies (i) the subset of resources on which it should run, and (ii) the commands or scripts to be executed. Tasks can be executed sequentially or in parallel, and may include synchronization points that ensure certain conditions are met (e.g., successful service startup or completion of a previous phase) before proceeding. This model allows Kiso to coordinate multi-stage experiments spanning multiple sites while preserving clear execution order and dependencies.

The COLMENA experiment workflow is illustrated in Figure 3. In Kiso, this workflow is described using a YAML specification composed of two main sections: resource allocation and configuration (sites), and experiment execution and output collection (experiments). For the COLMENA experiment, we implemented a set of scripts that execute the required commands to block until successful completion, which is verified by detecting specific keywords in Docker logs or text files.

In the sites section, we specify the FABRIC sites to be allocated, including the number of nodes per site and their hardware characteristics. Nodes are also assigned labels, which are later used to map experiment steps to specific resources. The experiments section defines the execution steps by matching these node labels to execution scripts, and it specifies how experiment outputs are collected by defining source and destination paths for each output file.

The experiment itself is composed of several sequential steps. First, all nodes are configured. This setup phase includes installing dependencies, exchanging agent hostnames, configuring Docker and Zenoh, and exporting the required environment variables. Next, a dedicated infrastructure node is used to host centralized services, including the ANDES power system simulation (explained in Section 3.2.2) and a central database that supports the consensus process. Once these services are running, the COLMENA agent stack is deployed on all agent nodes together with the consensus policy. Subsequently, the service is deployed in a single agent; due to the COLMENA deployment mechanism, the service is automatically propagated to all other agents. Finally, the experiment waits for completion and retrieves the resulting data.

This integration of FABRIC and Kiso enables a scalable and repeatable experimental deployment by abstracting infrastructure complexity and allowing commands to be executed in parallel across nodes while preserving the required sequential instruction steps.

4 Results

This study investigates decentralized coordination of autonomous agents across distributed computing resources using the consensus policy. To demonstrate this capability, we designed an experiment showing how agents can dynamically self-organize, share data, and coordinate actions through a consensus policy implemented on the COLMENA middleware. Figure 4 provides a high-level overview of the experiment and its main components.

The experiment consists of allocating nodes at six different sites using the FABRIC testbed: Los Angeles (LOSA), Starlight (STAR), University of Bristol (BRIST), Florida International University (FIU), University of California San Diego (UCSD), and University of Hawai’i (HAWI). We vary the number of agents per area from three (18 agents total) to ten (60 agents total). We perform two types of contingencies, first disconnecting ten loads from the grid (Figures 5a–c) and the second disconnecting two generators (Figures 6a–c), for three agents per area. We then scale the number of agents per area up to ten (Figure 7) and compare the KPI detection time and consensus times to evaluate the scalability (Figure 8), repeating the experiments five times each.

Each node has 4 CPU Cores, 16 GB of RAM and 100 GB of Disk. The networking configuration is as follows: each site in FABRIC has its own network (Layer 3), and all sites are interconnected through a single shared network (also Layer 3). Zenoh is configured with one node per site acting as a backbone router. This node connects the site to other sites via entry points, while all other nodes are connected exclusively to the backbone router of their site, using it as their entry point. Although Zenoh supports self-discovery via multicast, we explicitly configure the topology using unicast entrypoints to avoid unnecessary traffic. Once connected, the routers use gossip to propagate metrics across all nodes in the experiment. All of these settings are specified in the Zenoh router configuration file in COLMENA and can be easily adapted to the deployment. For example, several backbone routers per area could be added to improve resilience, different topologies could be used to interconnect the areas, such as rings or meshes, and self-discovery could be activated in dynamic scenarios.

In each experiment, one node runs the ANDES simulator and the database used by the consensus mechanism (where agents share the state of their resources). This node also runs a COLMENA agent that will run the role that interacts with the simulation. The other nodes run COLMENA agents that emulate operation within a specific area of the electrical grid. To differentiate between agents, we use the hardware variable in COLMENA as a logical capability descriptor that is mapped to the requirements of each role. Specifically, nodes assigned the value SIMULATOR host the ANDES simulator and run the Simulation Manager role, while nodes labeled AREA represent agents operating within a specific control area of the electrical grid; these nodes run the Monitoring role and may also run the DMPC role.

All nodes except the simulator also run the Consensus process, which will run continuously and wait for gRPC messages from the agent. In addition, each agent runs a container that allocates a random amount of RAM to simulate that agents are heterogeneous and have different amounts of available resources. Once the nodes are set up, we select one of the nodes to build and deploy the use case as a service. Images are built and uploaded to DockerHub, and the service will be deployed by sending the service description via Zenoh. When the service is deployed, the Simulator Role advances the simulation, which will perform the disconnection of ten loads or two generators in the NPCC 140-Bus system, depending on the experiment. Simultaneously, each agent within an area will run the Monitoring role, which publishes the frequency value of their area. When the KPI is broken, agents then self-organize using Consensus to run the Optimizer role. The distributed MPC optimization algorithm is then executed by six agents (one per area) to respond to the contingency and bring the grid frequency back to stable values.

Figure 5a illustrates the KPI evolution in an experiment with three agents per area. The KPI represents the absolute difference between the nominal and actual frequency over each 5 time window and should remain below 0.001. Following the disconnection of ten loads, the grid frequency exceeds this threshold, entering a broken state (red region). After an average of 4.04 s, the agents detect the violation and initiate leader selection, during which the selected leaders execute the frequency optimizer role. The consensus process lasts, on average, 2.39 s, after which the distributed optimization algorithm restores the frequency to acceptable levels by computing optimal setpoints and interacting with the simulation. The frequency returns to within the 0.001 band in 26.82 s, representing a significantly faster restoration process compared to conventional secondary frequency control schemes, which typically require up to 15 min for full deployment (Rebours et al., 2007). The results obtained from ANDES are presented in Figures 5b,c, corresponding to the cases without and with COLMENA, respectively.

A similar experiment is depicted in Figure 6a, in which two generators are disconnected. These generators correspond to GENROU_46 and GENROU_38, which have the highest inertia in the grid. On average, 6.17 s after the contingency, the agents detect the KPI violation and initiate the consensus process, which lasts, on average, 1.26 s. In response, the DMPC roles perform load curtailment to shed loads and restore the frequency to its desired range, a process that takes 40 s. The conventional response shown in Figure 6b is insufficient, as the frequency remains below 0.998. In contrast, the impact of deploying DMPC and COLMENA is illustrated in Figure 6c, where the frequency is successfully restored to values above 0.999.

The KPI detection and consensus times for the agents assigned the optimizer role (10, 13, 29, 34, 43, and 53) are shown in Figure 7, considering the case with ten agents per area. Detection times range from 5.79 s to 7.92 s, closely matching the 5 s KPI aggregation window. Consensus times range from 1.44 s to 2.59 s. Consensus times range from 1.44 s to 2.59 s. Figure 8 shows the KPI detection time and consensus time (mean and variance) varying the number of agents per area from 3 to 10. The number of areas is fixed at six; therefore, the total number of agents ranges from 9 to 60. The mean detection time increases from 6.93 ± 0.11 s (3 agents per area) to 7.84 ± 0.69 s (10 agents per area), while the mean consensus time increases from 1.52 ± 0.014 s to 1.91 ± 0.004 s. The consensus time increases only slightly, since the consensus process has to deal with more votes and messages. The KPI Detection time increases more because the metric aggregation in COLMENA is done over all agents (60), and for the same KPI window, it introduces extra latency.

Figure 9a shows CPU and memory usage over an experiment. COLMENA runs from t = 0, with CPU usage below 1 % until service roles are executed. CPU usage rises when the Monitoring role is activated, followed by the Optimizer role after the KPI breach (green dashed line), but remains below 7.5 %. Memory usage stays largely constant around the pre-allocated baseline. In this experiment, Agent 3 was selected as the leader due to its minimal memory usage (which will have a lower cost function, Equation 1). Baseline memory was subtracted to evaluate COLMENA’s memory usage, showing stable values around 1.5 %, which corresponds to 240 MB (Figure 9b).

Overall, these results demonstrate that COLMENA can effectively support the decentralized coordination of event-driven applications in hyper-distributed environments with low overhead and predictable latency using a distributed orchestration. The consensus mechanism enables agents to reliably self-organize in response to QoS violations, even as the number of participating agents increases, while maintaining bounded consensus times and minimal resource consumption. Importantly, the integration of KPI-driven triggering with cost-aware leader selection allows coordination decisions to be both context-aware and resource-efficient. Together, these results validate the feasibility of embedding consensus directly into the service operation to support adaptive, scalable role-based coordination in geographically distributed systems.

5 Discussion

In this work, we present a decentralized coordination framework for autonomous agents in hyper-distributed environments using a consensus-based role selection approach. Leveraging the COLMENA programming environment, agents are able to dynamically self-organize, monitor application-specific indicators, and autonomously decide which roles to execute based on their capabilities and real-time QoS indicators (i.e., KPIs). We extended a PBFT-inspired consensus mechanism, originally designed for job selection, to support role selection, allowing agents to elastically respond to QoS violations. The integration of this mechanism into COLMENA enables fully decentralized orchestration of services across the Compute Continuum.

To evaluate the effectiveness of our approach, we implement a realistic power grid use case focused on frequency optimization in the NPCC 140-bus system. We show that COLMENA agents deployed across six geographically distributed FABRIC sites monitor local frequency deviations and leverage consensus to elect leaders responsible for executing a DMPC algorithm. Our experiments with up to 60 agents demonstrate that agents effectively detect contingencies, coordinate role execution, and restore grid frequency to acceptable ranges. KPI detection and consensus times remain low even as the number of agents increases, and CPU and memory usage show minimal overhead, illustrating the scalability and efficiency of the approach.

Agent coordination and consensus are realized through direct peer-to-peer and broadcast communication using gRPC. All coordination messages are agent-initiated, and no centralized controller or arbiter exists. Each agent independently interprets received messages and executes local decision logic. The system includes a global database used exclusively for sharing resource availability and persisting execution results. This database does not participate in coordination, consensus, or decision-making. Its role is limited to state dissemination and experimental convenience, and it can be replaced by decentralized dissemination mechanisms such as gRPC-based heartbeats, gossip protocols, or publish–subscribe messaging.

Our results highlight several implications, described in the following subsections.

5.1 Role-based services

At the application level, role-based programming complements traditional workflow systems by enabling the adaptive execution of hyper-distributed services, while automatic control strategies remain better suited to structured and predictable computations. Within this framework, roles can be directly associated with classical power-system control layers, including local stabilizing functions analogous to primary control and area-level coordination corresponding to secondary control. Unlike conventional fixed control hierarchies, these roles are not statically assigned. Instead, when the specific roles are activated, agents dynamically send control signals based on level KPIs derived from physical grid measurements, such as frequency deviation or voltage constraint violations. This role-based control paradigm enables more precise and inherently distributed control strategies, in which control responsibilities and actions are continuously adapted to local system conditions while remaining aligned with area-wide operational objectives.

To validate this claim, we expressed a use case from modern power systems as a set of role definitions, abstractions, control signals defined for the electrical devices and service-level KPIs. We showed that the agents themselves dynamically start and stop roles based on the KPI values observed during service operation, allowing the system to adapt continuously throughout the experiments. This adaptive role management enables continuous reconfiguration of control responsibilities without relying on static control strategies or a centralized decision making. Additionally, the results demonstrate that role-based control mechanisms can coexist with conventional automatic control strategies, with each addressing different temporal and structural aspects of the control problem. Overall, this work contributes to a deeper understanding of the intersection between traditional workflow models and swarm-like service compositions, highlighting how role-based abstractions can bridge structured orchestration with adaptive, decentralized execution.

5.2 Coordination using consensus

From a coordination perspective, consensus-based coordination provides a robust backbone that can be extended with more advanced decision-making techniques, including predictive or learning-based policies, to collaboratively meet QoS in complex distributed systems. Rather than being tied to a specific algorithmic choice, consensus acts as an enabling mechanism that allows heterogeneous decision-making strategies to coexist within a unified coordination framework. These strategies may include bio-inspired algorithms (Kar, 2016), multi-agent reinforcement learning methods (Canese et al., 2021), device-first AI (Alamouti et al., 2025) and agentic AI based on large language models (Acharya et al., 2025). More broadly, this suggests that consensus can be treated as a reusable coordination primitive in swarm computing, applicable to multiple stages of collective behavior beyond the specific scenario considered in this work.

Decision-making strategies can also be integrated within choreography-based approaches, where agents act autonomously without participating in blocking agreement protocols (Islam et al., 2025). Rather, they act autonomously based on local intelligence, share intent and knowledge asynchronous, and coordinate through emergent alignment. These systems complement consensus-based coordination for use cases where latency is critical and scenarios are extremely large-scale (hundreds of thousands of nodes). In those scenarios, the O ( n 2 ) message complexity of PBFT-based protocols could become a limitation. Consensus-based coordination, however, remains essential when systems require agreement on specific role executions, strong consistency guarantees, or robustness against crash or Byzantine failures.

5.3 Towards decentralized power systems

The presented approach is highly relevant to modern decentralized power grids, where distributed renewable and inverter-based resources require rapid, localized responses without relying on centralized control. By enabling adaptive role execution and consensus-based coordination, the framework supports coordinated yet autonomous responses to frequency deviations, voltage fluctuations, and other operational contingencies. This is particularly important in grids with high penetrations of inverter-based resources, microgrids, and prosumers, where traditional centralized control is either too slow or impractical. Moreover, the modular role-based design allows different control strategies—such as DMPC, local PI controllers, and grid-forming converter control—to coexist and interact seamlessly, each operating on its own timescale and activation conditions. These capabilities demonstrate that decentralized coordination mechanisms can maintain system stability while reducing communication overhead and computational bottlenecks, highlighting the potential for scalable, resilient, and adaptive power system management in future energy networks.

Translating the presented use case to a real scenario in the electrical grid would involve additional challenges. In this scenario, sensors measure monitoring values and publish them into the platform, and computational nodes react to contingencies. Here additional mechanisms can be implemented that ensure data quality, integrity and trust, such as multi-source data fusion (Zhang, 2010).

5.4 Experimental automation with Kiso

This work benefited from the use of Kiso as an experiment automation substrate. Kiso enabled the end-to-end deployment, configuration, execution, and result collection of the distributed experiments presented in this paper to be expressed declaratively and executed reproducibly across heterogeneous FABRIC sites. By automating multi-stage deployment workflows and verifying the successful completion of each stage before proceeding, Kiso reduced manual orchestration effort, limited configuration drift across runs, and improved robustness in the presence of transient failures and partial misconfigurations. This allowed our experiments to be repeatedly deployed as the number of agents, sites, and execution parameters vary, supporting systematic and controlled evaluation of decentralized coordination strategies in the Compute Continuum.

5.5 Conclusion

In summary, this work illustrates that combining decentralized consensus mechanisms with role-based service orchestration provides a scalable, resilient, and adaptable framework for managing complex, hyper-distributed systems. The methods, results, and insights presented here open avenues for integrating adaptive multi-agent coordination into other domains within the Compute Continuum, from scientific workflows and edge-cloud applications to large-scale infrastructure management, thereby bridging the gap between traditional workflows and fully autonomous distributed services.

Correction note

A correction has been made to this article. Details can be found at: 10.3389/fcpxs.2026.1919371.

Generative AI statement

The author(s) declared that generative AI was used in the creation of this manuscript. ChatGPT 5 was used only to review grammar and clarity during the manuscript preparation.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Rodrigo Picos, University of the Balearic Islands, Spain

Reviewed by: Abdorasoul Ghasemi, Coventry University, United Kingdom

Siavash Alamouti, Mimik Technology Inc., United States