The co-evolution of costly markers and cooperation in social dilemmas

Costly cooperation and the evolution of costly signals are both difficult to reconcile with simple fitness maximization, yet both are common in biological and social systems. We study a model in which agents emit costly signals and condition their actions on the signals they observe. The signals are arbitrary, non-costless markers: they need not reveal hidden biological quality, and their value can arise through the heritable response rules associated with them. Across the Prisoner’s Dilemma (PD), Snowdrift (SD), and Stag Hunt (SH) games, we ask when this coevolutionary process can sustain cooperation and how it changes across well-mixed populations, spatial lattices, and fluctuating strategic environments. The simulations show that signals are selected less by their raw production costs than by the cooperative responses they currently elicit. No-signal baseline controls show that the signal response layer is essential for maintaining cooperation in the PD and under the SD update rule studied here, whereas SH already favors cooperative coordination. In well-mixed populations, the mechanism sustains partial cooperation in PD and SD and drives near-complete cooperation in SH. On lattices, cooperation is strengthened further by local assortment. An independence-based mean-field analysis reproduces the qualitative SD and SH regimes but not partial cooperation in the PD. A separately parameterized, phenomenological rarity correction is sufficient to produce a partially cooperative PD regime, illustrating one possible role for omitted frequency-dependent correlations without identifying them uniquely. Our results therefore delineate a class of settings in which arbitrary costly signals persist because they transiently organize cooperative responses and thereby reshape the effective strategic environment.

Cooperation and costly signaling are usually introduced as separate evolutionary puzzles. Cooperation is difficult to explain because a cooperator pays a cost that can be exploited by freeriders, while costly signaling is difficult to explain because the signal itself consumes resources and may increase risk. Both therefore look incompatible with simple fitness maximization (Hamilton, 1964; Trivers, 1971; Zahavi, 1975; Axelrod and Hamilton, 1981; Grafen, 1990; Lehmann and Keller, 2006; Nowak, 2006; Rand and Nowak, 2013). Yet both are widespread in biological and social systems, which suggests that the relevant selective pressures are often more structured than a direct cost argument would imply. Here we ask whether an arbitrary signal that is costly to produce can help stabilize cooperation without encoding hidden biological quality, simply by becoming coupled to heritable response strategies.

Quality-based signaling models explain reliability through links among signal cost, condition, and differential ability to bear the cost (Zahavi, 1975; Grafen, 1990; Johnstone and Grafen, 1993). Trade-off approaches emphasize more generally that selection acts on the balance between signaling costs and the benefits generated by receiver responses (Lachmann et al., 2001; Számado, 2011; Számado et al., 2023; Számado et al., 2026). These approaches are compatible with the present model but address a different source of signal value. Our model contains no hidden quality variable, stable cooperative type, or differential ability to pay. Its signals are observable costly markers whose effects are created by coevolving response rules, placing the mechanism alongside work on coordination markers, tag-based assortment, and endogenous conventions (Riolo et al., 2001; McElreath et al., 2003; Jansen and van Baalen, 2006; Efferson et al., 2008; Chen and Li, 2009).

The question is therefore not whether cost certifies quality, but whether the cooperation elicited by a marker can offset its production cost. In related applications, costly prosocial acts may reveal quality, commitment, or trustworthiness (Roberts, 1998; Gintis et al., 2001; Barclay and Willer, 2007; Fehrler and Przepiorka, 2013; Raihani and Smith, 2015); in the setting studied here, signal value instead emerges from the response convention attached to the marker. A strategic-conflict perspective likewise emphasizes that signalers and receivers can coevolve rather than transmit fixed information (Dawkins and Krebs, 1978).

For the evolution of cooperation, diverse mechanisms have been suggested. Kin selection can promote cooperation by channeling benefits toward relatives (Hamilton, 1964). Direct reciprocity can reward past cooperation in repeated interactions (Trivers, 1971; Axelrod and Hamilton, 1981). Reputation-based indirect reciprocity can also stabilize cooperation (Nowak and Sigmund, 1998; Nowak and Sigmund, 2005), and a substantial literature has examined the assessment rules (Panchanathan and Boyd, 2003; Ohtsuki and Iwasa, 2004) and social norms (Salahshour, 2022) required to sustain such systems. Population structure can protect clusters of cooperators from invasion (Nowak and May, 1992; Nowak, 2006; Ohtsuki and Nowak, 2006), while multilevel selection can favor cooperation when more cooperative groups outperform less cooperative ones (Traulsen and Nowak, 2006). Sanctioning can also sustain cooperation, through prosocial punishment (Fehr and Gächter, 2002; Salahshour, 2021a) and reward-based incentives (Cressman et al., 2012). Signaling, including language (Salahshour, 2020), tag-based recognition (Riolo et al., 2001; Jansen and van Baalen, 2006), and socially shared or symbolic markers (McElreath et al., 2003; Efferson et al., 2008; Chen and Li, 2009) can also promote cooperation. Other mechanisms can act independently as well: voluntary participation and commitment (Hauert et al., 2002; Song and Han, 2025), partner choice (Noë and Hammerstein, 1994; Barclay and Willer, 2007; Fu et al., 2008), freedom of choice between public resources (Salahshour, 2021b), heterogeneity in institutions and personalities (Salahshour, 2021c; Salahshour, 2021d), and ecological resource flows (Salahshour, 2023) can each support cooperation. Recent work further suggests that rational decision-making itself can promote cooperation through perceptual rationality (Salahshour, 2025) and rational reciprocity (Salahshour and Couzin, 2025).

Recently, it has also been suggested that costly signaling can provide an avenue for resolving conflict in strategic interactions and, in this way, promote cooperation (Salahshour, 2019). This view builds on, and refines, the evolutionary arms-race perspective of costly signals (Dawkins and Krebs, 1978). Rather than treating signals only as attempts to manipulate receivers, it emphasizes that signals can acquire value because they help organize social responses in a population where behavior is conditioned on observable cues (McNamara et al., 2009; Baker et al., 2012). In that sense, it is closer to literatures in which markers become meaningful through shared expectations and coordinated responses than to models in which signals matter only by revealing hidden type (McElreath et al., 2003; Efferson et al., 2008; Chen and Li, 2009; Salahshour, 2020). In the model introduced by Salahshour (2019), signals have no fixed meaning and need not reveal an underlying quality. Their meaning emerges endogenously through the response conventions attached to them. A costly signal is favored when it elicits cooperation, but this advantage is temporary because successful signals also attract exploitation and eventually lose their effectiveness. We use the phrase “signaling war” in this specific sense: signal classes rise when they elicit cooperation, become vulnerable as they become common and exploitable, and are then replaced by other signals that temporarily receive more favorable responses. The result is a moving coevolutionary process in which costly signals can help create, stabilize, and then erode temporary cooperative niches. This is the mechanism whose scope we revisit here.

We build on the well-mixed model introduced by Salahshour (2019). That study showed that costly signals and partial cooperation can coevolve when agents condition behavior on signals, generating a moving “signaling war” in which profitable signals spread until they attract exploitation. The earlier formulation allowed strategies to depend on both the focal individual’s own signal and the opponent’s signal (Salahshour, 2019), whereas here we restrict attention to the simpler case in which responses depend only on the opponent’s signal. This simplification is empirically motivated by cases in which the strategically relevant cue is carried primarily by the partner being assessed. In humans, for example, trust and cooperation can depend on observed group markers and perceived trustworthiness (Chen and Li, 2009; Fehrler and Przepiorka, 2013). The model is most directly relevant when the observed cue is a socially learned badge, style, ritualized display, group marker, or linguistic cue that affects how others respond, and less directly relevant when the signal is known to reveal stable individual quality. More generally, both human and animal work have examined cases in which individuals condition their responses on social information about others, including observed cooperative tendencies and cues extracted from prior interactions or conflicts (Nowak and Sigmund, 1998; McNamara et al., 2009; Baker et al., 2012). Focusing on the opponent-conditioned rule therefore isolates the receiver-side discrimination mechanism that is central to the present argument. However, it is still unclear whether the same mechanism extends across distinct social dilemmas, how spatial structure changes its stability, and what ingredients are needed for a reduced mathematical account of why it works in some games but not in others.

Here, we address those gaps. We examine the coevolution of signals and cooperation across the three canonical 2 × 2 social dilemmas: the Prisoner’s Dilemma (PD), which belongs to the dominance class; the Snowdrift game (SD), which belongs to the anti-coordination class; and the Stag Hunt (SH), which belongs to the coordination class (Szabó and Fáth, 2007; Płatkowski, 2017; Salahshour and Couzin, 2025). Together, these games span qualitatively distinct strategic settings within symmetric two-player, two-strategy interactions and therefore provide a compact test of whether the signaling mechanism depends on a specific payoff structure or carries across strategically different contexts. We then extend the analysis in two directions. First, we study a spatial lattice, where local clustering can interact with signalbased partner discrimination. Second, we consider fluctuating strategic environments in which the operative game changes probabilistically across encounters. This extension is motivated by the fact that, in realistic settings, social interactions often unfold across multiple institutional or ecological contexts rather than within a single fixed game (Salahshour, 2021b; Salahshour, 2022). Across these settings, one empirical pattern keeps returning. Signals are not sorted by production cost alone; they rise and fall according to the cooperative responses they elicit.

The results show that the same signal-response process can sustain partial cooperation in PD and SD, but near-complete cooperation in SH, and that spatial structure strengthens cooperation further. The reduced analysis also clarifies an important asymmetry across games. An independence approximation reproduces the qualitative SD and SH regimes, but it cannot sustain cooperation in the PD. A phenomenological rarity-dependent correction provides one sufficient example of how omitted frequency-dependent correlations could generate partial cooperation in the PD; it is neither a unique closure nor independently fitted to the agent-based simulations.

The central claim is therefore deliberately narrow. The model shows how an arbitrary signal that is costly to produce can support cooperation without a hidden quality variable, because signal production coevolves with heritable response rules (Riolo et al., 2001; Jansen and van Baalen, 2006; Salahshour, 2019). Quality-based and strategic-conflict accounts remain compatible explanations for systems with different biological assumptions (Zahavi, 1975; Dawkins and Krebs, 1978; Grafen, 1990; Gintis et al., 2001).

We consider a population of N agents who repeatedly interact in a one-shot two-player game. Each agent α carries two inherited components. The first is a signal-production distribution

where p α 1 is the probability that agent α emits signal σ i. Each signal σ i has a fixed production cost c i ∈ [0,c max ], drawn once at the start of the simulation from a uniform distribution on [0,c max ] and then kept fixed for all agents and all generations. The second component is a conditional response rule

so that s α (σ i) = C means that agent α cooperates after observing signal σ i from its opponent. In the present model, actions depend on the opponent’s signal only. This is simpler than the formulation in (Salahshour, 2019), where behavior depended on both the focal agent’s own signal and the opponent’s signal. At initialization, each agent’s signal-production vector is drawn independently from a symmetric Dirichlet distribution with concentration parameters (1,…,1), and each entry of its response rule is independently set to C or D with equal probability.

Interactions and payoffs

Agents interact and accumulate payoffs according to one of two population structures.

In a match between agents α and γ, each player emits a signal sampled from its own distribution, observes the opponent’s signal, and then chooses an action according to its response rule. The focal agent’s payoff is the game payoff associated with the action pair, minus the cost of the focal agent’s own emitted signal. The total payoff w α is the sum of net payoffs over all interactions in that generation. For lattice populations, the score used in the exponential fitness weight is divided by the four lattice interactions, so that well-mixed and structured populations are compared on a per-interaction payoff scale under the same β. The interaction structure is one of the following:

Well-Mixed Population: In each generation, N agents are randomly paired to play a single-shot, two-player game. Each agent participates in exactly one interaction per generation.

Structured Population (Lattice). Agents occupy the vertices of a two-dimensional L × L square lattice (N = L 2) with periodic boundary conditions. In each generation, every agent plays a game with its four immediate von Neumann neighbors.

These structures serve as two benchmark interaction regimes. Random matching removes persistent neighborhoods and provides a null case without stable local assortment. The periodic square lattice is a minimal regular spatial network: every individual has the same degree, while repeated local interactions allow clustering and local signal-response conventions. We use these two cases to isolate the contrast between global mixing and regular local assortment, not to claim that the results are independent of network architecture.

The encounter schedules differ by construction. A well-mixed individual receives one realized game payoff per generation, whereas a lattice individual plays all four incident edges and receives the mean net payoff across those four encounters. Dividing the lattice total by four prevents degree from multiplying the scale of selection under exp(βw), so β acts on a perinteraction payoff in both structures. This normalization does not make the stochastic sampling identical: the lattice score averages four encounters and therefore has less encounter-level sampling variance than the single-match well-mixed score. Comparisons between the two structures should consequently be interpreted as comparisons between these explicit interaction protocols, with local assortment as their principal topological difference.

In two-player social dilemmas, the four standard payoffs are: R (reward for mutual cooperation), S (sucker’s payoff received by a cooperator when the opponent defects), T (temptation payoff to a defector against a cooperator), and P (punishment for mutual defection). The ordering of these values defines the type of game (e.g., Prisoner’s Dilemma, Snowdrift, Stag Hunt).

The payoff values used here are summarized in Table 1.

Selection and mutation

The population evolves through Wright-Fisher replacement. After payoffs are collected, the entire population is replaced by a new generation of N offspring.

Reproduction: Each site in the new generation is filled by an offspring from a parent selected from the previous generation. In the well-mixed model, any individual in the population can be chosen as a parent. The probability that agent α is selected is proportional to its fitness,

where w α is its total payoff and β is the intensity of selection. The corresponding selection probability is

In the lattice model, reproduction is local: the parent of a site is chosen from the focal agent and its four nearest neighbors, with the same exponential fitness weighting but normalized over that local neighborhood. Local fitness weights use the mean payoff per neighbor interaction rather than the unnormalized four-game total.

Inheritance and Mutation: An offspring inherits the complete genome (p parent,s parent) of its chosen parent, after which mutations may affect signals and strategies independently.

TEXTS

– Signal Mutation: With probability ν σ, a random signal index i is chosen and the offspring’s signal-production distribution is shifted toward σ i by adding d to that component and then renormalizing. This implements gradual change in signaling propensity.

– Strategy Mutation: With probability ν s, the offspring’s response rule is mutated at n s randomly chosen entries, and each selected entry is flipped from C to D or from D to C. Strategy mutation therefore explores the space of conditional responses more abruptly than signal mutation explores the signal distribution. This asymmetry follows from the representation used here: signal production is a continuous probability distribution over signal classes, whereas receiver response is encoded as a binary action for each signal.

Figure 1a summarizes signal production, observation, action choice, and payoff accumulation; Figure 1b shows fitness weighting, reproduction, inheritance, and mutation. The default parameter values used throughout the paper are listed in Table 2.

We used independent stochastic replicates for all aggregate simulation results. Fixed-game action and outcome trajectories and the signaling and no-signal comparisons use five runs for each game, population structure, and condition. Fluctuating-environment sweeps use ten runs for each value of q PD in each game mixture and population structure. Supplementary Table 1 summarizes these samples and the uncertainty shown in the figures. Each run lasts 2 × 10 4 generations, with statistics recorded every 20 generations, giving 1,000 samples from generation 0 to generation 19,980. Fixed-game trajectory figures report means across runs, with standard-error whiskers at selected time points. Fluctuating-environment figures report means and standard errors after averaging each run over its last 200 recorded samples (generations 16,000–19,980). Signaling and no-signal comparisons use the same late-time summary, while mean signal-cost trajectories use standard-error bands. Signal-class scatter plots, signal-usage heatmaps, dominant-signal-rank traces, and final spatial snapshots show individual runs because signal identities and lattice configurations cannot be aligned across replicates.

To isolate the effect of the signal-response layer, we also simulated no-signal populations with the same Wright–Fisher replacement schedule, payoff matrices, β, and per-offspring strategy mutation probability. Individuals in these populations carry one unconditional binary action, initialized as C or D with equal probability; a strategy mutation flips that action, and no signal cost is paid. Because selection is applied to realized one-round payoffs through the exponential weight exp(βw), the no-signal well-mixed population need not converge to the mixed Nash equilibrium of the expected-payoff game. For example, in the SD payoff matrix used here, the classical expected-payoff mixed equilibrium is x C = (S − P) / (T − R + S − P) = 1 / 3, whereas the update used here equalizes expected reproductive weights, x C e β R + (1 − x C) e β S = x C e β T + (1 − x C) e β P. For (R, S, T, P) = (3, 1, 5, 0) this gives x C = e β S − e β P e β S − e β P + e β T − e β R ≈ 0.013 at β = 1. The SH population should be interpreted differently. For the SH payoff matrix (R, S, T, P) = (5, 0, 3, 1), the classical expected-payoff mixed equilibrium is the unstable coordination threshold x C = (P − S) / (R − T + P − S) = 1 / 3. Under this exponential update, the corresponding indifference threshold is x C = e β P − e β S e β R − e β S − e β T + e β P ≈ 0.013 at β = 1. Since the random initial condition is far above this threshold, the no-signal SH population is expected to approach the cooperative equilibrium.

We organize the results around four questions: how the mechanism behaves in the three canonical games, how it changes in a spatial population, how it responds to fluctuating strategic environments, and what determines which signals remain prevalent.

Canonical games in well-mixed populations

Evolution of cooperation

We begin with well-mixed populations. In the PD, cooperation remains well above zero but defection is more common (Figure 2a), and asymmetric CD interactions account for the largest share of outcomes (Figure 2b). The SD shows the same broad pattern: cooperation persists without taking over (Figure 2c), while CD outcomes remain prevalent alongside DD (Figure 2d). In both games, signaling therefore stabilizes a heterogeneous regime in which cooperation persists but remains exposed to exploitation.

The SH behaves differently. In Figure 2e, cooperation rises quickly toward fixation, and Figure 2f shows the corresponding dominance of CC. The contrast with PD and SD is mechanistically informative. Once SH is pushed across its coordination threshold, signaling no longer has to hold cooperation against persistent exploitation. Instead, it mainly helps the population settle on the cooperative equilibrium. These results show that the same signal-response mechanism leads to different social outcomes depending on the underlying game: partial cooperation in PD and SD, but near-complete coordination in SH.

The no-signal controls sharpen this interpretation. Under the same replacement and mutation schedule, the unconditional-action baseline collapses to near-zero cooperation in PD and remains much lower than signaling in SD, whereas SH reaches cooperation even without signals. This SH outcome is consistent with the Stag Hunt structure: the no-signal runs begin above the unstable mixed threshold and are therefore drawn toward the cooperative equilibrium. The low SD baseline reflects the realized-payoff exponential update used here, not the classical expected-payoff Nash value of the underlying SD matrix. Thus, signaling has its clearest dilemma-mitigating role in PD, changes the realized SD outcome under this update rule despite the anti-coordination structure of the game, and acts mainly as a coordination aid in SH. This is why we interpret the mechanism as signal-mediated convention and assortment rather than as a general effect of signal cost alone.

In Supplementary Figure 1, we show that this strategic pattern is robust but not rigid. Across most parameter changes, the well-mixed PD remains in a partially cooperative regime with substantial weight on asymmetric CD interactions. The clearest departures occur when the reward for mutual cooperation is increased, which pushes the system toward CC, and when selection becomes too strong, which makes exploitative responses more effective and raises DD. Lower strategy mutation and moderate signal mutation are especially favorable because they preserve useful response rules while still allowing the population to discover profitable new signals. See Supplementary Information, Section S1, for more detail.

Evolution of costly signals

When we first look at signal frequency as a function of cost, the familiar puzzle of costly signaling is immediately visible. In the PD, signals with nonzero costs reach appreciable frequencies (Figure 3a), but the corresponding fitness panel shows that more frequent signals tend to have higher average net payoffs (Figure 3b). The SD (Figures 3c, d) and SH (Figures 3e, f) show the same broad contrast between cost and net payoff. The dynamics therefore cannot be reduced to simple minimization of signaling cost. Costly signals persist when the cooperative returns they generate are sufficient to offset their production costs.

Each point in Figures 3a–f is one signal class, with usage and fitness averaged over the 1,000 recorded samples from generation 0 to generation 19,980. The corresponding mean signal-cost time series add a complementary view. Average signal cost does not simply fall to zero in PD or SD, but it remains much lower in SH once coordination is established. Supplementary Figure 11 shows the corresponding variance in used signal costs and the cost of the currently dominant signal through time. This supports the trade-off interpretation: costs are paid when the current response convention makes them worthwhile, not because waste is favored for its own sake.

The differences across games are also informative. In PD and SD, strategic conflict remains relatively strong. Signals that attract cooperation also create continuing opportunities for exploitation, which is reflected in the persistent weight of asymmetric CD outcomes in Figures 2b, d. No signal-response convention remains secure for long. As a result, selection repeatedly shifts toward newly favorable signals, and both signal fitness and signal frequencies remain comparatively dispersed in Figures 3a–d. The resulting turnover is visible in Supplementary Figure 4, where the dominant signal bands continue to move over time. The same turnover is summarized by the dominant-signal-rank time series. This continuing movement in PD and SD is what we refer to as the “signaling war.

In SH, by contrast, the conflict is weaker. Once a cooperative convention becomes common, mutual cooperation is more attractive than exploiting it, as seen in the dominance of CC in Figures 2e, f. The incentive to escape exploitation by shifting toward new, weakly established signals therefore does not exist. Instead, signals mainly serve to coordinate behavior on an already favorable equilibrium. Consequently, signal frequency becomes concentrated and tends to settle on a narrower, cheaper, and fitter part of signal space in Figures 3e, f.

Cost therefore matters, but not in isolation. Its evolutionary effect is mediated by the response convention attached to the signal and by the strategic conflict surrounding that convention. Consequently, signal fitness, defined as the net benefit of emitting a signal (benefit minus cost), is often a better predictor of signal frequency than its production cost alone.

In Supplementary Figures S2, S3, we show that this conclusion survives parameter variation. The strongest regularity is that signal prevalence follows signal fitness more reliably than it follows signal cost. As ν σ rises from very small values, the dependence of prevalence on cost first sharpens because the population explores signal space more effectively. At very high ν σ, however, the cost relation weakens again even though the fitness relation remains strong. The reason is that signals then turn over faster than strategies, so the response landscape changes only slowly relative to signal frequencies. Selection therefore sorts signals mainly by their current net payoff, while rapid mutation continually reintroduces both cheap and costly variants and blurs any simple dependence on cost alone. At the opposite extreme of very slow signal mutation, exploration is too weak and incumbency matters, so cost again becomes a poor predictor of prevalence. See Supplementary Information, Section S2, for details.

Canonical games in structured populations

Evolution of cooperation

To examine how signaling interacts with network reciprocity, we repeated all three canonical games on a 40×40 lattice (see Supplementary Figure 6 for parameter dependence). Figure 4a shows cooperation and defection in structured PD, and Figure 4b shows the corresponding outcomes. Figures 4c, d show the same quantities for SD. In both games, cooperation rises substantially, CC becomes more common, and DD is reduced. Figures 4e, f show that cooperation and CC dominate in SH, so spatial structure mainly stabilizes rather than transforms that outcome. Because lattice fitness is based on mean payoff per neighbor interaction, this comparison is not driven by a fourfold increase in payoff scale. The direct late-cooperation comparison shows that structured signaling yields higher cooperation than well-mixed signaling in PD and SD, while the no-signal structured populations remain much less cooperative in those two games.

These results are consistent with network reciprocity and strategic signaling reinforcing one another, although the mechanism is not simply additive. In the classical snowdrift game without signaling, spatial structure often inhibits rather than promotes cooperation (Hauert and Doebeli, 2004). The opposite pattern here suggests that once signaling is coupled to action, space changes more than encounter frequencies. Local interactions can allow signal-response conventions to persist, so that agents repeatedly encounter neighbors who attach similar responses to locally common signals. At cluster boundaries, exploitative or mismatched neighbors may be less likely to trigger the same response. Population structure may therefore promote cooperation in SD not by clustering unconditional cooperators, but by stabilizing local coordination over who is treated as a cooperative partner. Representative final 40×40 lattice maps show that the structured runs do not end in a completely homogeneous state in PD and SD. The spatial pattern is weaker than a full domain analysis, but it is consistent with the interpretation that local response neighborhoods preserve partially cooperative conventions.

Evolution of costly signals

Figure 5a plots structured-PD signal usage against cost, and Figure 5b plots usage against fitness. Figures 5c, d show the corresponding SD relationships, followed by SH in Figures 5e, f. As in the corresponding well-mixed panels in Figures 3a–f, more frequent signals tend to be cheaper and to have higher average fitness. But the lattice clouds are noticeably more scattered, especially in PD and SD. In structured populations, a signal’s success can depend not only on its population-wide strategic value but also on which neighborhood carries it and which local response rules co-occur with it. Signals may therefore persist with successful local lineages even when they are not globally strongest, broadening the structured distributions. In Supplementary Figure 5, the structured PD and SD heatmaps show patchier but longer-lived signal bands than in the well-mixed case, whereas SH settles rapidly onto a narrow low-cost band. This pattern is consistent with local neighborhoods prolonging competition among alternative signals in PD and SD while reinforcing a more uniform cooperative convention in SH once coordination is established.

In Supplementary Figure 6, we explore the effect of parameter values and show that higher cooperation than in the well-mixed case is observed across most of the explored space.

Adaptation in fluctuating strategic environments

So far, we have focused on fixed strategic environments. In many biological and social settings, however, the strategic problem itself can vary over time (Salahshour, 2021b; Salahshour, 2022). To investigate that possibility, we let the operative game change probabilistically in each encounter. Figures 6a, b show the well-mixed PD–SD and PD–SH transitions, respectively, followed by the corresponding structured transitions in Figures 6c, d. In these simulations, the game for each interacting pair is drawn randomly between two specified games with the prescribed probability of playing PD. The draws are independent across encounters in the present analysis; temporal autocorrelation in the environmental state is not included here and remains a natural extension.

In well-mixed PD–SD environments, cooperation changes little as the probability of PD increases (Figure 6a). The two component games support similar partially cooperative regimes, so changing the mixture alters the quantitative balance of payoffs more than the overall signalresponse pattern. The well-mixed PD–SH transition is different (Figure 6b): cooperation is close to one when SH is common and declines gradually as PD becomes more frequent.

The structured PD–SD transition follows the same broad pattern at a higher level of cooperation, with a modest decline as the probability of PD rises (Figure 6c). The structured PD–SH transition also declines gradually, but the lattice maintains appreciably more cooperation deeper into the PD region (Figure 6d). Thus, PD–SH mixtures move between two qualitatively different uses of signaling. In SH, signaling mainly helps select and stabilize a cooperative convention that is already payoff-favored. In PD, conditional responses help shield cooperation from exploitation.

The results for fluctuating strategic environments therefore suggest that signal-mediated cooperation is adaptable rather than finely tuned, and that this adaptability survives the addition of spatial structure. When the component games favor similar social states, as in PD-SD, the aggregate outcome is stable because the same partially cooperative signal-mediated regime is effective in both. When they pull in different directions, as in PD-SH, the population shifts continuously between convention selection and dilemma mitigation. The absence of a sharp jump in either topology shows that the same evolving signal system can move smoothly across changing strategic conditions rather than requiring a separate signaling regime for each game.

Simulation controls and signal dynamics

Figure 7a compares late cooperation in signaling and no-signal populations across games and population structures. Figure 7b shows mean signal cost through time, and Figure 7c tracks dominant-signal rank in representative PD and SD runs. Figures 7d–f show final structured-population response maps for PD, SD, and SH, respectively. Together, these panels provide the direct signaling baseline, the cost and turnover time series, and the spatial snapshots used in the interpretations above.

Mean-field approach

To provide a mathematical perspective on the coevolution of costly signaling and cooperation, we develop two coarse-grained mean-field descriptions. For each signal σ i, let ρ i (t) denote its population frequency at generation t, and let x i (t) denote the fraction of the population that cooperates when facing that signal. The variables ρ i and x i therefore summarize, respectively, which signals are common and what response convention the population currently attaches to them. The mean-field analysis is not intended as a faithful reduction of the full agent-based process. Instead, it distinguishes effects implied by average population frequencies from effects that can arise after one illustrative form of frequency dependence is added. It does not identify a unique omitted correlation or derive the correction from the simulations. For clarity, we refer to the two reduced descriptions below as the independence closure and the rarity-dependent closure.

Independence closure

We begin with the simplest closure, in which signal production and response rules are treated as unlinked at the population level. This approximation retains the marginal signal frequencies ρ i and the response probabilities x i, but neglects higher-order associations between who emits a signal and which response convention is attached to it. It therefore provides a natural baseline: if this closure reproduces a qualitative result, average population feedback is sufficient within the reduced description; if it fails, the independence assumption is insufficient, but that failure alone does not identify which omitted association is responsible.

To reflect the nonoverlapping-generation structure of the simulations, we write the reduced dynamics in discrete time. In the absence of the stochastic terms introduced below, the model reduces to deterministic discrete-time replicator-mutator dynamics (Nowak, 2006). At generation t, the expected payoffs from cooperating and defecting in response to a signal are

Using exponential fitnesses f C (t) = e β π C (t) and f D (t) = e β π D (t), the response profile first evolves by selection,

We then include strategy mutation in the form

which reflects random reassignment of the conditioned action with probability µ s. This symmetric coarse-grained mutation operator differs from the entry-flip rule in the agent simulations; it is used here as a simple diffusion toward equal response frequencies. Finally, population stochasticity is added as

where ξ x, i (t) is a standard normal draw and σ x controls the strength of finite-population fluctuations. After this stochastic step, values are truncated back to the interval (0,1). This form is used because, in a finite population with nonoverlapping generations, random sampling around the expected post-mutation frequency has leading-order variance proportional to x i mut (t + 1) (1 − x i mut (t + 1)) / N. The stochastic term therefore provides a Gaussian diffusion approximation to Wright–Fisher-type demographic noise rather than introducing an additional strategic effect (Imhof and Nowak, 2006).

Given the updated response profile, the expected payoff from emitting signal σ i is

Under the present opponent-conditioned model, when a focal individual emits σ i and encounters an opponent emitting σ j, the focal responds to the opponent’s signal and therefore cooperates with probability x j (t + 1), while the opponent cooperates with probability x i (t + 1). Defining f ρ, i (t + 1) = e β π ρ, i (t + 1), the signal frequencies evolve by selection,

followed by mutation,

where µ σ is the average drift toward the uniform distribution induced by the signal-mutation rule. Population stochasticity is then added as

where ξ ρ, i (t) is again standard normal and σ ρ measures demographic noise. After this step, negative entries are set to zero and the resulting vector is renormalized so that ∑ i ρ i (t + 1) = 1.

We solve the independence closure with the same finite-population stochastic terms used in the reduced model, using the default well-mixed simulation parameters from Table 2. The resulting dynamics are shown in Figures 8a–c, and the associated signal statistics are shown in Figures 9a–f). In the Supplementary Information, Section S5, we also show the deterministic limit (deterministic discrete replicator dynamics) obtained by setting σ x = σ ρ = 0. The independence closure already reproduces an important asymmetry across games. In the PD, cooperation declines toward zero (Figure 8a). This can be seen directly from the payoff difference

Because T > R and P > S in the PD, Δ π (t) is negative for every 〈 x 〉 ρ ∈ [ 0, 1 ], so the deterministic part of the dynamics drives all response classes toward defection. In Supplementary Figures S7, S8, we show that removing population stochasticity does not change this conclusion. The deterministic limit still collapses to defection in PD.

The SD and SH behave differently. In the SD, the sign of Δ π (t) depends on the current average level of cooperation, and the system settles into a partially cooperative state, consistent with Figure 8b. In the SH, once the population moves above the coordination threshold, cooperation becomes self-reinforcing and the system approaches near-complete cooperation, as shown in Figure 8(c). The signal statistics in Figures 9a–f are consistent with this picture: signal usage is negatively related to cost and positively related to net signal payoff, with the strongest concentration in SH where the cooperative convention becomes most stable. Supplementary Information, Section S5, shows that the deterministic limit preserves the same SD–SH contrast, but in the baseline closure it also drives signal usage much more strongly toward the cheapest signal. Population stochasticity therefore improves the empirical appearance of the signal-frequency distributions by keeping more signals in circulation, even though it does not alter the closure’s qualitative success in SD and SH or its failure in PD. The main limitation of the independence closure is therefore specific and informative. Average population feedback is already sufficient to account qualitatively for SD and SH, but not for the partially cooperative PD regime. That failure points to the importance of correlations that are absent from this baseline description.

An illustrative rarity-dependent correction

The independence closure does not reproduce the partially cooperative PD regime. One possible omitted effect is transient frequency dependence: an uncommon signal may initially be associated with more favorable responses before exploitative strategies become concentrated around it. The turnover observed in the simulations motivates this possibility, but it does not uniquely establish rarity-based protection or determine its functional form. We therefore extend the discrete-time stochastic replicator system with a simple rarity-dependent advantage as an illustrative sufficiency test.

At generation t, the modified payoffs for responding to signal σ i are

where η > 0 measures the imposed advantage of cooperating against a rare signal. The linear form η (1 – ρ i) is a convenient phenomenological choice that is strongest when a signal is rare and vanishes as it becomes common; it is not derived from the agent-based dynamics. With f C, i ′ (t) = e β π C, i ′ (t) and f D, i ′ (t) = e β π D, i ′ (t), the response profile again evolves first by selection,

followed by mutation,

and population stochasticity,

As in the independence closure, ξ x, i (t) is a standard normal draw, and values are truncated back to (0,1) after the stochastic step. When σ x = 0 and σ ρ = 0, the deterministic part of the model reduces to a discrete-time replicator-mutator system with rarity-modified payoffs.

The same idea is applied to signal production. Under the opponent-conditioned model, the payoff from emitting signal σ i is

where δ > 0 is the signal-side rarity bonus. Defining f ρ, i ′ (t + 1) = e β π ρ, i ′ (t + 1), the signal frequencies first evolve by selection,

then by mutation,

and finally by population stochasticity,

Here ξ ρ, i (t) is again standard normal. After the stochastic step, negative entries are set to zero and the resulting vector is renormalized so that ∑ i ρ i (t + 1) = 1.

This second closure is neither derived from nor independently calibrated against the agentbased process. Its role is limited to asking whether one simple form of frequency-dependent protection can be sufficient to produce the partially cooperative PD regime that the independence closure misses. The rarity terms are phenomenological parameters, and other corrections involving signal–response linkage, inheritance, or local assortment may produce similar behavior. The agent-based simulations do not distinguish among these alternatives. The closure therefore demonstrates a possibility rather than identifying a necessary mechanism.

We solve the rarity-dependent closure with the same finite-population stochastic terms used in the reduced model, again using the default well-mixed simulation parameters from Table 2. The resulting dynamics are shown in Figures 10a-c, and the associated signal statistics are shown in Figures 11a-f. The non-rarity parameters are held fixed across games, while the rarity parameters are selected separately by qualitative comparison: (η,δ) = (1.4,0.3) for PD, (0.2,0.05) for SD, and (0.1,0.05) for SH. Because these values were neither derived nor independently measured in the agent-based simulations, comparisons among their game-specific magnitudes have no inferential meaning. They provide examples for which the reduced system displays the corresponding qualitative regimes. In the Supplementary Information, Section S5, we also show the deterministic limit obtained by setting σ x = σ ρ = 0.

For the selected parameter values, the reduced model displays the main asymmetry across games. In the PD, the imposed rarity terms prevent cooperation from collapsing to zero and generate a partially cooperative regime. In SD and SH, the smaller illustrative corrections leave the qualitative states already produced by the independence closure unchanged. The point is not exact fitting or identification of the omitted process, but the narrower result that a frequency-dependent correction of this form can be sufficient to generate partial cooperation in the reduced PD model. Supplementary Figures S9, S10 show the corresponding deterministic examples without population stochasticity.

The signal statistics in Figures 11a-f describe these illustrative solutions. Across all three games, signal usage decreases with cost and increases with interaction-derived signal payoff, with the sharpest concentration occurring in SH. For the payoff panels, we plot the interactionderived signal payoff excluding the explicit rarity term, so that the horizontal axis remains directly comparable to the simulation figures. Supplementary Information, Section S5, gives the deterministic counterparts. Taken together, these examples show sufficiency only: the chosen rarity-dependent correction can generate a partially cooperative PD regime while preserving the qualitative SD and SH patterns, but the analysis does not show that this correction is unique or necessary.

We showed that signal-response feedback can support cooperation across a broad range of strategic settings, but it does so in different ways in different games. In well-mixed populations, the mechanism sustains partial cooperation in PD and SD and drives near-complete cooperation in SH. The no-signal controls clarify the scope of this statement: the signal-response layer is doing substantial work in PD and under the SD update rule studied here, while SH can reach cooperation even without signals because it is already a coordination game. On the lattice, cooperation rises further because local clustering and signal-based discrimination reinforce one another. In fluctuating strategic environments, the same mechanism adapts smoothly: PD-SD mixtures remain close to the partially cooperative regime, while PD-SH mixtures interpolate between near-full cooperation and the harsher PD regime.

The signal statistics clarify why raw cost alone is a poor predictor of prevalence. Across games, low-cost signals are often favored, but what really determines persistence is whether a signal continues to attract cooperation. Once a signal becomes too common and begins to attract defection, its effective fitness falls and its prevalence declines. This pattern is consistent with a trade-off interpretation: production costs are weighed against the interaction benefits generated by the current response convention (Lachmann et al., 2001; Számado, 2011; Számado et al., 2023; Számado et al., 2026). An arbitrary costly marker can therefore persist without encoding hidden biological quality when its association with response strategies directs enough cooperation toward its users. The results are therefore better described as competition among signals for favorable response conventions than as a simple ranking by production cost.

The mean-field analysis separates behavior reproduced by the independence approximation from behavior that it misses. The decoupled closure reproduces the qualitative SD and SH regimes but fails in the PD because one-shot defection remains dominant under purely average mixing. The rarity-dependent closure then supplies one sufficient illustrative correction that generates partial cooperation in the reduced PD model. Because its game-specific parameters were not derived from or independently measured in the agent-based simulations, it does not identify the omitted correlation uniquely; linkage disequilibrium, inheritance, spatial assortment, or other frequency-dependent effects remain possible explanations.

Relative to (Salahshour, 2019), the present results clarify the scope conditions of the mechanism. The earlier work identified partial cooperation in a well-mixed setting. Here the same line of inquiry is followed across multiple games and population structures, and in a simplified behavioral formulation where actions depend only on the opponent’s signal. Taken together, the simulations and reduced descriptions show where the mechanism carries over and where average mixing is insufficient, without determining which omitted correlation is responsible for the PD regime.

The interpretation is complementary to quality-based signaling accounts rather than a replacement for them. Such accounts remain especially plausible when displays track stable differences in quality, commitment, or trustworthiness and receivers use them in choosing longterm partners, mates, or allies (Zahavi, 1975; Grafen, 1990; Roberts, 1998; Gintis et al., 2001; Barclay and Willer, 2007; Fehrler and Przepiorka, 2013; Raihani and Smith, 2015). The present model addresses a different empirical possibility, in which the value of an otherwise arbitrary marker is generated endogenously by the responses it elicits.

The mechanism studied here is more plausible in a different empirical regime: one in which cues are socially learned, partly arbitrary, or only weakly tied to intrinsic quality, but still become behaviorally important because they organize expectations about whom to help, trust, or avoid. Work on ethnic markers, symbolic labels, group identity, and language-like coordination shows that such cues can reshape cooperation because they sort interactions and support shared response conventions (Riolo et al., 2001; McElreath et al., 2003; Jansen and van Baalen, 2006; Efferson et al., 2008; Chen and Li, 2009; Salahshour, 2020). This places the model in direct contact with theories of evolving coordination markers and assortment mechanisms, while the cost side of the result connects it to trade-off accounts of signaling. In those settings, costly signaling need not be understood primarily as honest revelation. It can instead be understood as a way of stabilizing a convention long enough for cooperation to become locally advantageous. In that sense, the present model is most relevant for cases in which an observable badge, style, ritualized display, or linguistic cue acquires value through the cooperation it elicits, rather than through a fixed one-to-one link with hidden quality (Salahshour, 2019).

This perspective also suggests different empirical expectations. In settings where the link between signal cost and hidden quality is weak, unknown, or absent, it predicts weaker links between signal cost and underlying quality, stronger dependence on local convention, and more rapid turnover once an initially successful cue is imitated widely enough to attract exploitation. Relative to explanations based only on kinship or static clustering, it predicts that visible markers can matter in their own right by defining who is treated as a preferred partner within the same population structure. These predictions are broadly consistent with the literatures on symbolic markers and convention-based assortment, though testing them directly would require longitudinal data on how cooperative cues spread, concentrate locally, and lose value over time (McElreath et al., 2003; Efferson et al., 2008; Chen and Li, 2009; Salahshour, 2019). More specifically, the model predicts that an initially rare marker should receive more cooperation when it is locally associated with cooperative responses, that its spread should be followed by increased exploitation or reduced net payoff, and that its decline should coincide with the rise of alternative markers. In spatial populations, local clusters of marker use and cooperative response should persist longer than in well-mixed populations, and the erosion of a marker’s value should begin near boundaries where exploitative or mismatched response rules are common. These predictions are qualitative, but they point to measurable time series: marker prevalence, local clustering, cooperation received, exploitation received, and net payoff should co-vary over the life cycle of a convention.

The model is deliberately stylized: signals are arbitrary, interactions are one-shot, and receivers condition behavior only on observed cues. Its value is therefore not that it captures every empirical form of costly signaling, but that it isolates a mechanism by which signaling cost can become worthwhile through the cooperative responses it organizes. We also held the mutation operators and Wright–Fisher replacement rule fixed. Alternative signal and strategy mutation operators, overlapping-generation processes, pairwise-comparison rules, Moran updates, and other birth–death schedules were not tested, so robustness to those evolutionary choices remains unknown. Future work could extend the same framework in several directions. More realistic networks would show whether the effect depends mainly on local clustering or on finer community structure. A richer signal space could test whether the same turnover persists when signals are continuous or combinatorial. Autocorrelated environmental change would also be useful, because many ethological contexts shift in bouts rather than as independent encounterlevel draws. It would also be useful to derive a more explicit theory of the genotype-level correlations that the present mean-field correction summarizes only phenomenologically.