Multi-Vehicle Cooperative Decision-Making

My PhD Research

My doctoral research centers on multi-vehicle cooperative decision-making in mixed traffic. I developed an integrated multi-agent decision-making framework to address the coordination of high-dimensional vehicle interactions under stochastic environments. The framework is organized around 4 interconnected methodological pillars: reward shaping, search optimization, interactive representation, and topological exploration. These methodological advances are complemented by closed-loop validation in a cyber-physical scaled-vehicle testbed.

Problem Setting: The problem is formulated in continuous roadway segments characterized by two key features: stochastic traffic conditions and mixed traffic composed of CAVs and hHDVs. Stochasticity arises from random vehicle inflows, varying initial and target lanes, and heterogeneous destination requirements. Mixed traffic introduces interactions between controllable CAVs and HDVs whose behaviors are only partially observable and cannot be directly controlled. The resulting decision problem is therefore to coordinate a group of CAVs to improve traffic efficiency and safety under uncertain traffic conditions and heterogeneous, partially controllable interactions.

Illustration of the cooperative decision-making problem for CAVs in stochastic mixed traffic segments.

Hybrid Differential Reward (HDR): To address reward sparsity and credit-assignment challenges arising from high-frequency control in continuous environments, I developed the HDR reward-shaping mechanism. HDR combines the temporal variation of state potentials with the marginal contributions of individual actions to provide more informative learning signals. The resulting reward structure preserves optimal-policy invariance while substantially improving the discriminability of the reward landscape, accelerating MARL convergence in high-frequency decision-making settings. (Han et al., 2026)

Schematic diagram of the proposed HDR reward shaping framework.

Parallel Update MCTS: To address the combinatorial growth of the joint-action space $\mathcal{O}(\mathcal{A}^N)$ in multi-agent coordination, I developed the PE-MCTS algorithm. By analyzing the “safety short-board” effect in vehicular interactions, PE-MCTS introduces safety-similarity-based soft action pruning and a Parallel-update mechanism, together with Experiential-preference-guided expansion. The resulting search procedure provides an asymptotically optimal, non-myopic decision oracle for high-dimensional coordination, offering a quantitative performance reference against which real-time learned policies can be evaluated. (Han et al., 2025; Han et al., 2026)

The baseline search process using the PE-MCTS algorithm.

Spatiotemporal Representation Network (RepNet): To capture dynamic and non-Euclidean interactions among vehicles, I designed RepNet, an encoder-only architecture inspired by the ViT. RepNet tokenizes spatiotemporal traffic-state representations with physical position encodings and employs global multi-head self-attention to capture dependencies across the multi-agent traffic topology. It transforms variable-length and heterogeneous traffic states into fixed-dimensional, decision-oriented embeddings that can be directly integrated with downstream policy networks. (Han et al., 2024)

Architecture overview of RepNet for multi-agent interaction representation.

Topology-Enhanced Exploration: To overcome the limitations of blind trial-and-error in strong game-theoretic environments, I proposed the TPE framework. It utilizes locality-sensitive hashing to project continuous kinematic manifolds into discrete topological quotient spaces, ensuring game-equivalence via MCTS oracle policy. By incorporating information-theoretic measures such as conditional mutual information gain, TPE converts largely random exploration into a structured process of topology-guided information discovery, leading to more robust and cooperative multi-agent policies. (Han et al., 2026)

Overview of the Topology-Enhanced (TPE) mechanism for MAR.

Cyber-Physical Validation: To bridge the gap between algorithms and physical deployment, I led the development of a multi-agent scaled-vehicle testbed. The platform implements and experimentally validates an edge-cloud architecture, with centralized, high-dimensional decision-making performed at the roadside/cloud level and distributed execution at the vehicle level. Physical experiments further demonstrated the robustness of the proposed game-theoretic decision-making methods under practical engineering constraints, including V2X communication latency, actuator delays, and positioning noise.

The hardware architecture and physical environment of the scaled-vehicle experimental platform.

References

2026

preprint

Hybrid Differential Reward: Combining Temporal Difference and Action Gradients for Efficient Multi-Agent Reinforcement Learning in Cooperative Driving

Ye Han, Lijun Zhang, Dejian Meng, and 1 more author

arXiv preprint, 2026

Abs DOI HTML

In multi-vehicle cooperative driving tasks involving high-frequency continuous control, traditional state-based reward functions suffer from the issue of vanishing reward differences. This phenomenon results in a low signal-to-noise ratio (SNR) for policy gradients, significantly hindering algorithm convergence and performance improvement. To address this challenge, this paper proposes a novel Hybrid Differential Reward (HDR) mechanism. We first theoretically elucidate how the temporal quasi-steady nature of traffic states and the physical proximity of actions lead to the failure of traditional reward signals. Building on this analysis, the HDR framework innovatively integrates two complementary components: (1) a Temporal Difference Reward (TRD) based on a global potential function, which utilizes the evolutionary trend of potential energy to ensure optimal policy invariance and consistency with long-term objectives; and (2) an Action Gradient Reward (ARG), which directly measures the marginal utility of actions to provide a local guidance signal with a high SNR. Furthermore, we formulate the cooperative driving problem as a Multi-Agent Partially Observable Markov Game (POMDPG) with a time-varying agent set and provide a complete instantiation scheme for HDR within this framework. Extensive experiments conducted using both online planning (MCTS) and Multi-Agent Reinforcement Learning (QMIX, MAPPO, MADDPG) algorithms demonstrate that the HDR mechanism significantly improves convergence speed and policy stability. The results confirm that HDR guides agents to learn high-quality cooperative policies that effectively balance traffic efficiency and safety.
preprint

Policy Optimality Measurement for Multi-Vehicle Decision-Making: From Extrinsic Indicators to Intrinsic Quality

Ye Han, Lijun Zhang, Dejian Meng, and 1 more author

arXiv preprint, 2026

Abs DOI HTML

Evaluating Multi-Agent Reinforcement Learning (MARL) policies in autonomous driving fundamentally relies on extrinsic statistical indicators (e.g., reward curves and success rates), which often mask intrinsic policy degradation and algorithmic blind spots. To break this black-box evaluation, this letter proposes a novel information-theoretic diagnostic framework. By leveraging a fully converged Monte Carlo Tree Search (MCTS) as an asymptotic oracle, we establish a theoretical ground-truth baseline distribution. We formulate a bounded policy optimality score (\mathcalM_opt) using the forward KL divergence to rigorously penalize fatal collaborative omissions. Crucially, we semantically decouple this metric into lateral and longitudinal dimensions, creating a granular “semantic microscope”. Extensive spatial and temporal diagnostics on state-of-the-art MARL architectures and exploration mechanisms demonstrate that our framework conclusively exposes hidden directional biases, identifies temporal average-policy traps, and transforms heuristic hyperparameter tuning into a visually trackable trajectory optimization. This framework establishes a rigorous, model-agnostic standard for benchmarking intrinsic multi-agent policy quality.
preprint

Topology Enhanced MARL for Multi-Agent Cooperative Decision-Making of CAVs

Ye Han, Lijun Zhang, Dejian Meng, and 1 more author

arXiv preprint, 2026

Abs DOI HTML PDF

Decentralized multi-agent cooperative decision-making in continuous environments is fundamentally bottlenecked by the curse of dimensionality, where undirected exploration typically converges to conservative local optima. We propose Topology-Enhanced Multi-Agent Reinforcement Learning (TPE-MARL) to reformulate multi-agent exploration as a structured topological traversal. We introduce the Game Topology Tensor, utilizing locality-sensitive hashing to project the continuous physical manifold into a discrete quotient space. This abstraction operates as an adversarial Information Bottleneck, decoupling strategic coordination intents from environmental noise. Within this space, a dual intrinsic reward mechanism drives exploration: a novelty reward maximizes the marginal entropy of visited topologies, while a collaboration reward, optimized via a variational Evidence Lower Bound (ELBO), minimizes conditional entropy to exploit cooperative joint configurations. Evaluations demonstrate that TPE-MARL achieves near-optimal decision distributions, closely approximating the theoretical bounds established by a Monte Carlo Tree Search (MCTS) oracle. Furthermore, physical testbed experiments validate the framework’s zero-shot out-of-distribution (OOD) generalization. Supported by a spatial relaxation mechanism, the learned representations reliably execute dynamic negotiations, such as cooperative zipper-merging, exhibiting inherent robustness against real-world covariate shifts and actuation latencies.

2025

IEEE T-ITS

A Value Based Parallel Update MCTS Method for Multi-Agent Cooperative Decision Making of Connected and Automated Vehicles

Ye Han, Lijun Zhang, Dejian Meng, and 3 more authors

IEEE Transactions on Intelligent Transportation Systems, 2025

Abs DOI HTML

This paper addresses the problem of long term joint lateral and longitudinal decision-making for multi-vehicle cooperative driving among connected and automated vehicles (CAVs). We propose a Monte Carlo tree search (MCTS) method featuring parallel update mechanism and experiential action preference, with the problem modeled as a finite-horizon, time-discounted multi-agent Markov game. By enabling a single simulation rollout to prune entire families of hazardous actions simultaneously, parallel update mechanism dramatically accelerates the convergence of the search within the vast joint action space. The search is further focused by an experiential action preference heuristic that biases exploration towards efficient maneuvers. Evaluated in randomly generated mixed-traffic scenarios, our method significantly outperforms standard MCTS and reinforcement learning baselines in both safety and traffic efficiency. Crucially, in-depth analysis reveals the discovery of emergent, non-myopic cooperative strategies that are inaccessible to myopic planners. This work provides a computationally tractable solution for complex, multi-agent MCTS, achieving a higher level of strategic cooperative decision-making.

2024

IEEE ITSC

SPformer: A transformer based DRL decision making method for connected automated vehicles

Ye Han, Lijun Zhang, Dejian Meng, and 2 more authors

In IEEE International Conference on Intelligent Transportation Systems, 2024

Abs DOI HTML

In mixed autonomy traffic environment, every decision made by an autonomous-driving car may have a great impact on the transportation system. Because of the complex interaction between vehicles, it is challenging to make decisions that can ensure both high traffic efficiency and safety now and futher. Connected automated vehicles (CAVs) have great potential to improve the quality of decision-making in this continuous, highly dynamic and interactive environment because of their stronger sensing and communicating ability. For multi-vehicle collaborative decision-making algorithms based on deep reinforcement learning (DRL), we need to represent the interactions between vehicles to obtain interactive features. The representation in this aspect directly affects the learning efficiency and the quality of the learned policy. To this end, we propose a CAV decision-making architecture based on transformer and reinforcement learning algorithms. A learnable policy token is used as the learning medium of the multi-vehicle joint policy, the states of all vehicles in the area of interest can be adaptively noticed in order to extract interactive features among agents. We also design an intuitive physical positional encodings, the redundant location information of which optimizes the performance of the network. Simulations show that our model can make good use of all the state information of vehicles in traffic scenario, so as to obtain high-quality driving decisions that meet efficiency and safety objectives. The comparison shows that our method significantly improves existing DRL-based multi-vehicle cooperative decision-making algorithms.