about

Publications

2024

1. RPCBF: Constructing Safety Filters Robust to Model Error and Disturbances via Policy Control Barrier Functions

   Luzia Knoedler*, Oswin So*, Ji Yin, Mitchell Black, Zachary Serlin, Panagiotis Tsiotras, Javier Alonso-Mora, and Chuchu Fan

   2025 IEEE International Conference on Robotics and Automation (ICRA) (Under Review)

   Website Cite PDF

   @inproceedings{knoedler2025rpcbf,
     title={RPCBF: Constructing Safety Filters Robust to Model Error and Disturbances via Policy Control Barrier Functions},
     author={Knoedler*, Luzia and So*, Oswin and Yin, Ji and Black, Mitchell and Serlin, Zachary and Tsiotras, Panagiotis and Alonso-Mora, Javier and Fan, Chuchu},
     booktitle={2025 IEEE International Conference on Robotics and Automation (ICRA) (Under Review)},
     year={2024},
   }

2. NeurIPS 2024

   Solving Minimum-Cost Reach Avoid using Reinforcement Learning

   Oswin So*, Cheng Ge*, and Chuchu Fan

   Thirty-Eighth Conference on Neural Information Processing Systems , 2024

   Website Cite Abstract PDF

   Current reinforcement-learning methods are unable to directly learn policies that solve the minimum cost reach-avoid problem to minimize cumulative costs subject to the constraints of reaching the goal and avoiding unsafe states, as the structure of this new optimization problem is incompatible with current methods. Instead, a surrogate problem is solved where all objectives are combined with a weighted sum. However, this surrogate objective results in suboptimal policies that do not directly minimize the cumulative cost. In this work, we propose RC-PPO, a reinforcement-learning-based method for solving the minimum-cost reach-avoid problem by using connections to Hamilton-Jacobi reachability. Empirical results demonstrate that RC-PPO learns policies with comparable goal-reaching rates to while achieving up to 57% lower cumulative costs compared to existing methods on a suite of minimum-cost reach-avoid benchmarks on the Mujoco simulator.

   @inproceedings{so2024solving,
     title={Solving Minimum-Cost Reach Avoid using Reinforcement Learning},
     author={So*, Oswin and Ge*, Cheng and Fan, Chuchu},
     booktitle={Thirty-Eighth Conference on Neural Information Processing Systems},
     year={2024},
   }

3. T-RO

   Gcbf+: A neural graph control barrier function framework for distributed safe multi-agent control

   Songyuan Zhang*, Oswin So*, Kunal Garg, and Chuchu Fan

   IEEE Transactions on Robotics (Under Review)

   Website arXiv Cite Abstract

   Distributed, scalable, and safe control of large-scale multi-agent systems (MAS) is a challenging problem. In this paper, we design a distributed framework for safe multi-agent control in large-scale environments with obstacles, where a large number of agents are required to maintain safety using only local information and reach their goal locations. We introduce a new class of certificates, termed graph control barrier function (GCBF), which are based on the well-established control barrier function (CBF) theory for safety guarantees and utilize a graph structure for scalable and generalizable distributed control of MAS. We develop a novel theoretical framework to prove the safety of an arbitrary-sized MAS with a single GCBF. We propose a new training framework GCBF+ that uses graph neural networks (GNNs) to parameterize a candidate GCBF and a distributed control policy. The proposed framework is distributed and is capable of directly taking point clouds from LiDAR, instead of actual state information, for real-world robotic applications. We illustrate the efficacy of the proposed method through various hardware experiments on a swarm of drones with objectives ranging from exchanging positions to docking on a moving target without collision. Additionally, we perform extensive numerical experiments, where the number and density of agents, as well as the number of obstacles, increase. Empirical results show that in complex environments with nonlinear agents (e.g., Crazyflie drones) GCBF+ outperforms the handcrafted CBF-based method with the best performance by up to 20% for relatively small-scale MAS for up to 256 agents, and leading reinforcement learning (RL) methods by up to 40% for MAS with 1024 agents. Furthermore, the proposed method does not compromise on the performance, in terms of goal reaching, for achieving high safety rates, which is a common trade-off in RL-based methods.

   @article{zhang2024gcbf+,
     title={Gcbf+: A neural graph control barrier function framework for distributed safe multi-agent control},
     author={Zhang*, Songyuan and So*, Oswin and Garg, Kunal and Fan, Chuchu},
     journal={IEEE Transactions on Robotics (Under Review)},
     year={2024},
   }

4. ARC

   Learning safe control for multi-robot systems: Methods, verification, and open challenges

   Kunal Garg, Songyuan Zhang, Oswin So, Charles Dawson, and Chuchu Fan

   Annual Reviews in Control

   arXiv Cite Abstract

   In this survey, we review the recent advances in control design methods for robotic multi-agent systems (MAS), focussing on learning-based methods with safety considerations. We start by reviewing various notions of safety and liveness properties, and modeling frameworks used for problem formulation of MAS. Then we provide a comprehensive review of learning-based methods for safe control design for multi-robot systems. We start with various types of shielding-based methods, such as safety certificates, predictive filters, and reachability tools. Then, we review the current state of control barrier certificate learning in both a centralized and distributed manner, followed by a comprehensive review of multi-agent reinforcement learning with a particular focus on safety. Next, we discuss the state-of-the-art verification tools for the correctness of learning-based methods. Based on the capabilities and the limitations of the state of the art methods in learning and verification for MAS, we identify various broad themes for open challenges: how to design methods that can achieve good performance along with safety guarantees; how to decompose single-agent based centralized methods for MAS; how to account for communication-related practical issues; and how to assess transfer of theoretical guarantees to practice.

   @article{garg2024learning,
     title={Learning safe control for multi-robot systems: Methods, verification, and open challenges},
     author={Garg, Kunal and Zhang, Songyuan and So, Oswin and Dawson, Charles and Fan, Chuchu},
     journal={Annual Reviews in Control},
     year={2024},
   }

5. ICRA 2024

   How to train your neural control barrier function: Learning safety filters for complex input-constrained systems

   Oswin So, Zachary Serlin, Makai Mann, Jake Gonzales, Kwesi Rutledge, Nicholas Roy, and Chuchu Fan

   2024 International Conference on Robotics and Automation (ICRA)

   Website arXiv Cite Abstract

   Control barrier functions (CBF) have become popular as a safety filter to guarantee the safety of nonlinear dynamical systems for arbitrary inputs. However, it is difficult to construct functions that satisfy the CBF constraints for high relative degree systems with input constraints. To address these challenges, recent work has explored learning CBFs using neural networks via neural CBF (NCBF). However, such methods face difficulties when scaling to higher dimensional systems under input constraints. In this work, we first identify challenges that NCBFs face during training. Next, to address these challenges, we propose policy neural CBF (PNCBF), a method of constructing CBFs by learning the value function of a nominal policy, and show that the value function of the maximum-over-time cost is a CBF. We demonstrate the effectiveness of our method in simulation on a variety of systems ranging from toy linear systems to an F-16 jet with a 16-dimensional state space. Finally, we validate our approach on a two-agent quadcopter system on hardware under tight input constraints.

   @inproceedings{so2024train,
     title={How to train your neural control barrier function: Learning safety filters for complex input-constrained systems},
     author={So, Oswin and Serlin, Zachary and Mann, Makai and Gonzales, Jake and Rutledge, Kwesi and Roy, Nicholas and Fan, Chuchu},
     booktitle={2024 International Conference on Robotics and Automation (ICRA)},
     year={2024},
   }

2023

1. RSS 2023

   Solving Stabilize-Avoid Optimal Control via Epigraph Form and Deep Reinforcement Learning

   Oswin So, and Chuchu Fan

   Robotics: Science and Systems , 2023

   Website arXiv Cite Abstract

   Tasks for autonomous robotic systems commonly require stabilization to a desired region while maintaining safety specifications. However, solving this multi-objective problem is challenging when the dynamics are nonlinear and high-dimensional, as traditional methods do not scale well and are often limited to specific problem structures. To address this issue, we propose a novel approach to solve the stabilize-avoid problem via the solution of an infinite-horizon constrained optimal control problem (OCP). We transform the constrained OCP into epigraph form and obtain a two-stage optimization problem that optimizes over the policy in the inner problem and over an auxiliary variable in the outer problem. We then propose a new method for this formulation that combines an on-policy deep reinforcement learning algorithm with neural network regression. Our method yields better stability during training, avoids instabilities caused by saddle-point finding, and is not restricted to specific requirements on the problem structure compared to more traditional methods. We validate our approach on different benchmark tasks, ranging from low-dimensional toy examples to an F16 fighter jet with a 17-dimensional state space. Simulation results show that our approach consistently yields controllers that match or exceed the safety of existing methods while providing ten-fold increases in stability performance from larger regions of attraction.

   @inproceedings{so2023solving,
     title={Solving Stabilize-Avoid Optimal Control via Epigraph Form and Deep Reinforcement Learning},
     author={So, Oswin and Fan, Chuchu},
     booktitle={Robotics: Science and Systems},
     year={2023},
   }

2. ICRA 2023

   MPOGames: Efficient Multimodal Partially Observable Dynamic Games

   Oswin So, Paul Drews, Thomas Balch, Velin Dimitrov, Guy Rosman, and Evangelos A. Theodorou

   2023 IEEE International Conference on Robotics and Automation (ICRA)

   arXiv Cite Abstract

   Game theoretic methods have become popular for planning and prediction in situations involving rich multi-agent interactions. However, these methods often assume the existence of a single local Nash equilibria and are hence unable to handle uncertainty in the intentions of different agents. While maximum entropy (MaxEnt) dynamic games try to address this issue, practical approaches solve for MaxEnt Nash equilibria using linear-quadratic approximations which are restricted to unimodal responses and unsuitable for scenarios with multiple local Nash equilibria. By reformulating the problem as a POMDP, we propose MPOGames, a method for efficiently solving MaxEnt dynamic games that captures the interactions between local Nash equilibria. We show the importance of uncertainty-aware game theoretic methods via a two-agent merge case study. Finally, we prove the real-time capabilities of our approach with hardware experiments on a 1/10th scale car platform.

   @inproceedings{so2022mpogames,
     title={MPOGames: Efficient Multimodal Partially Observable Dynamic Games},
     author={So, Oswin and Drews, Paul and Balch, Thomas and Dimitrov, Velin and Rosman, Guy and Theodorou, Evangelos A.},
     booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)},
     year={2023},
   }

2022

1. ML4PS 2022

   Data-driven discovery of non-Newtonian astronomy via learning non-Euclidean Hamiltonian

   Oswin So, Gongjie Li, Evangelos A Theodorou, and Molei Tao

   Machine Learning and the Physical Sciences Workshop NeurIPS , 2022

   arXiv Cite Abstract

   Incorporating the Hamiltonian structure of physical dynamics into deep learning models provides a powerful way to improve the interpretability and prediction accuracy. While previous works are mostly limited to the Euclidean spaces, their extension to the Lie group manifold is needed when rotations form a key component of the dynamics, such as the higher-order physics beyond simple point-mass dynamics for N-body celestial interactions. Moreover, the multiscale nature of these processes presents a challenge to existing methods as a long time horizon is required. By leveraging a symplectic Lie-group manifold preserving integrator, we present a method for data-driven discovery of non-Newtonian astronomy. Preliminary results show the importance of both these properties in training stability and prediction accuracy.

   @inproceedings{so2022data,
     title={Data-driven discovery of non-Newtonian astronomy via learning non-Euclidean Hamiltonian},
     author={So, Oswin and Li, Gongjie and Theodorou, Evangelos A and Tao, Molei},
     booktitle={Machine Learning and the Physical Sciences Workshop NeurIPS},
     year={2022},
   }

2. NeurIPS 2022

   Deep Generalized Schrodinger Bridge

   Guan-Horng Liu, Tianrong Chen*, Oswin So*, and Evangelos A Theodorou

   Thirty-Sixth Conference on Neural Information Processing Systems , 2022

   arXiv Cite Abstract

   Mean-Field Game (MFG) serves as a crucial mathematical framework in modeling the collective behavior of individual agents interacting stochastically with a large population. In this work, we aim at solving a challenging class of MFGs in which the differentiability of these interacting preferences may not be available to the solver, and the population is urged to converge exactly to some desired distribution. These setups are, despite being well-motivated for practical purposes, complicated enough to paralyze most (deep) numerical solvers. Nevertheless, we show that Schrödinger Bridge - as an entropy-regularized optimal transport model - can be generalized to accepting mean-field structures, hence solving these MFGs. This is achieved via the application of Forward-Backward Stochastic Differential Equations theory, which, intriguingly, leads to a computational framework with a similar structure to Temporal Difference learning. As such, it opens up novel algorithmic connections to Deep Reinforcement Learning that we leverage to facilitate practical training. We show that our proposed objective function provides necessary and sufficient conditions to the mean-field problem. Our method, named Deep Generalized Schrödinger Bridge (DeepGSB), not only outperforms prior methods in solving classical population navigation MFGs, but is also capable of solving 1000-dimensional opinion depolarization, setting a new state-of-the-art numerical solver for high-dimensional MFGs. Our code will be made available at https://github.com/ghliu/DeepGSB.

   @inproceedings{liu2022deep,
     title={Deep Generalized Schrodinger Bridge},
     author={Liu, Guan-Horng and Chen*, Tianrong and So*, Oswin and Theodorou, Evangelos A},
     booktitle={Thirty-Sixth Conference on Neural Information Processing Systems},
     year={2022},
   }

3. Multimodal Maximum Entropy Dynamic Games

   Oswin So, Kyle Stachowicz, and Evangelos A. Theodorou

   arXiv preprint (in submission)

   arXiv Cite Abstract Video

   Environments with multi-agent interactions often result a rich set of modalities of behavior between agents due to the inherent suboptimality of decision making processes when agents settle for satisfactory decisions. However, existing algorithms for solving these dynamic games are strictly unimodal and fail to capture the intricate multimodal behaviors of the agents. In this paper, we propose MMELQGames (Multimodal Maximum-Entropy Linear Quadratic Games), a novel constrained multimodal maximum entropy formulation of the Differential Dynamic Programming algorithm for solving generalized Nash equilibria. By formulating the problem as a certain dynamic game with incomplete and asymmetric information where agents are uncertain about the cost and dynamics of the game itself, the proposed method is able to reason about multiple local generalized Nash equilibria, enforce constraints with the Augmented Lagrangian framework and also perform Bayesian inference on the latent mode from past observations. We assess the efficacy of the proposed algorithm on two illustrative examples: multi-agent collision avoidance and autonomous racing. In particular, we show that only MMELQGames is able to effectively block a rear vehicle when given a speed disadvantage and the rear vehicle can overtake from multiple positions.

   @article{so2022multimodal,
     title={Multimodal Maximum Entropy Dynamic Games},
     author={So, Oswin and Stachowicz, Kyle and Theodorou, Evangelos A.},
     journal={arXiv preprint (in submission)},
     year={2022},
   }

4. RSS 2022

   Decentralized Safe Multi-agent Stochastic Optimal Control using Deep FBSDEs and ADMM

   Marcus A. Pereira, Augustinos D. Saravanos, Oswin So, and Evangelos A. Theodorou

   Robotics: Science and Systems , 2022

   arXiv Cite Abstract Video

   In this work, we propose a novel safe and scalable decentralized solution for multi-agent control in the presence of stochastic disturbances. Safety is mathematically encoded using stochastic control barrier functions and safe controls are computed by solving quadratic programs. Decentralization is achieved by augmenting to each agent’s optimization variables, copy variables, for its neighboring agents. This allows us to decouple the centralized multi-agent optimization problem. How- ever, to ensure safety, neighboring agents must agree on what is safe for both of us and this creates a need for consensus. To enable safe consensus solutions, we incorporate an ADMM- based approach. Specifically, we propose a Merged CADMM- OSQP implicit neural network layer, that solves a mini-batch of both, local quadratic programs as well as the overall con- sensus problem, as a single optimization problem. This layer is embedded within a Deep FBSDEs network architecture at every time step, to facilitate end-to-end differentiable, safe and decentralized stochastic optimal control. The efficacy of the proposed approach is demonstrated on several challenging multi- robot tasks in simulation. By imposing requirements on safety specified by collision avoidance constraints, the safe operation of all agents is ensured during the entire training process. We also demonstrate superior scalability in terms of computational and memory savings as compared to a centralized approach.

   @inproceedings{pereira2022decentralized,
     title={Decentralized Safe Multi-agent Stochastic Optimal Control using Deep FBSDEs and ADMM},
     author={Pereira, Marcus A. and Saravanos, Augustinos D. and So, Oswin and Theodorou, Evangelos A.},
     booktitle={Robotics: Science and Systems},
     year={2022},
   }

2021

1. ICRA 2022

   Maximum Entropy Differential Dynamic Programming

   Oswin So, Ziyi Wang, and Evangelos A. Theodorou

   2022 International Conference on Robotics and Automation (ICRA)

   arXiv Cite Abstract Video

   In this paper, we present a novel maximum entropy formulation of the Differential Dynamic Programming algorithm and derive two variants using unimodal and multimodal value functions parameterizations. By combining the maximum entropy Bellman equations with a particular approximation of the cost function, we are able to obtain a new formulation of Differential Dynamic Programming which is able to escape from local minima via exploration with a multimodal policy. To demonstrate the efficacy of the proposed algorithm, we provide experimental results using four systems on tasks that are represented by cost functions with multiple local minima and compare them against vanilla Differential Dynamic Programming. Furthermore, we discuss connections with previous work on the linearly solvable stochastic control framework and its extensions in relation to compositionality.

   @inproceedings{so2021maximum,
     title={Maximum Entropy Differential Dynamic Programming},
     author={So, Oswin and Wang, Ziyi and Theodorou, Evangelos A.},
     booktitle={2022 International Conference on Robotics and Automation (ICRA)},
     year={2021},
   }

2. RSS 2021

   Variational Inference MPC using Tsallis Divergence

   Ziyi Wang*, Oswin So*, Jason Gibson, Bogdan Vlahov, Manan S Gandhi, Guan-Horng Liu, and Evangelos A Theodorou

   Robotics: Science and Systems , 2021

   arXiv Cite Abstract

   In this paper, we provide a generalized framework for Variational Inference-Stochastic Optimal Control by using thenon-extensive Tsallis divergence. By incorporating the deformed exponential function into the optimality likelihood function, a novel Tsallis Variational Inference-Model Predictive Control algorithm is derived, which includes prior works such as Variational Inference-Model Predictive Control, Model Predictive PathIntegral Control, Cross Entropy Method, and Stein VariationalInference Model Predictive Control as special cases. The proposed algorithm allows for effective control of the cost/reward transform and is characterized by superior performance in terms of mean and variance reduction of the associated cost. The aforementioned features are supported by a theoretical and numerical analysis on the level of risk sensitivity of the proposed algorithm as well as simulation experiments on 5 different robotic systems with 3 different policy parameterizations.

   @inproceedings{wang2021variational,
     title={Variational Inference MPC using Tsallis Divergence},
     author={Wang*, Ziyi and So*, Oswin and Gibson, Jason and Vlahov, Bogdan and Gandhi, Manan S and Liu, Guan-Horng and Theodorou, Evangelos A},
     booktitle={Robotics: Science and Systems},
     year={2021},
   }

3. Spatio-Temporal Differential Dynamic Programming for Control of Fields

   Ethan N Evans, Oswin So, Andrew P Kendall, Guan-Horng Liu, and Evangelos A Theodorou

   arXiv preprint (in submission)

   arXiv Cite Abstract

   We consider the optimal control problem of a general nonlinear spatio-temporal system described by Partial Differential Equations (PDEs). Theory and algorithms for control of spatio-temporal systems are of rising interest among the automatic control community and exhibit numerous challenging characteristic from a control standpoint. Recent methods focus on finite-dimensional optimization techniques of a discretized finite dimensional ODE approximation of the infinite dimensional PDE system. In this paper, we derive a differential dynamic programming (DDP) framework for distributed and boundary control of spatio-temporal systems in infinite dimensions that is shown to generalize both the spatio-temporal LQR solution, and modern finite dimensional DDP frameworks. We analyze the convergence behavior and provide a proof of global convergence for the resulting system of continuous-time forward-backward equations. We explore and develop numerical approaches to handle sensitivities that arise during implementation, and apply the resulting STDDP algorithm to a linear and nonlinear spatio-temporal PDE system. Our framework is derived in infinite dimensional Hilbert spaces, and represents a discretization-agnostic framework for control of nonlinear spatio-temporal PDE systems.

   @article{evans2021spatio,
     title={Spatio-Temporal Differential Dynamic Programming for Control of Fields},
     author={Evans, Ethan N and So, Oswin and Kendall, Andrew P and Liu, Guan-Horng and Theodorou, Evangelos A},
     journal={arXiv preprint (in submission)},
     year={2021},
   }

4. L4DC 2021

   Adaptive Risk Sensitive Model Predictive Control with Stochastic Search

   Ziyi Wang, Oswin So, Keuntaek Lee, and Evangelos A. Theodorou

   Learning for Dynamics & Control Conference , 2021

   arXiv Cite Abstract

   We present a general framework for optimizing the Conditional Value-at-Risk for dynamical systems using stochastic search. The framework is capable of handling the uncertainty from the initial condition, stochastic dynamics, and uncertain parameters in the model. The algorithm is compared against a risk-sensitive distributional reinforcement learning framework and demonstrates outperformance on a pendulum and cartpole with stochastic dynamics. We also showcase the applicability of the framework to robotics as an adaptive risk-sensitive controller by optimizing with respect to the fully nonlinear belief provided by a particle filter on a pendulum, cartpole, and quadcopter in simulation.

   @inproceedings{wang2021adaptive,
     title={Adaptive Risk Sensitive Model Predictive Control with Stochastic Search},
     author={Wang, Ziyi and So, Oswin and Lee, Keuntaek and Theodorou, Evangelos A.},
     booktitle={Learning for Dynamics & Control Conference},
     year={2021},
   }