Revolutionizing Sampling: KIT, NVIDIA, and Zuse Institute’s Breakthrough in Handling Degenerate Noise

Diffusion processes have gained attention as effective methods for sampling from intricate distributions but encounter substantial difficulties with multimodal targets. Conventional techniques relying on overdamped Langevin dynamics often show slow convergence rates when moving between different distribution modes. Although underdamped Langevin dynamics introduce an extra momentum variable and have demonstrated empirical improvements, they still face inherent limitations. The noise structure in these underdamped models, where Brownian motion does not directly impact the space variable, results in smoother paths but poses challenges for theoretical understanding.

Current methods such as Annealed Importance Sampling (AIS) link prior and target distributions via transition kernels, with Unadjusted Langevin Annealing (ULA) utilizing uncorrected overdamped Langevin dynamics in this context. Monte Carlo Diffusion (MCD) aims to reduce marginal likelihood variance, while Controlled Monte Carlo Diffusion (CMCD) and Sequential Controlled Langevin Diffusion (SCLD) emphasize kernel optimization through resampling strategies. Other techniques propose backward transition kernels, including Path Integral Sampler (PIS), Time-Reversed Diffusion Sampler (DIS), and Denoising Diffusion Sampler (DDS). Some strategies, like the Diffusion Bridge Sampler (DBS), independently learn forward and backward kernels.

Research teams from the Karlsruhe Institute of Technology, NVIDIA, Zuse Institute Berlin, dida Datenschmiede GmbH, and the FZI Research Center for Information Technology have developed a broadened framework for learning diffusion bridges that transform prior distributions into target distributions. This framework encompasses both existing diffusion models and underdamped variations with selective dimensional noise impact. It provides a solid theoretical basis by equating score-matching in underdamped scenarios to maximizing a likelihood lower bound, addressing the challenge of sampling from unnormalized densities without direct samples from the target distribution.

This framework facilitates comparative analysis of five principal diffusion-based sampling methods: ULA, MCD, CMCD, DIS, and DBS. It highlights the novel contributions of underdamped variants of DIS and DBS to the field. The evaluation approach employs a diverse set of real-world benchmarks, covering Bayesian inference tasks (Credit, Cancer, Ionosphere, Sonar), parameter inference problems (Seeds, Brownian), and high-dimensional sampling with the Log Gaussian Cox process (LGCP) involving 1600 dimensions. Additionally, synthetic benchmarks include the intricate Funnel distribution, with regions exhibiting vastly different concentration levels, challenging sampling methods across varying dimensional and complexity conditions.

Results indicate that underdamped Langevin dynamics consistently outperform overdamped counterparts on both real-world and synthetic benchmarks. The underdamped DBS excels over other methods even with only 8 discretization steps. This efficiency leads to substantial computational savings while ensuring high-quality sampling. In terms of numerical integration schemes, specialized integrators show clear enhancements over traditional Euler methods for underdamped dynamics. The OBAB and BAOAB integrators provide considerable performance improvements without added computational costs, whereas the OBABO integrator yields the best results, despite requiring twice the control parameter evaluations per discretization step.

In summary, this work presents a comprehensive framework for diffusion bridges with degenerate stochastic processes. The underdamped diffusion bridge sampler achieves leading-edge results across various sampling tasks with minimal hyperparameter adjustments and a few discretization steps. Extensive ablation studies reveal that the performance gains are due to the synergistic integration of under