Hengchang Lu

Problem Overview

Current AI alignment paradigms adopt a corrective approach. Instead of embedding safety in intelligence's substrate, these methods treat alignment as an adversarial, post-hoc stage. They sculpt outputs only after capabilities solidify. I contend this disjoint method relies on static proxy objectives. Training a single reward model to approximate monolithic human preferences fails to capture values' dynamic, partially-observable nature. The optimization process consequently becomes a game of exploiting this static proxy; policies push against divergence constraints to maximize scores. The result is a fragile alignment subject to reward misspecification and over-optimization, with models exploiting proxy metrics instead of internalizing underlying intent.

Although these empirical methods drive rapid benchmark progress, treating alignment as purely corrective risks nonlinear scaling issues. Models already show emergent capabilities breaking training assumptions. To manage this, we must evolve beyond post-hoc fine-tuning and build alignment directly into the generative process.

This conviction defines my research objective: building inherently aligned, natively steerable systems attuned to dynamic human preferences. My current research project stems from this principle. To overcome reward over-optimization and diversity loss in fine-tuning, I developed a theoretically grounded, training-free alignment method based on Sequential Monte Carlo sampling and Feynman-Kac Correctors. This framework rethinks alignment by integrating online preference signals directly into the latent generative process through stochastic dynamics. Unlike classifier guidance requiring specific discriminators, this method decouples the generative prior from the value-alignment likelihood. We guide generation with reward-tilted perturbations in real time, requiring only incremental, sparse, and partially-observable feedback from users with unknown preferences.

Prepared for submission to ICML 2026, this solves alignment's optimal transport problem: shifting the pre-trained distribution to preferences while minimizing generative manifold distortion. This method provides principled control over the transport map, enabling fine-grained steering without retraining. In image generation with high-dimensional latent spaces, non-convex reward landscapes, and probabilistic transformations, the approach demonstrates robust convergence to high-reward regions while preserving distributional diversity. It maintains geometric coherence and semantic meaning, avoiding mode collapse and reward hacking. This proves a new paradigm where steerability is inherent.

Abstract

Diffusion models are highly effective at modeling complex data distributions, including images and text. However, in applications like personalized recommender systems, the objective often shifts to modeling specific regions of the distribution that maximize user preferences—initially unknown but gradually uncovered through interactive feedback. This can naturally be framed as a reinforcement learning problem, where the goal is to fine-tune a diffusion model to maximize a reward function based on preferences. However, the main challenge lies in learning a parameterized reward model, which typically requires large-scale preference data—something that is often not feasible in practice. In this work, we introduce a novel framework that bypasses the requirement for a pretrained reward model by directly optimizing the dynamics of the reverse diffusion process using real-time user feedback. Our framework enables feedback-efficient preference alignment, drawing inspiration from the Feynman-Kac-based Fokker-Planck framework. We demonstrate our framework's effectiveness through extensive experiments and ablation studies across diverse domains. Additionally, based on theoretical insights, we propose an enhanced fine-tuning strategy that requires less computational budget and accelerates the fine-tuning process, further boosting its suitability for real-world deployment.

Introduction

Diffusion models are powerful deep generative frameworks that synthesize data by reversing a diffusion process, enabling them to capture complex distributions such as natural image manifolds. Yet, in applications like personalized product recommendation, the goal shifts to steering generation toward items that match individual user preferences—preferences that gradually emerge from user interactions. Similar challenges occur in other domains. For example, diffusion models trained on large internet datasets are often used for image generation, but practical use cases demand outputs with specifically desired attributes, such as high aesthetic quality. Comparable situations arise in drug discovery, where generation must be guided toward molecules with strong bioactivity. These tasks can be formulated as reinforcement learning (RL) problems, where the diffusion model is fine-tuned to maximize a reward function encoding target properties or user preferences. However, RL-based methods typically require substantial preference data to learn accurate reward models, making them impractical for settings like personalized recommendation systems, where user feedback is limited and expensive to collect interactively.

The challenge is twofold: Firstly, achieving this objective requires efficient exploration. However, in high-dimensional spaces, such as those of natural images, this goes beyond simply discovering new regions. It also necessitates respecting the structural constraints of the problem. For instance, in areas like product recommendation, valid solutions—such as realistic-looking products—are typically confined to a lower-dimensional manifold within a much larger design space. Therefore, an effective, feedback-efficient fine-tuning method must explore this space while staying within the feasible area, as venturing outside would lead to wasteful invalid queries. Moreover, fine-tuning the diffusion model to aggressively optimize based on the preferences collected so far can reduce sample diversity. This is because human preferences are often multimodal, and the model, if overly focused on a narrow set of preferences, may fail to capture the full spectrum of diverse user preferences, leading to a less varied sample generation. Therefore, efficient exploration is crucial to maintaining the quality of generated samples and ensuring greater diversity in sample generation.

Secondly, a key challenge in many applications is the high cost of acquiring feedback for the ground-truth reward function. For instance, in a product recommendation system, determining user preferences requires subjective human judgment, which is both costly and time-consuming. This challenge is further compounded by the need for the model to not only explore new options but also to exploit the information it has gathered to generate samples that align with the user's preferences. If the model continues to explore without producing samples that meet the user's expectations, it risks disengaging the user. In a nutshell, the model must strike a balance between exploration and exploitation—effectively generating preference-aligned samples while minimizing costly reward queries. While several recent works have proposed RL-based fine-tuning methods for diffusion models, none directly tackle the challenge of feedback efficiency in an online setting. Uehera et. al. introduced a framework that accounts for the online nature of feedback but still relies on a separate parameterized reward model for optimization. Our goal instead is to develop a feedback-efficient online fine-tuning approach that entirely eliminates the need for a separate pre-trained reward model, instead directly leveraging inference time scaling of a base diffusion model using real-time user feedback.

Framework Architecture

Our framework integrates online preference signals directly into the latent generative process through stochastic dynamics, rethinking alignment by decoupling the generative prior from the value-alignment likelihood. The architecture consists of several interconnected components working together to achieve feedback-efficient preference alignment.