Democratic ICAI: Debating Our Way to Steering Principles from Preferences

Preference-based alignment matters to learners because it struggles to capture the reasoning that underlies human judgments when pairwise labels reveal only the final choice rather than the considerations that shape preferences.

Preference-based alignment often struggles to capture the reasoning that underlies human judgments. Many evaluations rely on multiple interacting criteria, yet pairwise labels reveal only the final choice rather than the considerations that shape preferences. Inverse…