Stick To Your Role! Leaderboard

The Stick to Your Role! leaderboard compares LLMs based on undesired sensitivity to context change. It focuses on the stability of personal value expression in simulated personas. As proposed in our paper, unwanted context-dependence should be seen as a property of LLMs - a dimension of LLM comparison (alongside others such as model size speed or expressed knowledge). This leaderboard aims to provide such a comparison and extends our paper with a more focused and elaborate experimental setup. Standard benchmarks present many questions from the same minimal contexts (e.g. multiple choice questions), we present same questions from many different contexts.

We leverage the Schwartz's theory of Basic Personal Values, which defines 10 values Self-Direction, Stimulation, Hedonism, Achievement, Power, Security, Conformity, Tradition, Benevolence, Universalism), and the associated PVQ-40 and SVS questionnaires (available here).

Using the methodology from psychology, we focus on population-level (interpersonal) value stability, i.e. Rank-Order stability (RO stability). Rank-Order stability refers to the extent the order of different personas (in terms of expression of some value) remains the same along different contexts. Refer here or to our paper for more details.

In addition to Rank-Order stability we compute validity metrics (Stress, Separability, CFI, SRMR, RMSEA), which are a common practice in psychology. Validity refers to the extent the questionnaire measures what it purports to measure. It can be seen the questionnaire's accuracy in measuring the intended factors, i.e. values. For example, basic personal values should be organized in a circular structure, and questions measuring the same value should be correlated. The table below additionally shows the validity metrics, refer here for more details.

We aggregate Rank-Order stability and validation metrics to rank the models. We do so in two ways: Cardinal and Ordinal. Following, this paper, we compute the stability and diversity of those rankings. See here for more details.

To sum up here are the metrics used:

• RO-stability: the correlation in the order of simulated participants (ordered based on the expression of the same values) over different contexts
• Stress: the MDS fit of the observed value structure to the theoretical circular structure. Stress of 0 indicates 'perfect' fit, 0.025 excellent, 0.05 good, 0.1 fair, and 0.2 poor.
• Separability: the extent to which questions corresponding to different values are linearly separable in the 2D MDS space (linear multi-label classifier accuracy)
• CFI, SRMR, RMSEA: Common Confirmatory Factor Analysis (CFA) metrics showing the fit between the data and the posited model of the relation of items (questions) to factors (values), applied here with Magnifying Glass CFA. For CFI >.90 is considered acceptable fit, for SRMR and RMSEA is <.05 considered good fit and <.08 reasonable.
• Ordinal - Win Rate: the score averaged over all metrics (with descending metrics inverted), context pairs (for stability) and contexts (for validity metrics)
• Cardinal - Score: percentage of won games against all models, where a game is a comparison between two models for each metric, each context pair (for stability) and each context (for validity metrics)

• Contact: grgur.kovac@inria.fr
• See the Project website
• See the Flowers team blog and website
• See Grgur's website and other projects: https://grgkovac.github.io