Stick To Your Role! Leaderboard

LLMs can role-play different personas by simulating their values and behavior, but can they stick to their role whatever the context? Is simulated Joan of Arc more tradition-driven than Elvis? Will it still be the case after playing chess?

Benchmarks usually compare models with many questions from a single minimal context, e.g. as multiple choices questions. This kind of evaluation is little informative of LLMs' behavior in deployment, where they are exposed to new contexts (especially considering the LLMs highly context-dependant nature). We argue that undesired context-dependence can be seen as a property of LLMs: a dimension of LLM comparison alongside others like size, speed, or knowledge. While some context-dependence is desired (e.g. following instructions), some is undesired (e.g. drastically changing the simulated value expression based on the interlocutor). We evaluate LLMs by asking the same questions from many different contexts .

LLMs are often used to simulate personas and populations, we study the stability and coherence of a simulated population - in contrast to evaluating each persona separately, we evaluate the stability of simulated personas relative to each other, i.e. as a population. We study simulated populations over different contexts, i.e. conversations on different topics. To do that, we leverage the psychological methodology to study the interpersonal stability of personal value expression in a simulated population. We adopt the Schwartz Theory of Basic Personal Values, which defines 10 values: Self-Direction, Stimulation, Hedonism, Achievement, Power, Security, Conformity, Tradition, Benevolence, and Universalism. To evaluate their expression we use the associated questionnaires: PVQ-40, and SVS.

You can browse the questionnaires, population, and contexts used on our 🤗 StickToYourRole dataset.

The Stick to Your Role! leaderboard aims to provide an up-to-date comparison of recent LLMs based on their ability to coherently simulate popultions. It, in tandem with other minimal-context benchmarks, should enable you to choose the best-suited model for your usecase! If you want to evaluate or add your model you can follow the instructions here.

To evaluate the stability on a population level we need to be able to evaluate a value profile expressed by a simulated individual in a specific context (conversation topic). We do with the following procedure:

1. The Tested model is instructed to simulate a persona
2. A separate model instance - The Interlocutor - is instructed to simulate a “human using a chatbot”
3. A conversation topic is induced by manually setting the first Interlocutor’s message (e.g. Tell me a joke)
4. A conversation is simulated
5. A question from the questionnaire is set as the last Interlocutor’s last message and The Tested model’s response is recorded (this is repeated for every item in the questionnaire)
6. The questionnaire is scored to obtain scores for the 10 personal values

We aim to score the expressed value profile for each simulated persona in different contexts. More precisely a population (50 personas) is evaluated with a context chunk (50 topics: one per persona). Then, the simulated population in one context chunk is compared to the same population in another context chunk. Here are the considered context chunks:

• no_conv : no conversation is simulated the questions from the PVQ-40 questionnaire are given directly
• no_conv_svs : no conversation is simulated the questions from the SVS questionnaire are given directly
• chunk_0-chunk-4 : 50 reddit posts used as the initial Interlocutor model messages (one per persona). chunk_0 contains the longest posts, chunk_4 the shortest.
• chess : "1. e4" is given as the initial message to all personas, but for each persona the Interlocutor model is instructed to simulate a different persona (instead of a human user)
• grammar : like chess, but "Can you check this sentence for grammar? \n Whilst Jane was waiting to meet hers friend their nose started bleeding." is given as the initial message.

You can browse the simulated population, questionnaires, and contexts used on our 🤗 StickToYourRole dataset.

Validity refers to the extent the questionnaire measures what it purports to measure. It can be seen as the questionnaire's accuracy in measuring the intended factors, i.e. values. Following the recommendations in this paper, the validation consists of two phases: Theory-Based Multidimensional Scaling (MDS) and Confirmatory Factor Analysis (CFA).

Theory-Based Multidimensional Scaling (MDS) tests that the expressed values are organized in a circular structure as predicted by the theory. Values should be ordered in a circle in the same order as shown on the figure below (Tradition and Conformity should be on the same angle with Tradition closer to the center). To compute the structure in our data, we calculate the intercorrelations between different items (questions). This provides us with 40 points in a 40D space (for PVQ-40), which is space is then reduced to 2D by MDS. Crucially, MDS is initialized with the theoretical circular value structure, i.e. items corresponding to the same value are assigned the same angle. When MDS is fit, it provides the Stress (↓) metric ('Stress-1 index') indicating the goodness of the fit. A value of 0 indicates 'perfect' fit, 0.025 excellent, 0.05 good, 0.1 fair, and 0.2 poor.

Confirmatory Factor Analysis (CFA) fits a model on the data. The model is defined according to the theory and the fit of this model is used as a metric. Due to the circular structure of basic personal values, it is recommended to employ a Magnifying glass CFA strategy. Four separate models are fit, one for each of the high level values (consisting of several low-level values): Conservation (security, conformity, tradition), Openness to Change (self-direction, stimulation, hedonism), Self-transcendence (benevolence, universalism), Self-enhancement (achievement, power). Fit is measured with three standard metrics: Comparative Fit Index - CFI (↑) - compares the fit of a model to a more restricted baseline model (>.90 considered acceptable fit). Standardized root mean square residual - SRMR (↓) compares the sample variances and covariances to the estimated ones. (<.05 considered good fit, <.08 considered reasonable fit). Root mean square error of approximation - RMSEA (↓) reflects the degree to which a model fits the population covariance matrix, while taking into account the degrees of freedom and sample size (<.05 considered good fit; < .08 considered reasonable fit).

Rank-Order stability (↑) is used to estimate the stability of some value inside a population. In psychology, it is computed as the correlation in the order of individuals at two points in time (individuals are ordered based on their expression of that value). Intuitively, this can be seen as addressing the following question: "Does Jack always (in every context) value Tradition more than Jane does?". As shown below, instead of comparing two points in time, we compare the simulated population in different contexts (simulated conversations of different topics). We then average over different context pairs and values to obtain the final estimate.

To rank models, we aggregate the rank-order and validity metrics in two ways :

• Cardinal - Score (↑) - the score is averaged over all metrics (with descending metrics inverted), context pairs (for stability) and contexts (for validity metrics), i.e. the average of \( \binom{n\_context\_chunks}{2} + n\_validity\_metrics*n\_context\_chunks \) values.
• Ordinal - Win rate (↑) - for each metric, each context pair (for stability) and each context (for validity metrics) is considered as a game between two models, the win rate of a model is the percentage of won games against all models, i.e. the average of \( (n\_models-1) * ( \binom{n\_context\_chunks}{2} + n\_validity\_metrics*n\_context\_chunks) \).

Following this paper and associated benchbench library, we can compute the diversity and the sensitivity of the two ranking methods. A benchmark is considered diverse if different tasks order models in different ways. We use the reversed Kendall’s coefficient of concordance (W) diversity metric. A benchmark is considered sensitive if the model ordering is sensitive to the addition of new irrelevant models (for ordinal benchmarks), or to the label noise (for cardinal benchmarks). We use the max rank change (MRC) sensitivity metric.

This leaderboard is grounded in the methodology presented in our research paper. The paper contains various experiments which are not included in the leaderboard such as: multiple populations, within-person stability, stability on downstream tasks, correlations of value expression and behavior on downstream tasks, and so on. The leaderboard focused on population-level stability (Rank-Order) and contains various additions to the methodology. These changes were made to keep up with the newly released model and to make the evaluation more detailed. We describe additions made in the leaderboard here for clarity:

1. a new population was created and was balanced with respect to gender
2. context chunks - instead of evaluating the stability of a population between pairs of contexts, where all personas are given the same topic (e.g. chess), we evaluate it between pairs of context chunks, where each participant is given a different random context
3. more diverse and longer contexts (up to 6k tokens) were created with reddit posts from the webis dataset (the dataset was cleaned to exclude posts from NSFW subreddits)
4. different interlocutors - chess and grammar topic were still introduced as in the paper (same context for all participants), but the interlocutor model was instructed to simulate a random persona from the same population (as opposed to a human user in other settings)
5. in the paper, multiple seeds for the order of suggested answers were used, given that the results didn't vary much between seeds, here, a single seed was used facilitating the analysis with more longer contexts
6. evaluations were also done without simulating conversations (no_conv setting)
7. evaluations were also done with the SVS questionnaire (in the no_conv setting)
8. validation metrics - Stress, CFI, SRMR, RMSEA metrics were introduced
9. cardinal and ordinal ordering with sensitivity and diversity estimates were added
10. newer models were evaluated