Updating content

templates/about.html

@@ -181,18 +181,30 @@
         </div>
         <div class="section">
             <div class="section-title">Motivation</div>
+            <p>
+                LLMs can role-play different personas by simulating their values and behavior, but can they stick to their role whatever the context?
+                Is simulated Joan of Arc more tradition-driven than Elvis?
+                Will it still be the case after playing chess?
+            </p>
             <p>
                 Benchmarks usually compare models with <b>many questions</b> from <b>a single minimal context</b>, e.g. as multiple choices questions.
-                This kind of evaluation is little informative of LLMs' behavior in deployment when exposed to new contexts (especially when we consider the LLMs highly context-dependant nature).
-                We argue that <b>context-dependence</b> can be seen as a <b>property of LLMs</b>: a dimension of LLM comparison alongside others like size, speed, or knowledge.
+                This kind of evaluation is little informative of LLMs' behavior in deployment, where they are exposed to new contexts (especially considering the LLMs highly context-dependant nature).
+                We argue that <b>undesired context-dependence</b> can be seen as a <b>property of LLMs</b>: a dimension of LLM comparison alongside others like size, speed, or knowledge.
+                While some context-dependence is desired (e.g. following instructions), some is undesired (e.g. drastically changing the simulated value expression based on the interlocutor).
                 We evaluate LLMs by asking the <b> same questions </b> from <b> many different contexts </b>.
             </p>
             <p>
-                LLMs are often used to simulate personas and populations.
-                We study the coherence of simulated populations over different contexts (conversations on different topics).
-                To do that we leverage the psychological methodology to study the interpersonal stability of personal value expression of those simulated populations.
-                We adopt the Schwartz Theory of Basic Personal Values that defines 10 values: Self-Direction, Stimulation, Hedonism, Achievement, Power, Security, Conformity, Tradition, Benevolence, and Universalism,
-                to evaluate their expression we use the associated questionnaires: PVQ-40, and SVS.
+                LLMs are often used to simulate personas and populations, we study the stability and coherence of a simulated <b>population</b> - in contrast to evaluating each persona separately, <b>we evaluate the stability of simulated personas relative to each other, i.e. as a population</b>.
+                We study simulated populations over different contexts, i.e. conversations on different topics.
+                To do that, we leverage the psychological methodology to study the interpersonal stability of personal value expression in a simulated population.
+                We adopt the <b>Schwartz Theory of Basic Personal Values</b>, which defines 10 values: Self-Direction, Stimulation, Hedonism, Achievement, Power, Security, Conformity, Tradition, Benevolence, and Universalism.
+                To evaluate their expression we use the associated questionnaires: <b>PVQ-40</b>, and <b>SVS</b>.
+            </p>
+            <p>
+                The Stick to Your Role! leaderboard aims to provide an up-to-date comparison of recent LLMs based on their ability to coherently simulate popultions.
+                It, in tandem with other minimal-context benchmarks, should enable you to choose the best-suited model for your usecase!
+                If you want to evaluate or add your model you can follow the instructions
+                <a href="{{ url_for('new_model') }}">here</a>.
             </p>
         </div>
         <div class="section">
@@ -234,7 +246,7 @@
             <div class="section-title">Validation</div>
             <p>
                 Validity refers to the extent the questionnaire measures what it purports to measure.
-                It can be seen the questionnaire's accuracy in measuring the intended factors, i.e. values.
+                It can be seen as the questionnaire's accuracy in measuring the intended factors, i.e. values.
                 Following the recommendations in <a href="https://pubmed.ncbi.nlm.nih.gov/22329443/">this paper</a>,
                 the validation consists of two phases: Theory-Based Multidimensional Scaling (MDS) and Confirmatory Factor Analysis (CFA).
             </p>

templates/index.html

@@ -23,13 +23,18 @@
             box-shadow: 0 4px 8px rgba(0,0,0,0.1);
         }
 
-        h1 {
+        .container h1 {
             color: #333;
             text-align: center;
         }
-        h2 {
-            color: #333;
-            margin-top: 30px;
+
+        .container h3 {
+<!--            font-weight: normal;-->
+<!--            color: #333;-->
+            margin-top: 10px;
+            margin-bottom: 50px;
+            margin-left: 20px;
+            margin-right: 20px;
             text-align: center;
         }
 
@@ -192,20 +197,56 @@
             text-align: center;
         }
 
+        /* Tooltip styling */
+        .tooltip-inner {
+            max-width: none;
+        }
+
+        th[title] {
+            position: relative;
+            cursor: help;
+        }
+
+        th[title]:hover::after {
+            content: attr(title);
+            position: absolute;
+            bottom: 100%;
+            left: 50%;
+            transform: translateX(-50%);
+            background-color: #333;
+            color: white;
+            padding: 5px 10px;
+            border-radius: 4px;
+            white-space: nowrap;
+            z-index: 1;
+            font-weight: normal;
+            font-size: 14px;
+        }
     </style>
 </head>
 <body>
     <div class="container">
         <h1 class="mt-5">Stick To Your Role! Leaderboard</h1>
+        <h3>
+            LLMs can role-play different personas by simulating their values and behavior, but can they stick to their role whatever the context?
+            Is simulated Joan of Arc more tradition-driven than Elvis?
+            Will it still be the case after playing chess?
+        </h3>
         <p>
             The Stick to Your Role! leaderboard compares LLMs based on <b>undesired sensitivity to context change</b>.
-            It focuses on the stability of personal value expression in simulated personas.
+            LLM-exhibited behavior always depends on the context (prompt), while some context-dependence is desired (e.g. following instructions),
+            some is undesired (e.g. drastically changing the simulated value expression based on the interlocutor).
             As proposed in our <a href="https://arxiv.org/abs/2402.14846">paper</a>,
-            unwanted context-dependence should be seen as a <b>property of LLMs</b> - a dimension of LLM comparison (alongside others such as model size speed or expressed knowledge).
+            undesired context-dependence should be seen as a <b>property of LLMs</b> - a dimension of LLM comparison (alongside others such as model size speed or expressed knowledge).
             This leaderboard aims to provide such a comparison and extends our paper with a more focused and elaborate experimental setup.
             Standard benchmarks present <b>many</b> questions from the <b>same minimal contexts</b> (e.g. multiple choice questions),
             we present <b>same</b> questions from <b>many different contexts</b>.
         </p>
+        <p>
+            The Stick to You Role! leaderboard focuses on the <b>stability of simulated personal values during role-playing</b>.
+            We study the <b>coherence of a simulated population</b>.
+            In contrast to evaluating each simulated persona separately, we evaluate personas relative to each other, i.e. as a population.
+        </p>
         <div class="table-responsive main-table">
             <!-- Render the table HTML here -->
             {{ main_table_html|safe }}
@@ -231,7 +272,7 @@
         <p>
             In addition to Rank-Order stability we compute <b>validity metrics (Stress, Separability, CFI, SRMR, RMSEA)</b>, which are a common practice in psychology.
             Validity refers to the extent the questionnaire measures what it purports to measure.
-            It can be seen the questionnaire's accuracy in measuring the intended factors, i.e. values.
+            It can be seen as the questionnaire's accuracy in measuring the intended factors, i.e. values.
             For example, basic personal values should be organized in a circular structure, and questions measuring the same value should be correlated.
             The table below additionally shows the validity metrics, refer <a href="{{ url_for('about', _anchor='metrics') }}">here</a> for more details.
         </p>
@@ -245,11 +286,11 @@
                 <li><b>RO-stability</b>: the correlation in the order of simulated participants (ordered based on the expression of the same values) over different contexts</li>
                 <!--Validation metrics:-->
                 <li><b>Stress</b>: the MDS fit of the observed value structure to the theoretical circular structure. Stress of 0 indicates 'perfect' fit, 0.025 excellent, 0.05 good, 0.1 fair, and 0.2 poor.</li>
-                <li><b>Separability</b>: the extent to which questions corresponding to different values are linearly separable in the 2D MDS space (linear multi-label classifier accuracy)</li>
-                <li><b>CFI, SRMR, RMSEA</b>: Common Confirmatory Factor Analysis (CFA) metrics showing the fit between the data and the posited model of the relation of items (questions) to factors (values), applied here with Magnifying Glass CFA. For CFI >.90 is considered acceptable fit, for SRMR and RMSEA is <.05 considered good fit and <.08 reasonable.</li>
+                <li><b>Separability</b>: the extent to which questions corresponding to different values are linearly separable in the 2D MDS space (linear multi-label SVM classifier accuracy)</li>
+                <li><b>CFI, SRMR, RMSEA</b>: Common Confirmatory Factor Analysis (CFA) metrics showing the fit of the posited model of the relation of items (questions) to factors (values) on the observed data, applied here with Magnifying Glass CFA. For CFI >.90 is considered acceptable fit, for SRMR and RMSEA is <.05 considered good fit and <.08 reasonable.</li>
                 <!--Aggregate metrics:-->
                 <li><b>Ordinal - Win Rate</b>: the score averaged over all metrics (with descending metrics inverted), context pairs (for stability) and contexts (for validity metrics)</li>
-                <li><b>Cardinal - Score</b>: percentage of won games against all models, where a game is a comparison between two models for each metric, each context pair (for stability) and each context (for validity metrics)</li>
+                <li><b>Cardinal - Score</b>: the percentage of won games, where a game is a comparison each model pair, each metric, and each context pair (for stability) or context (for validity metrics)</li>
             </ul>
         </p>
         <div class="table-responsive full-table">
@@ -280,7 +321,7 @@
         </div>
         <ul>
             <li>Contact: <a href="mailto: [email protected]">[email protected]</a></li>
-            <li>See the <a href="https://sites.google.com/view/llmvaluestability">Project website<a/></li>
+            <li>See the <a href="https://sites.google.com/view/llmvaluestability">Project website</a></li>
             <li>See the Flowers team <a href="http://developmentalsystems.org">blog</a> and <a href="https://flowers.inria.fr/">website</a></li>
             <li>See Grgur's website and other projects: <a href="https://grgkovac.github.io">https://grgkovac.github.io</a></li>
         </ul>