error-analysis app

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -1,13 +1,9 @@
 ---
-title: Error Analysis
-emoji: 😻
-colorFrom: red
-colorTo: green
+title: Interactive Error Analysis
+emoji: 🐛
+colorFrom: yellow
+colorTo: orange
 sdk: streamlit
-sdk_version: 1.9.0
 app_file: app.py
 pinned: false
-license: apache-2.0
 ---
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces#reference

app.py

@@ -0,0 +1,283 @@
+## LIBRARIES ###
+## Data
+import numpy as np
+import pandas as pd
+import torch
+import json
+from tqdm import tqdm
+from math import floor
+from datasets import load_dataset
+from collections import defaultdict
+from transformers import AutoTokenizer
+pd.options.display.float_format = '${:,.2f}'.format
+
+# Analysis
+# from gensim.models.doc2vec import Doc2Vec
+# from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score
+import nltk
+from nltk.cluster import KMeansClusterer
+import scipy.spatial.distance as sdist
+from scipy.spatial import distance_matrix
+# nltk.download('punkt') #make sure that punkt is downloaded
+
+# App & Visualization
+import streamlit as st
+import altair as alt
+import plotly.graph_objects as go
+from streamlit_vega_lite import altair_component
+
+
+
+# utils
+from random import sample
+from error_analysis import utils as ut
+
+
+def down_samp(embedding):
+    """Down sample a data frame for altiar visualization """
+    # total number of positive and negative sentiments in the class
+    #embedding = embedding.groupby('slice').apply(lambda x: x.sample(frac=0.3))
+    total_size = embedding.groupby(['slice','label'], as_index=False).count()
+
+    user_data = 0
+    # if 'Your Sentences' in str(total_size['slice']):
+    #     tmp = embedding.groupby(['slice'], as_index=False).count()
+    #     val = int(tmp[tmp['slice'] == "Your Sentences"]['source'])
+    #     user_data = val
+
+    max_sample = total_size.groupby('slice').max()['content']
+
+    # # down sample to meeting altair's max values
+    # # but keep the proportional representation of groups
+    down_samp = 1/(sum(max_sample.astype(float))/(1000-user_data))
+
+    max_samp = max_sample.apply(lambda x: floor(x*down_samp)).astype(int).to_dict()
+    max_samp['Your Sentences'] = user_data
+
+    # # sample down for each group in the data frame
+    embedding = embedding.groupby('slice').apply(lambda x: x.sample(n=max_samp.get(x.name))).reset_index(drop=True)
+
+    # # order the embedding
+    return(embedding)
+
+
+def data_comparison(df):
+    selection = alt.selection_multi(fields=['cluster:N','label:O'])
+    color = alt.condition(alt.datum.slice == 'high-loss', alt.Color('cluster:N', scale = alt.Scale(domain=df.cluster.unique().tolist())), alt.value("lightgray"))
+    opacity = alt.condition(selection, alt.value(0.7), alt.value(0.25))
+
+    # basic chart
+    scatter = alt.Chart(df).mark_point(size=100, filled=True).encode(
+        x=alt.X('x:Q', axis=None),
+        y=alt.Y('y:Q', axis=None),
+        color=color,
+        shape=alt.Shape('label:O', scale=alt.Scale(range=['circle', 'diamond'])),
+        tooltip=['cluster:N','slice:N','content:N','label:O','pred:O'],
+        opacity=opacity
+    ).properties(
+        width=1000,
+        height=800
+    ).interactive()
+
+    legend = alt.Chart(df).mark_point(size=100, filled=True).encode(
+        x=alt.X("label:O"),
+        y=alt.Y('cluster:N', axis=alt.Axis(orient='right'), title=""),
+        shape=alt.Shape('label:O', scale=alt.Scale(
+        range=['circle', 'diamond']), legend=None),
+        color=color,
+    ).add_selection(
+        selection
+    )
+    layered = scatter | legend
+    layered = layered.configure_axis(
+        grid=False
+    ).configure_view(
+        strokeOpacity=0
+    )
+    return layered
+
+def quant_panel(embedding_df):
+    """ Quantitative Panel Layout"""
+    all_metrics = {}
+    st.warning("**Error slice visualization**")
+    with st.expander("How to read this chart:"):
+        st.markdown("* Each **point** is an input example.")
+        st.markdown("* Gray points have low-loss and the colored have high-loss. High-loss instances are clustered using **kmeans** and each color represents a cluster.")
+        st.markdown("* The **shape** of each point reflects the label category --  positive (diamond) or negative sentiment (circle).")
+    st.altair_chart(data_comparison(down_samp(embedding_df)), use_container_width=True)
+
+
+def frequent_tokens(data, tokenizer, loss_quantile=0.95, top_k=200, smoothing=0.005):
+    unique_tokens = []
+    tokens = []
+    for row in tqdm(data['content']):
+        tokenized = tokenizer(row,padding=True, return_tensors='pt')
+        tokens.append(tokenized['input_ids'].flatten())
+        unique_tokens.append(torch.unique(tokenized['input_ids']))
+    losses = data['loss'].astype(float)
+    high_loss = losses.quantile(loss_quantile)
+    loss_weights = (losses > high_loss)
+    loss_weights = loss_weights / loss_weights.sum()
+    token_frequencies = defaultdict(float)
+    token_frequencies_error = defaultdict(float)
+
+    weights_uniform = np.full_like(loss_weights, 1 / len(loss_weights))
+
+    num_examples = len(data)
+    for i in tqdm(range(num_examples)):
+        for token in unique_tokens[i]:
+            token_frequencies[token.item()] += weights_uniform[i]
+            token_frequencies_error[token.item()] += loss_weights[i]
+
+    token_lrs = {k: (smoothing+token_frequencies_error[k]) / (smoothing+token_frequencies[k]) for k in token_frequencies}
+    tokens_sorted = list(map(lambda x: x[0], sorted(token_lrs.items(), key=lambda x: x[1])[::-1]))
+
+    top_tokens = []
+    for i, (token) in enumerate(tokens_sorted[:top_k]):
+        top_tokens.append(['%10s' % (tokenizer.decode(token)), '%.4f' % (token_frequencies[token]), '%.4f' % (
+            token_frequencies_error[token]), '%4.2f' % (token_lrs[token])])
+    return pd.DataFrame(top_tokens, columns=['Token', 'Freq', 'Freq error slice', 'lrs'])
+
+
+@st.cache(ttl=600)
+def get_data(inference, emb):
+    preds = inference.outputs.numpy()
+    losses = inference.losses.numpy()
+    embeddings = pd.DataFrame(emb, columns=['x', 'y'])
+    num_examples = len(losses)
+    # dataset_labels = [dataset[i]['label'] for i in range(num_examples)]
+    return pd.concat([pd.DataFrame(np.transpose(np.vstack([dataset[:num_examples]['content'], 
+                    dataset[:num_examples]['label'], preds, losses])), columns=['content', 'label', 'pred', 'loss']), embeddings], axis=1)
+
+def clustering(data,num_clusters):
+    X = np.array(data['embedding'].tolist())
+    kclusterer = KMeansClusterer(
+        num_clusters, distance=nltk.cluster.util.cosine_distance,
+        repeats=25,avoid_empty_clusters=True)
+    assigned_clusters = kclusterer.cluster(X, assign_clusters=True)
+    data['cluster'] = pd.Series(assigned_clusters, index=data.index).astype('int')
+    data['centroid'] = data['cluster'].apply(lambda x: kclusterer.means()[x])
+    return data, assigned_clusters
+
+def kmeans(df, num_clusters=3):
+    data_hl = df.loc[df['slice'] == 'high-loss']
+    data_kmeans,clusters = clustering(data_hl,num_clusters)
+    merged = pd.merge(df, data_kmeans, left_index=True, right_index=True, how='outer', suffixes=('', '_y'))
+    merged.drop(merged.filter(regex='_y$').columns.tolist(),axis=1,inplace=True)
+    merged['cluster'] = merged['cluster'].fillna(num_clusters).astype('int')
+    return merged
+
+def distance_from_centroid(row):
+    return sdist.norm(row['embedding'] - row['centroid'].tolist())
+
+@st.cache(ttl=600)
+def topic_distribution(weights, smoothing=0.01):
+    topic_frequencies = defaultdict(float)
+    topic_frequencies_spotlight = defaultdict(float)
+    weights_uniform = np.full_like(weights, 1 / len(weights))
+    num_examples = len(weights)
+    for i in range(num_examples):
+        example = dataset[i]
+        category = example['title']
+        topic_frequencies[category] += weights_uniform[i]
+        topic_frequencies_spotlight[category] += weights[i]
+
+    topic_ratios = {c: (smoothing + topic_frequencies_spotlight[c]) / (
+        smoothing + topic_frequencies[c]) for c in topic_frequencies}
+
+    categories_sorted = map(lambda x: x[0], sorted(
+        topic_ratios.items(), key=lambda x: x[1], reverse=True))
+
+    topic_distr = []
+    for category in categories_sorted:
+        topic_distr.append(['%.3f' % topic_frequencies[category], '%.3f' %
+                           topic_frequencies_spotlight[category], '%.2f' % topic_ratios[category], '%s' % category])
+
+    return pd.DataFrame(topic_distr, columns=['Overall frequency', 'Error frequency', 'Ratio', 'Category'])
+    # for category in categories_sorted:
+    #    return(topic_frequencies[category], topic_frequencies_spotlight[category], topic_ratios[category], category)
+
+def populate_session(dataset,model):
+    data_df = read_file_to_df('./assets/data/'+dataset+ '_'+ model+'.parquet')
+    if model == 'albert-base-v2-yelp-polarity':
+        tokenizer = AutoTokenizer.from_pretrained('textattack/'+model)
+    else:
+        tokenizer = AutoTokenizer.from_pretrained(model)
+    if "user_data" not in st.session_state:
+        st.session_state["user_data"] = data_df
+    if "selected_slice" not in st.session_state:
+        st.session_state["selected_slice"] = None
+ 
+@st.cache(allow_output_mutation=True)
+def read_file_to_df(file):
+   return pd.read_parquet(file)
+
+if __name__ == "__main__":
+    ### STREAMLIT APP CONGFIG ###
+    st.set_page_config(layout="wide", page_title="Interactive Error Analysis")
+
+    ut.init_style()
+
+    lcol, rcol = st.columns([2, 2])
+    # ******* loading the mode and the data
+    #st.sidebar.mardown("<h4>Interactive Error Analysis</h4>", unsafe_allow_html=True)
+
+    dataset = st.sidebar.selectbox(
+        "Dataset",
+        ["amazon_polarity", "yelp_polarity"],
+        index = 1
+    )
+
+    model = st.sidebar.selectbox(
+        "Model",
+        ["distilbert-base-uncased-finetuned-sst-2-english",
+            "albert-base-v2-yelp-polarity"],
+    )
+
+    ### LOAD DATA AND SESSION VARIABLES ###
+    ##uncomment the next next line to run dynamically and not from file
+    #populate_session(dataset, model)
+    data_df = read_file_to_df('./assets/data/'+dataset+ '_'+ model+'.parquet')
+    loss_quantile = st.sidebar.slider(
+        "Loss Quantile", min_value=0.5, max_value=1.0,step=0.01,value=0.95
+    )
+    data_df['loss'] = data_df['loss'].astype(float)
+    losses = data_df['loss']
+    high_loss = losses.quantile(loss_quantile)
+    data_df['slice'] = 'high-loss'
+    data_df['slice'] = data_df['slice'].where(data_df['loss'] > high_loss, 'low-loss') 
+
+    with rcol:
+        with st.spinner(text='loading...'):
+            st.markdown('<h3>Word Distribution in Error Slice</h3>', unsafe_allow_html=True)
+            #uncomment the next two lines to run dynamically and not from file
+            #commontokens = frequent_tokens(data_df, tokenizer, loss_quantile=loss_quantile)
+            commontokens = read_file_to_df('./assets/data/'+dataset+ '_'+ model+'_commontokens.parquet')
+            with st.expander("How to read the table:"):
+                st.markdown("* The table displays the most frequent tokens in error slices, relative to their frequencies in the val set.")
+            st.write(commontokens)
+
+    run_kmeans = st.sidebar.radio("Cluster error slice?", ('True', 'False'), index=0)
+
+    num_clusters = st.sidebar.slider("# clusters", min_value=1, max_value=20, step=1, value=3)
+
+    if run_kmeans == 'True':
+        with st.spinner(text='running kmeans...'):
+            merged = kmeans(data_df,num_clusters=num_clusters)
+    with lcol:
+        st.markdown('<h3>Error Slices</h3>',unsafe_allow_html=True)
+        with st.expander("How to read the table:"):
+            st.markdown("* *Error slice* refers to the subset of evaluation dataset the model performs poorly on.")
+            st.markdown("* The table displays model error slices on the evaluation dataset, sorted by loss.")
+            st.markdown("* Each row is an input example that includes the label, model pred, loss, and error cluster.")
+        with st.spinner(text='loading error slice...'):
+            dataframe=read_file_to_df('./assets/data/'+dataset+ '_'+ model+'_error-slices.parquet')
+        #uncomment the next next line to run dynamically and not from file
+        # dataframe = merged[['content', 'label', 'pred', 'loss', 'cluster']].sort_values(
+        #     by=['loss'], ascending=False)
+        # table_html = dataframe.to_html(
+        #     columns=['content', 'label', 'pred', 'loss', 'cluster'], max_rows=50)
+        # table_html = table_html.replace("<th>", '<th align="left">')  # left-align the headers
+            st.write(dataframe,width=900, height=300)
+    with st.spinner(text='loading visualization...'):
+        quant_panel(merged)

assets/data/amazon_polarity.test.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e57ae9ce39c5251e432b4a6dce31915782276b98a7751281eb66b8cff3b46b6
+size 5864011

assets/data/amazon_polarity_albert-base-v2-yelp-polarity.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bce0297bedc66865c01644421ea934008d74807befb7b0bd94aa92729bd02a59
+size 56644779

assets/data/amazon_polarity_albert-base-v2-yelp-polarity_commontokens.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de69efcda9ab5c3aa8dc616c016cace08096cbc21478dd894f9cccf0b843ede4
+size 6067

assets/data/amazon_polarity_albert-base-v2-yelp-polarity_error-slices.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62ce63230551fe9870919f051dfeead6892bb917ba63c7edfcc0e819867ed2cd
+size 5954640

assets/data/amazon_polarity_distilbert-base-uncased-finetuned-sst-2-english.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a193c26851f48b7b76a35986ced0dc1fddafd26b92f1aaf9a4e69fd83fd2f2e4
+size 56643545

assets/data/amazon_polarity_distilbert-base-uncased-finetuned-sst-2-english_commontokens.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de69efcda9ab5c3aa8dc616c016cace08096cbc21478dd894f9cccf0b843ede4
+size 6067

assets/data/amazon_polarity_distilbert-base-uncased-finetuned-sst-2-english_error-slices.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6626d19361cfe06ba70be19004d18eb23a1764926d15ed5b103ec36fc2d8eaea
+size 5954642

assets/data/amazon_test_emb.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eaf8bcf2691858dc39d0f83872f425a839b05de4b916c111cba6e9c69747e467
+size 50449725

assets/data/yelp_polarity_albert-base-v2-yelp-polarity.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a56147880841c6f78a868fb58f6e97661547009e570c2887ef7c12ffd54474e
+size 103294569

assets/data/yelp_polarity_albert-base-v2-yelp-polarity_commontokens.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e5241d13a23656bbab7851d4a8ad1df70d8675e9db62f3e9ade719b41c524db
+size 6681

assets/data/yelp_polarity_albert-base-v2-yelp-polarity_error-slices.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7140867800b4e9ac5f34b5c181ec69c261638582ef55c142aaf23320e9e56743
+size 9765767

assets/data/yelp_polarity_distilbert-base-uncased-finetuned-sst-2-english.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:165515be2837df9b02f782fe1e7bd3b31bb01c49960e73238f77541eee7589ad
+size 61796202

assets/data/yelp_polarity_distilbert-base-uncased-finetuned-sst-2-english_commontokens.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c243229a8f55a1910fcab1823cd4810d51a59c6442244a47ee5ee621da069518
+size 6509

assets/data/yelp_polarity_distilbert-base-uncased-finetuned-sst-2-english_error-slices.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ec890d0df88c0d7fdaba7ca9e50f715c50d173f99ae021fd4c49534d4ef12a9
+size 9803781

error_analysis/utils/__init__.py

@@ -0,0 +1 @@
+from .style_hacks import *

error_analysis/utils/style_hacks.py

@@ -0,0 +1,86 @@
+"""
+ placeholder for all streamlit style hacks
+"""
+import streamlit as st
+
+
+def init_style():
+    return st.markdown(
+        """
+    <style>
+    /* Side Bar */
+    [data-testid="stSidebar"][aria-expanded="true"] > div:first-child {
+         width: 250px;
+       }
+    [data-testid="stSidebar"][aria-expanded="false"] > div:first-child {
+        width: 250px;
+    }
+    [data-testid="stSidebar"]{
+        flex-basis: unset;
+    }
+    .css-1outpf7 {
+        background-color:rgb(254 244 219);
+        width:10rem;
+        padding:10px 10px 10px 10px;
+    }
+
+    /* Main Panel*/  
+    .css-18e3th9 {
+        padding:10px 10px 10px -200px;
+    }
+    .css-1ubw6au:last-child{
+        background-color:lightblue;
+    }
+
+    /* Model Panels : element-container */
+    .element-container{
+            border-style:none
+    }
+
+    /* Radio Button Direction*/
+    div.row-widget.stRadio > div{flex-direction:row;}
+
+    /* Expander Boz*/
+    .streamlit-expander {
+        border-width: 0px;
+        border-bottom: 1px solid #A29C9B;
+        border-radius: 10px;
+    }
+
+    .streamlit-expanderHeader {
+        font-style: italic;
+        font-weight :600;
+        font-size:16px;
+        padding-top:0px;
+        padding-left: 0px;
+        color:#A29C9B
+
+    /* Section Headers */
+    .sectionHeader {
+        font-size:10px;
+    }
+    [data-testid="stMarkdownContainer]{
+        font-family: sans-serif;
+        font-weight: 500;
+        font-size: 1.5 rem !important;
+        color: rgb(250, 250, 250);
+        padding: 1.25rem 0px 1rem;
+        margin: 0px;
+        line-height: 1.4;
+    }
+
+    /* text input*/
+    .st-e5 {
+        background-color:lightblue;
+    }
+    /*line special*/
+    .line-one{
+        border-width: 0px;
+        border-bottom: 1px solid #A29C9B;
+        border-radius: 50px; 
+    }
+
+    </style>
+""",
+        unsafe_allow_html=True,
+    )

requirements.txt

@@ -0,0 +1,139 @@
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: osx-arm64
+absl-py==1.0.0; python_version >= '3.6'
+aiohttp==3.8.0
+aiosignal==1.2.0; python_version >= '3.6'
+altair==4.1.0
+antlr4-python3-runtime==4.8
+appnope==0.1.2; sys_platform == 'darwin' and platform_system == 'Darwin'
+argon2-cffi==21.1.0; python_version >= '3.5'
+astor==0.8.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+async-timeout==4.0.1; python_version >= '3.6'
+attrs==21.2.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
+backcall==0.2.0
+backports.zoneinfo==0.2.1; python_version >= '3.6' and python_version < '3.9'
+base58==2.1.1; python_version >= '3.5'
+bleach==4.1.0; python_version >= '3.6'
+blinker==1.4
+cachetools==4.2.4; python_version ~= '3.5'
+certifi==2021.10.8
+cffi==1.15.0
+charset-normalizer==2.0.7; python_version >= '3'
+click==7.1.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
+cython==0.29.24; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
+cytoolz==0.11.2; python_version >= '3.5'
+dataclasses==0.6
+datasets==1.15.1
+debugpy==1.5.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
+decorator==5.1.0; python_version >= '3.5'
+defusedxml==0.7.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
+dill==0.3.4; python_version >= '2.7' and python_version != '3.0'
+entrypoints==0.3; python_version >= '2.7'
+fastbpe==0.1.0
+filelock==3.3.2; python_version >= '3.6'
+frozenlist==1.2.0; python_version >= '3.6'
+fsspec[http]==2021.11.0; python_version >= '3.6'
+future==0.18.2; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
+fuzzywuzzy==0.18.0
+gitdb==4.0.9; python_version >= '3.6'
+gitpython==3.1.24; python_version >= '3.7'
+google-auth-oauthlib==0.4.6; python_version >= '3.6'
+google-auth==2.3.3; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
+grpcio==1.41.1
+idna==3.3; python_version >= '3'
+importlib-resources==5.4.0; python_version < '3.9'
+kaleido==0.2.1
+markdown==3.3.4; python_version >= '3.6'
+markupsafe==2.0.1; python_version >= '3.6'
+matplotlib-inline==0.1.3; python_version >= '3.5'
+meerkat-ml==0.1.2; python_version >= '3.7'
+mistune==0.8.4
+multidict==5.2.0; python_version >= '3.6'
+multiprocess==0.70.12.2
+nbclient==0.5.8; python_full_version >= '3.6.1'
+nbconvert==6.3.0; python_version >= '3.7'
+nbformat==5.1.3; python_version >= '3.5'
+nest-asyncio==1.5.1; python_version >= '3.5'
+nltk==3.6.5
+notebook==6.4.5; python_version >= '3.6'
+numpy==1.21.4
+oauthlib==3.1.1; python_version >= '3.6'
+omegaconf==2.1.1; python_version >= '3.6'
+packaging==21.2; python_version >= '3.6'
+pandas==1.3.4
+pandocfilters==1.5.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+parso==0.8.2; python_version >= '3.6'
+pexpect==4.8.0; sys_platform != 'win32'
+pickleshare==0.7.5
+pillow==8.4.0; python_version >= '3.6'
+plotly==5.3.1
+progressbar==2.5
+prometheus-client==0.12.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+prompt-toolkit==3.0.22; python_full_version >= '3.6.2'
+protobuf==3.19.1; python_version >= '3.5'
+ptyprocess==0.7.0; os_name != 'nt'
+pyahocorasick==1.4.2
+pyarrow==6.0.0; python_version >= '3.6'
+pyasn1-modules==0.2.8
+pyasn1==0.4.8
+pycparser==2.21; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+pydeck==0.7.1; python_version >= '3.7'
+pydeprecate==0.3.1; python_version >= '3.6'
+pygments==2.10.0; python_version >= '3.5'
+pympler==0.9
+pyparsing==2.4.7; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
+pyrsistent==0.18.0; python_version >= '3.6'
+python-dateutil==2.8.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+python-levenshtein==0.12.2
+pytorch-lightning==1.5.1; python_version >= '3.6'
+pytz-deprecation-shim==0.1.0.post0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
+pytz==2021.3
+pyyaml==6.0; python_version >= '3.6'
+pyzmq==22.3.0; python_version >= '3.6'
+regex==2021.11.10
+requests-oauthlib==1.3.0
+requests==2.26.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
+robustnessgym==0.1.3
+rsa==4.7.2; python_version >= '3.6'
+sacremoses==0.0.46
+scikit-learn==1.0.1; python_version >= '3.7'
+scipy==1.7.2; python_version < '3.11' and python_version >= '3.7'
+semver==2.13.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+send2trash==1.8.0
+six==1.16.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+sklearn==0.0
+smart-open==5.2.1; python_version >= '3.6' and python_version < '4'
+smmap==5.0.0; python_version >= '3.6'
+streamlit-vega-lite==0.1.0
+streamlit==1.2.0
+tenacity==8.0.1; python_version >= '3.6'
+tensorboard-data-server==0.6.1; python_version >= '3.6'
+tensorboard-plugin-wit==1.8.0
+tensorboard==2.7.0; python_version >= '3.6'
+terminado==0.12.1; python_version >= '3.6'
+testpath==0.5.0; python_version >= '3.5'
+threadpoolctl==3.0.0; python_version >= '3.6'
+tokenizers==0.10.3
+toml==0.10.2; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
+toolz==0.11.2; python_version >= '3.5'
+torch==1.10.0; python_full_version >= '3.6.2'
+torchmetrics==0.6.0; python_version >= '3.6'
+tornado==6.1; python_version >= '3.5'
+tqdm==4.62.3; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+traitlets==5.1.1; python_version >= '3.7'
+transformers==4.12.3; python_version >= '3.6'
+typing-extensions==3.10.0.2; python_version < '3.10'
+tzdata==2021.5; python_version >= '3.6'
+tzlocal==4.1; python_version >= '3.6'
+ujson==4.2.0; python_version >= '3.6'
+urllib3==1.26.7; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4'
+validators==0.18.2; python_version >= '3.4'
+wcwidth==0.2.5
+webencodings==0.5.1
+werkzeug==2.0.2; python_version >= '3.6'
+wheel==0.37.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
+widgetsnbextension==3.5.2
+xxhash==2.0.2; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
+yarl==1.7.2; python_version >= '3.6'
+zipp==3.6.0; python_version < '3.10'