mlcroissant
mlcroissant is a library to load datasets from Croissant metadata.
💡 Learn more about how to get the metadata from the dataset viewer API in the Get Croissant metadata guide.
Let’s start by parsing the Croissant metadata for the barilan/blog_authorship_corpus
dataset. Be sure to first install mlcroissant[parquet]
and GitPython
to be able to load Parquet files over the git+https protocol.
from mlcroissant import Dataset
ds = Dataset(jsonld="https://huggingface.co/api/datasets/barilan/blog_authorship_corpus/croissant")
To read from the first subset (called RecordSet in Croissant’s vocabulary), use the records
function, which returns an iterator of dicts.
records = ds.records(ds.metadata.record_sets[0].uid)
Finally use Pandas to compute your query on the first 1,000 rows:
import itertools
import pandas as pd
df = (
pd.DataFrame(list(itertools.islice(records, 1000)))
.groupby("horoscope")["text"]
.apply(lambda x: x.str.len().mean())
.sort_values(ascending=False)
.head(5)
)
print(df)
horoscope
b'Sagittarius' 1216.000000
b'Libra' 862.615581
b'Capricorn' 381.269231
b'Cancer' 272.776471
Name: text, dtype: float64