formal-grammar-llm-benchmark

简介

A formal grammar benchmark to study learning and memorization of large language models Usage Load a specific grammar (config) and split: from datasets import load_dataset ds = load_dataset( "<username>/formal-grammar-llm-benchmark", name="pcfg_cfg3b_eq_len_skewed_prob", ) # available splits per grammar: # train_sequences, test_sequences, non_grammatical_sequences, # non_grammatical_*_grammar_edit_*, non_grammatical_*_edit_distance_*… See the full description on the dataset page: https://huggingface.co/datasets/hoisoserious/formal-grammar-llm-benchmark.