Prose: a grammar-driven corpus generator for .NET

Prose reads plain text through the rules of grammar, keeps what it learns in a HoloDb database, and recombines that knowledge into new sentences that are grammatical, use words suited to their slots and read plausibly. It exists to grow a training corpus. Mine the grammatical knowledge already in your text, which nouns, verbs and adjectives go together and how, then generate more of it, along with question and answer pairs for reading-comprehension training data.

Language models need volume and variety of grammatically correct text. Writing or scraping more does not scale, and most data augmentation either pastes fragments together ungrammatically or needs a large model of its own. Prose takes a different route. It parses real text into its grammatical structure with deterministic rules, stores what nouns, verbs and adjectives actually go with, and recombines those observed pieces into fresh sentences that are checked to parse correctly before they are written out.

• Rules first, not model first

  Parsing, mining and generation all run on deterministic grammar rules with no training. An optional small language model can rank candidates by plausibility, but Prose falls back to pure rules and never depends on a model to produce correct output.

• No word is locked to one role

  The same surface form resolves to a noun or a verb by context, the way a reader who has learned grammar reads it. In the classic example, the panda eats shoots and leaves parses as one verb with two objects, while the panda eats, shoots, and leaves parses as three verbs. Same words: the punctuation and structure alone flip the reading.

• Nothing imaginary

  Every open-class word Prose emits is a form it observed in the source text, and it never invents an inflection. Generated sentences are re-parsed and checked by a validity gate (a real subject, a finite verb, subject and verb agreement, article and noun agreement) before they are kept. Question and answer pairs are checked the same way on both sides.

• Real grammatical structure

  Beyond simple subject, verb and object sentences, Prose mines and generates indirect objects (gave him a book), relative clauses (the panda that eats bamboo) and multi-sentence passages in which later sentences refer back to earlier ones by name or pronoun, the long-context structure comprehension training needs.

using Prose;

var prose = new ProseEngine();
prose.MineText("The wise king rules the peaceful kingdom. The kingdom is calm. " +
               "The king gives the poor man a book. The man reads the book. " +
               "The dog chases the red ball. The dog barks. The sky is blue. " +
               "The queen sends the man a letter. The queen is kind.");

foreach (var sentence in prose.Generate(5, seed: 1))
    Console.WriteLine(sentence);

A king rules the peaceful kingdom.
The peaceful ball barks.
A queen sends the letter.
A book chases the ball.
The peaceful kingdom rules the wise kingdom.

This is the complete program, compiled and run against the published package.

Parsing needs no mining and no database. The comma changes the parse, and the meaning with it.

using Prose;

// The same words, two different punctuation marks, two different parses.
foreach (var s in ProseEngine.Parse("The panda eats, shoots, and leaves. The panda eats shoots and leaves."))
    Console.WriteLine(ProseEngine.Explain(s));

The/Det panda/Noun eats/Verb shoots/Verb and/Cc leaves/Verb
template: DET NOUN VERB_FIN , VERB_FIN , CC VERB_FIN
  subject=panda  verbs=[eats,shoots,leaves]  indirectObject=-  objects=[-]  predicate=-
The/Det panda/Noun eats/Verb shoots/Noun and/Cc leaves/Noun
template: DET NOUN VERB_FIN NOUN CC NOUN
  subject=panda  verbs=[eats]  indirectObject=-  objects=[shoots,leaves]  predicate=-

The same tables produce reading-comprehension pairs: a context, a question and the answer the context supports.

using Prose;

var prose = new ProseEngine();
prose.MineText("The wise king rules the peaceful kingdom. The kingdom is calm. " +
               "The king gives the poor man a book. The man reads the book. " +
               "The dog chases the red ball. The dog barks. The sky is blue. " +
               "The queen sends the man a letter. The queen is kind.");

// Reading-comprehension pairs: a context, a question and the answer the context supports.
foreach (var pair in prose.GenerateQa(3, seed: 4))
    Console.WriteLine($"{pair.Context} | {pair.Question} -> {pair.Answer}");

// Multi-sentence passages, where later sentences refer back to earlier ones.
foreach (var pair in prose.GenerateQaPassages(2, sentencesPerPassage: 3, seed: 5))
    Console.WriteLine($"{pair.Question} -> {pair.Answer}");

A letter is wise. | What is a letter? -> Wise.
A letter is wise. | What is wise? -> A letter.
The poor man reads. | What reads? -> The poor man.
What reads the book? -> The man.
What kind of man reads the book? -> Poor.

For real use, mine a folder, optionally save the tables, then write a corpus and question and answer pairs to disk. This program compiles against the published package; it needs a folder of text to run.

using Prose;

var prose = new ProseEngine();
prose.Mine(@"C:\corpus\text");                        // parse every *.txt in the folder and mine it
prose.Save(@"prose.wal");                              // persist the mined tables to HoloDb (optional)

foreach (var sentence in prose.Generate(1000))         // new, grammatical sentences
    Console.WriteLine(sentence);

prose.WriteCorpus(@"C:\corpus\synthetic", 100_000);    // write sentences out as .txt files for a language model
prose.WriteQaPairs(@"C:\corpus\qa", 50_000);           // write context + question -> answer pairs as a TSV

• Parse

  Sentence splitting, part-of-speech tagging, phrase chunking and role assignment (subject, verb, objects, indirect object, predicate, relative clauses) with zero training and no external model.

• Mine

  Scans a corpus and builds a lexicon plus selectional tables: which subjects go with which verbs, which verbs take which objects and which adjectives modify which nouns. They are stored in a HoloDb database.

• Generate

  Samples a sentence shape from the mined patterns (simple, transitive, ditransitive, copular, with optional relative clauses), fills each slot with an observed, agreement-correct word, and keeps only sentences that re-parse validly.

• Plausibility scoring

  Optional. A small language model ranks generated candidates so the output reads more sensibly, without weakening the grammatical guarantee.

• Question and answer generation

  Turns generated sentences into context, question and answer triples, including multi-sentence passages with coreference and self-referential two-turn dialogues.

• Corpus and TSV export

  Write generated sentences straight to text files, or question and answer pairs to a prompt and target TSV, ready for a language model's training data.

• A small corpus recombines freely. Below 500 mined sentences generation falls back to the whole part-of-speech pool, so pairings need not have been seen. From 500 sentences on it draws only from pairings the tables contain.
• Grammatical is not the same as meaningful. The output is built from observed pairings and checked for form, and it can still say odd things. The optional scorer ranks candidates, and correct output never depends on it.
• Relative clauses are handled for subject relatives. Object relatives and non-restrictive relatives are not.
• Generation is deterministic for a seed. Parallel generation uses a fixed number of shards, so it is also deterministic across machines, but it draws a different sequence from the sequential path.
• It depends on HoloDb, which holds the mined tables, and on AlgFormer, which supplies the optional scorer. Installing the package brings both in.
• The package ships as a binary with no source.

Every public type is documented with a worked example in the package readme on NuGet. Open the NuGet page.

generated sentences that passed the validity gate on re-parse

1,966 of 1,966

sentences in which every word was observed in the corpus

1,966 of 1,966

sentences that were different from every other

1,966 of 1,966

Proprietary. Every package ships as a compiled library. Evaluation and non-commercial development or testing are permitted; commercial, production or revenue-generating use needs a commercial licence from the owner. Read the licence terms.