About

The Philosophical Transactions of the Royal Society is the longest-running scientific journal in the world, with its first volume appearing in 1665. As part of the Secrets to Patent project, we benchmark local LLMs on the task of extracting structured information from OCR'd article text.

This viewer compares model outputs side by side for a selection of articles. Models evaluated include gemma4, ministral-3, qwen3.5, phi3.5, and others. For each article you can select any two models and inspect their experiment classification and extracted locations against the original article text.

Prompts

The extraction schema evolved during benchmarking. Two prompt versions are present in the data. Structured output was enforced via JSON schema using the Ollama API.

v2 — 3 questions (gemma4, ministral-3):

You are analysing an article from the Philosophical Transactions of the Royal Society (17th–19th century). Answer three questions:

1. is_experiment: Does this article describe one or more experiments or systematic observations? Answer "yes", "no", or "unsure".

2. locations: List every spatial location where an experiment or observation took place. For each, provide:
   - place: the immediate setting at room or building scale (e.g. "private study", "ship's cabin", "kitchen"); null if not mentioned
   - geography: named place at city, region, country, estate, or vessel scale (e.g. "London", "aboard HMS Endeavour"); null if not mentioned
   - detail: specific spatial detail within the place (e.g. "by the south window", "in a dark corner"); null if not mentioned
Do not include apparatus or containers as locations. Return an empty list if no spatial setting is mentioned.

3. participants: List every person named in the article who conducted, observed, or contributed to the experiment or observation. For each, provide:
   - name: the person's name as it appears in the text
   - role: their role if stated (e.g. "experimenter", "observer", "subject", "correspondent", "author"); null if not clear
Return an empty list if no individuals are named.

v1 — 2 questions (llama3.2, phi3.5, qwen3.5): same as v2 but without question 3 (participants). Output schema: is_experiment + locations only.

Repository

Source code and data: github.com/DigitalHistory-Lund/SecToPat-PhilTransModelSelection

Team

Natacha Klein Käfer is the principal investigator of the Secrets to Patent project. The benchmarking pipeline and viewer were assembled by Mathias Johansson using Python, Ollama, and Claude.

Citation

If you use this site or its data in your work, please cite the repository. Machine-readable metadata is in CITATION.cff.

License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). See LICENSE.

Contact

For questions or feedback, contact Mathias Johansson at MathiasJohansson@kultur.lu.se, or open an issue.