A gap map · research compiled 20 November 2026

Seven things nobody has done yet.

These are the genuinely uncharted areas in one research programme — the places where the published record simply stops.

To prove each one is really empty, researchers pulled the closest existing work they could find: 225 published studies, papers and systems, at least 30 per area. Every area below shows what the world has managed so far, exactly where it falls short, and what the first real demonstration would have to look like. Written in plain language throughout — if a term needs explaining, it gets explained.

7
uncharted areas, each checked separately
225
closest published studies reviewed (30+ per area)
4
areas where targeted searching found no prior work at all
0
published demonstrations of any of the seven ideas themselves

How to read the numbers

Every figure on this page carries a small tag saying how solid it is. Nothing here is presented as firmer than its source allows.

[published] read in the original paper
[reported] from a reliable summary of it
[estimated] worked out by the researcher, basis shown
[unknown] could not be verified

One honest caveat up front: the “nobody has done this” verdicts rest on targeted searching — 32 search queries plus reading of primary sources (journals including Nature, Science, PNAS, PLOS and arXiv) — not on an exhaustive review of everything ever published. Treat them as high confidence, not absolute proof. The full list of things that couldn’t be verified is at the bottom of the page.

The map at a glance

All seven areas, the closest anyone has come, and the verdict. Tap any row to jump to its full page.

1
Mycology + neurology — a living brain–fungus junction
Closest: Brainoware (living neurons computing) + Adamatzky’s fungal computers (living mycelium computing). The two halves exist. They have never been connected.
Confirmed absence
2
First-person research — studying your own research process
Closest: self-study of teaching (S-STEP) and meta-research. Both mature fields — but one studies teaching, the other studies science in bulk. Neither studies one researcher’s own process.
Never aimed at research itself
3
The pipeline where the worker keeps the benefit
Closest: MyLifeBits and Luhmann’s Zettelkasten — two famous one-person proofs it can work. Never defined as a system, never measured, no benefit metric exists.
Exists as proofs · never measured
4
A ruler for “how wrong is this placement?” — and the signature question
Closest: RealDocBench layout scores. Benchmarks grade whether the right words were read, never how far a guessed box sits from the true field. And whether AI-copied handwriting on your own form is legally your signature: unasked, anywhere.
No metric · legal question unaddressed
5
Learning from the nudge — adjustment differencing
Closest: corrective-feedback systems and human-in-the-loop annotation. Everywhere, the human’s correction is swallowed as a fresh answer and the difference from the guess is thrown away.
The diff is discarded everywhere
6
Predicting what an unread paper found
Closest: replication prediction markets, 73% accurate — but forecasters always read the paper first and only bet yes/no. Predicting specific findings of a paper you haven’t read has never been tested.
Exact task never tested
7
Messages that carry their own context
Closest: human–AI “grounding” research, a young field still measuring the problem. Messages deliberately built to survive being read cold, out of context: no published work at all.
No published work
AREA 01 · MYCOLOGY + NEUROLOGY Confirmed absence

Brain cells, meet living fungus

A brain–computer interface built from living neurons wired into a living mycelium network. Both halves have been built. The connection between them has not.

In plain English

Scientists have grown tiny clumps of human brain cells in a dish and taught them to compute — one played the game Pong. Completely separately, other scientists have shown that mycelium — the root-like web a mushroom grows from — carries electrical signals and can act like simple computer parts.

Nobody, anywhere in the published record, has connected the two living things to each other. That missing connection is called a junction: a place where a signal can pass from neuron to fungus, or fungus to neuron. Building it would be a world first.

Living neurons compute on electrode grids Living mycelium signals, logic, sensors THE JUNCTION never demonstrated — in any paper, anywhere
Two mature literatures, one missing bridge. Everything on this page is about how close each side has come to the middle.
~800,000
living neurons in DishBrain, which learned Pong in about 5 minutes [reported]
1–21 hrs
duration of a single fungal electrical “spike” — neurons fire in thousandths of a second [reported]
50
spike patterns (“words”) grouped from fungal signals — a claimed fungal “language”, disputed by other researchers [reported]
>2,000
bends survived by a mycelium circuit-board skin (MycelioTronics) [published]

Closest anyone has come

Brainoware (2023) — a human brain organoid (a pea-sized clump of living brain cells) mounted on an electrode grid. It learned by physically rewiring itself, and reached speech recognition and predicting the solutions to nonlinear equations.

The neuron half of the junction, fully working — but its partner is silicon electrodes, not fungus.

Adamatzky’s fungal computing (2018–2022) — living mycelium used as memristors (memory resistors), oscillators and sensors, and wired into AND/OR/XOR logic gates.

The fungus half, fully working — but it talks to electrodes, never to a neuron.

“Living electrodes” (Serruya et al.) — engineered tracts of living neurons that grow into a host brain and form real synaptic connections with it.

Proof that a living-to-living junction can work at all — but both sides are neurons.

Cornell’s fungus-driven robot (2024) — a king oyster mushroom’s electrical signals steered a robot’s movement.

A living fungus inside a working control loop — fungus to machine, not fungus to neuron.

What’s still missing

  • The junction itself. No study couples a living neuronal network to a living mycelium network. Zero.
  • The speed problem. Neurons signal in milliseconds; fungal spikes unfold over minutes to hours. Nothing published bridges that mismatch.
  • The shared-soup problem. Both cell types would need one growth medium that keeps neurons alive without poisoning the fungus, and vice versa. (This is squarely a chemistry problem — media, gels, and what each side secretes.)
  • A crossing mechanism. Would the signal pass electrically, chemically, or some other way? Nobody has proposed a tested answer.
  • Keeping both alive together. Even brain organoids alone are hard to keep alive on hardware — a 2025/26 review lists six separate longevity factors, all unsolved at once.
What being first looks like

Grow the two together in one dish. Stimulate the neuron side and show — repeatably, with controls that rule out the equipment fooling you — that the mycelium’s electrical activity changes in response. Or the reverse. That single experiment, published, is the first neuron–fungus junction in history.

All 32 case studies for this area
Fungal computing & mycelium electronics
  1. Adamatzky (2018), “Towards fungal computer” — proposed fungi as computing devices: information carried as electrical spikes, logic emerging from the mycelium’s shape. Gap: the fungal half, fully specified; no neuron anywhere in the design. Source
  2. Adamatzky et al. (2022), “Fungal electronics” — living mycelium composites acting as memristors, oscillators, and pressure/light/chemical sensors. Gap: fungus as an electronic component — interfacing with electrodes, not neurons. Source
  3. Danninger et al. (2022), MycelioTronics — mycelium skins as biodegradable circuit boards: conductivity 9.75 ± 1.44 × 10⁴ S/cm [published], >2,000 bends [published], mycelium batteries ~3.8 mAh/cm² [published]. Gap: the mycelium here is a dried material, not a living signalling partner. Source
  4. LaRocco et al. (2025), Ohio State fungal memristors — dehydrated shiitake mycelium switching at up to ~5,850 cycles/s at ~90% accuracy [reported]. Gap: dried, passive material; no neural coupling. (Primary paper not located — see caveats.) Source
  5. Adamatzky et al. (2022), fungal “language” — spikes from four species grouped into up to 50 “words”; spikes last 1–21 hours at 0.03–2.1 mV [reported]; the language framing is disputed. Gap: fungal signalling described in isolation; never interfaced with neural spikes. Source
  6. Olsson & Hansson (1995) — first recording of action-potential-like impulses travelling through fungal cords [reported]. Gap: proves fungi are electrically excitable like neurons — no coupling attempted. Source
  7. Oyster mushroom spike trains — spike trains with periods of ~2.6 and ~14 minutes; a recording over 100 days found a roughly weekly rhythm [reported]. Gap: fungal tempo is minutes-to-hours against the neuron’s milliseconds — the mismatch nobody has bridged. Source
  8. Adamatzky et al. (2022), shiitake neuromorphic logic — mycelium spikes treated as 1s and 0s to build AND/OR/XOR gates [reported]. Gap: logic inside mycelium alone; the readout is electronic. Source
  9. Cornell biohybrid robot (2024) — king oyster mushroom electrical signals driving a robot’s locomotion [reported]. Gap: fungus → robot, not fungus → neuron; the closest living-fungus control loop. Source
  10. Fungal skin sensors — mycelium composites as touch and chemical sensors [reported]. Gap: sensing only. Source
  11. Mycorrhizal network imaging robot (2025) — over 500,000 nodes of growing fungal networks tracked; growth behaves like self-regulating travelling waves [reported]. Gap: network behaviour mapped; no neural interface. Source
Slime mould computing — the neighbouring living-computer field
  1. Whiting et al. (2014), slime mould logic gates — OR gate 90% correct, AND 77.8%, NOT 91.7%, full adder 58.8%; up to 30× faster than growth-based gates [published]. Gap: living-cell logic, but a slime mould — and no neurons. Source
  2. Mayne & Adamatzky (2014) — slime mould NOT/NAND gates with light control and electrical input/output; fault-tolerant and resettable. Gap: the living-substrate-to-hardware integration pattern a junction would need — with the wrong organism. Source
  3. Adamatzky (2010), ballistic slime mould gates — Boolean gates from travelling slime-mould fronts, cascaded into a half-adder (simulated). Gap: as above. Source
  4. Nakagaki et al. (2000) — slime mould solves a maze, retracting to the shortest path between two food sources [reported]. Gap: living computation without neurons. Source
  5. Tero et al. (2010) — slime mould grown over a map of Tokyo rebuilt a network resembling the real rail system [reported]. Gap: as above. Source
  6. Atlas of Physarum Computing (2015) — compendium of ~40 slime-mould prototypes: gates, sensors, self-repairing wires, robot navigation [reported]. Gap: a mature living-computing literature with no neuron coupling anywhere in it. Source
Living-neuron computing
  1. Cai et al. (2023), Brainoware — human brain organoid on a high-density electrode grid; unsupervised learning, speech recognition, nonlinear-equation prediction [published]. Gap: the neuron half, fully realised — partner is silicon, not mycelium. Source
  2. Kagan et al. (2022), DishBrain — ~800,000 mouse/human neurons learned Pong within ~5 minutes; hits per rally rose ~0.65 → ~0.85 across 280+ sessions [reported]. Gap: living neurons plus electronics; no fungal partner. Source
  3. Jordan et al. (2024), FinalSpark Neuroplatform — 16 human brain organoids rentable over the internet, researchers in 14 countries, ~$1,000/month [reported]. Gap: a neuron–silicon cloud platform; no mycelium. Source
  4. Cortical Labs CL1 (2025) — “world’s first code-deployable biological computer”: living human neurons on silicon, ~30 W per unit [reported]. Gap: commercial neuron–silicon; no fungal component. Source
  5. Koniku Konikore — engineered living neurons fused to silicon for smell detection; 64-neuron prototype; company claims up to 4,096 smells at once [reported — company claim, unverified]; $8M Airbus contract [reported]. Gap: a living neuron–silicon junction — exactly the interface class needed — but the biological partner is engineered neurons, not fungus. Source
  6. Hartung et al. (2024), organoid-intelligence ethics review — defines the field of Organoid Intelligence and surveys its ethics and governance [published]. Gap: frames neuron computing; fungi absent. Source
  7. The Baltimore Declaration (2023) — founding statement of organoid intelligence from the field’s first workshop [reported]. Gap: as above. Source
  8. Smirnova et al. (2023) — brain organoids are 300–500 µm across; ~40% of their neurons are myelinated (insulated, like real brain wiring) [published]. Gap: background on the neuron side. Source
  9. Borden (2025/26), organoid longevity review — six factors decide how long a brain organoid survives on hardware: life support, fluid flow, blood supply, interface durability, closed-loop balance, adaptive stimulation. Gap: longevity is a shared problem any neuron–fungus device inherits. Source
  10. Springer review (2025), “From brain organoids to organoid intelligence” — a literature search found 89 articles on organoid intelligence and its ethics [published]. Gap: confirms a recognised field — with no fungal branch. Source
Biohybrid & living interfaces
  1. Biohybrid regenerative bioelectronics (2023) — human muscle cells grown on flexible electrode arrays integrated with a rat’s forearm nerve for 4 weeks in freely moving animals [published]. Gap: living cells as the interface layer between nerve and electronics — the closest architectural cousin of a neuron–fungus junction, but the living layer is muscle. Source
  2. Serruya et al., “living electrodes” — engineered axon tracts that synaptically integrate with host neurons, leaving only living tissue in the brain [published]. Gap: a living-to-living junction, proven — with neurons on both sides. Source
  3. Stem-cell biohybrid cochlear implant (2025) — conductive hydrogel carrying neural stem cells: charge-transfer resistance down 97.7%, hearing thresholds improved 18.8–28.8 dB in guinea pigs [published]. Gap: biohybrid electrode–tissue coupling; no fungus. Source
  4. Nature Communications review (2025) — charts the path biomimetic → biohybrid → all-living interfaces; notes silicon is ~180 GPa stiff against the brain’s ~1–30 kPa [published]. Gap: even the review’s “all-living” endpoint is neuron-to-neuron; fungi never appear. Source
  5. Conventional electrodes (Utah arrays, Michigan probes, Neuropixels) — rigid metal/silicon probes; the body’s foreign-body response degrades their signals over time [published]. Gap: the incumbent technology a neuron–fungus interface would have to beat — entirely inorganic. Source
AREA 02 · FIRST-PERSON SELF-REFERENTIAL RESEARCH Never aimed at research itself

Studying your own research, as the research

The researcher treats their own research process — how they pick questions, spot gaps, choose methods — as the thing being studied, and the loop measurably improves itself.

In plain English

There are respected fields where people study themselves. Teachers study their own teaching. Anthropologists write about their own experience inside a culture. And “meta-research” studies science itself — but only in bulk, across thousands of papers at once.

What nobody has published: one researcher turning that lens on their own research process — recording their decisions, dead ends and revisions as data, and showing with numbers that studying the process made the research itself better. The loop that watches itself work, and improves because it watches.

Do the research decisions · searches · dead ends Study the traces teachers do this for teaching Measure the gain meta-research does this in bulk All three steps, on your own research process: never published
Each piece exists somewhere. Nobody has closed the loop on their own research.
~85%
of biomedical research effort estimated wasted through inefficiency — found by meta-research studying science in bulk [published]
1993
year self-study of teaching got its own academic society group (AERA SIG) — first-person method is institutionalised, for teaching
73%
of 295 medical papers in a 1966 study drew conclusions the data didn’t justify [reported]
0
published methods where the studied object is the researcher’s own research process

Closest anyone has come

S-STEP — self-study of teacher education (LaBoskey 2004; handbook tradition since 1993) — a mature, rigorous first-person methodology: self-initiated, improvement-aimed, with handbooks, a journal and systematic reviews.

The most developed first-person method anywhere — aimed at teaching, never at research.

Schön, “The Reflective Practitioner” (1983) — the classic account of reflection-in-action: a practitioner holds a “conversation” with their materials, spiralling through appreciation, action, reappreciation.

The canonical self-observing loop — described for architects and teachers, never instrumented or measured as a research method.

Merriweather (2015), “Changing ‘I’” — an autoethnography where the finding included the method changing the researcher-as-instrument.

The research loop feeding back on the researcher — but the object was mentorship and identity, not the research process.

Meta-research (Ioannidis 2015, 2018) — research studying research: methods, reporting, reproducibility, incentives — including the ~85%-waste estimate.

Research on research at field scale, third-person and aggregate. The first-person version is not part of the programme.

What’s still missing

  • The object. Every first-person tradition studies teaching, culture or identity. Meta-research studies the field. Nobody’s object is their own research process.
  • Measurement. Reflection is treated as a stance, not a loop with numbers: nobody reports “my gap-spotting got 30% faster because I studied my own traces.”
  • The improvement claim. Self-study aims at better teaching; nothing published claims — and tests — that self-study makes the research method itself better across iterations.
What being first looks like

A published study whose data are the researcher’s own process traces — decisions, revisions, dead ends — analysed against stated criteria, showing a measured improvement across iterations (fewer wasted searches, faster gap identification) that’s attributable to the self-study itself.

All 33 case studies for this area
Autoethnography — studying yourself inside a culture
  1. Ellis & Bochner (2000) — set the goal of personal narrative as research: “reveal the hand of the researcher rather than construct authority through absence” [reported]. Gap: self as data about culture, not about the research process. Source
  2. Drabble (2018), Spark 3(1) — autoethnographic dissertation excerpt blending literature review with fictional narrative [published]. Gap: reflexive reporting; reflects on teaching, not on research itself. Source
  3. Merriweather (2015), “Changing ‘I’” — autoethnography of an unusual mentorship; the method measurably changed the researcher-as-instrument [published]. Gap: closest in this cluster — the loop feeding back on the researcher — but the object is identity, not process. Source
  4. Morgan & Merriweather (2009) — mentorship study where the protégé is older than the mentor, using autoethnographic data [reported]. Gap: as above. Source
  5. Anderson (2006), “Analytic Autoethnography” — argued self-study can keep full analytic rigour and generalisation while using the self [reported]. Gap: rigour for self-study, but the object remains culture. Source
  6. Ellis, Adams & Bochner (overview) — autoethnography as both process and product, positioned against distanced social science [reported]. Gap: definitional; no self-improving research loop. Source
  7. Hayano (1979) / Firth (1966) — origins of the term “autoethnography” [reported]. Gap: historical. Source
  8. Heider (1975) — first use of the term, tying personal experience to larger cultural beliefs [reported]. Gap: historical. Source
  9. Chang (2008) — self-narration that “transcends mere narration of self to engage in cultural analysis” [reported]. Gap: as Ellis & Bochner. Source
  10. Boylorn & Orbe (2014) — autoethnographers “research themselves in relation to others” [reported]. Gap: the relational self, not the research-process self. Source
  11. Wall (2006) — autoethnography as room for non-traditional inquiry and expression [reported]. Gap: as above. Source
  12. Pelias (2003) — autoethnography “lets you use yourself to get to culture” [reported]. Gap: culture is the destination, not the research loop. Source
  13. Maréchal (2010) — defined autoethnography as self-observation and reflexive investigation within fieldwork [reported]. Gap: self-observation of fieldwork, not of research method. Source
  14. Adams et al., six features — personal experience used to critique culture; deep self-reflection; a balance of rigour, emotion, creativity [reported]. Gap: reflexivity as a stance, not a measured feedback loop on method. Source
  15. Lionnet (1990) — autoethnography as “a space of resistance between the individual (auto-) and the collective (-ethno-)” [reported]. Gap: theoretical framing. Source
S-STEP — self-study of teaching practice
  1. LaBoskey (2004), self-study methodology — the field’s methodology chapter: self-study is self-initiated, improvement-aimed, interactive, mainly qualitative, validated by trustworthiness [published]. Gap: the most developed first-person tradition — improving teaching, not research method. Source
  2. Vanassche et al., systematic review 1990–2012 — mapped the whole S-STEP literature; found two standing tensions: relevance vs rigour, effectiveness vs understanding [published]. Gap: proof first-person research exists at scale — for teaching. Source
  3. Loughran et al. (2004), International Handbook of Self-Study — the field’s foundational handbook; its society group dates to AERA 1993 [reported]. Gap: institutionalised first-person research — in teacher education only. Source
  4. Kleine et al., “A Call for Self-Study” — argues self-study’s value includes generalisability of process rather than results [published]. Gap: “generalisability of process” is adjacent to the target idea — applied to teaching. Source
  5. Cassandra (2025), dialogic-pedagogy self-study — five months (Sept 2022–Jan 2023) of recorded lessons, transcripts, video-stimulated journal reflections and critical-friend dialogue [published]. Gap: a worked act → record → reflect → revise loop — improving teaching, not research. Source
  6. Samaras (2011) — self-study demands rigorous, systematic inquiry [reported]. Gap: as LaBoskey. Source
  7. Pinnegar & Hamilton (2009) — self-study as documenting and analysing one’s own implementation, then disseminating it [reported]. Gap: as LaBoskey. Source
  8. Cochran-Smith & Lytle (1999) — practitioner inquiry, “inquiry as stance” [reported]. Gap: teachers researching practice; researchers researching research is the unfilled analogue. Source
  9. Loughran & Northfield (1998) — reflective practice as active practitioner inquiry [reported]. Gap: as above. Source
Reflective practice & metacognition
  1. Schön (1983), The Reflective Practitioner — reflection-in-action: the practitioner converses with the materials of the situation, spiralling through appreciation, action and reappreciation [reported]. Gap: the canonical self-observing loop — described, not instrumented or measured as research method. Source
  2. Tadesse, Schön/Brookfield with EFL student teachers — practitioner inquiry with 10 pre-service teachers; reflective journaling as method and strategy [published]. Gap: reflection-on-action studied in others, not in the researcher’s own research. Source
  3. Brookfield (1986/1995), critical reflection — critical lenses for examining one’s own practice [reported]. Gap: as Schön. Source
  4. Ayazova (2026) — a four-stage model (orientation, analytical, constructive, autonomous) that treats reflective ability as a measurable capacity [published]. Gap: reflection made measurable — in students, not in the research process. Source
  5. Visible Thinking + contemplative learning (2026) — routines like See–Think–Wonder externalise a learner’s cognition so it can be examined [published]. Gap: externalising your own thinking as an artefact — adjacent technique, different object. Source
  6. Buchanan et al. (2022) — a three-step pedagogy model starting with metacognition; the teacher educator does “research and practice simultaneously” [published]. Gap: simultaneity asserted, not operationalised as a method. Source
Meta-research — science studying science, in bulk
  1. Ioannidis (2018), “Meta-research: why research on research matters” — defines the field: study of research methods, reporting, reproducibility, evaluation and incentives [published]. Gap: research studying research — at field level, third-person, aggregate. The first-person version is not part of the programme. Source
  2. Ioannidis et al. (2015) — mapped meta-research; in biomedicine ~85% of invested effort and resources are wasted through inefficiencies [published]. Gap: as above. Source
  3. Metascience overview — a 1966 meta-research paper found ~73% of 295 papers across ten medical journals drew conclusions without valid justification [reported]. Gap: aggregate self-examination of science; no first-person loop. Source
AREA 03 · THE DATA-WORKER-BENEFICIARY PIPELINE Exists as proofs · never measured

The worker does the work — and keeps the benefit

A capture-everything workflow, owned by the person doing the work, where everything they feed in makes their own future work faster — instead of enriching a platform.

In plain English

Most digital pipelines run one way: you do the work, a platform keeps the data, the memory and the cut. Gig drivers watched their share of each fare fall from about 80% to as low as 55–65% in under a decade. Data labellers generate the value AI companies sell, and capture almost none of it.

The flipped design: a pipeline you own end to end — capture everything, sort it, enrich it, and get it back as usable context exactly when you need it — so the person doing the work is the person the system makes faster. Two famous one-person projects prove a human can run such a pipeline for decades. But nobody has ever defined it as a design, tested it properly, or even built a way to measure “benefit to the worker.”

CAPTURE TRIAGE ENRICH LEARN REVIEW TRANSFER the benefit flows back to the worker — that return loop is the untested part pipeline shape from a self-documented personal learning system (Handzel)
Every stage of this loop has evidence behind it. The loop as a whole — with the worker as beneficiary — has never been put on trial.
~90,000
paper slips in Luhmann’s Zettelkasten, with ~30,000 cross-references — feeding ~600 publications [reported]
160 GB
of one man’s life captured by MyLifeBits, including 100,000 photos; a whole lifetime estimated at ~1 TB [reported]
80% → 55–65%
Uber drivers’ share of each fare, 2014 → 2022 — the extraction this design is defined against [reported]
0
controlled studies of the flagship method (Zettelkasten) as a whole system — stated outright by a 2026 review

Closest anyone has come

MyLifeBits (Microsoft Research, 2001–2007) — Gordon Bell captured everything: documents, photos, calls, web pages, wearable-camera images, into a searchable personal database. Its creators said it plainly in 2008: “built to be entirely personal, to aid the individual.”

The purest worker-beneficiary pipeline ever built — and a single-subject experiment, never generalised into a design principle.

Luhmann’s Zettelkasten (1951–1996) — a sociologist’s lifelong slip-box: ~90,000 linked notes that “talked back,” supporting ~600 publications over 45 years.

The canonical self-benefiting pipeline — analogue, one person, and its benefit was never measured against any alternative.

Learning science validates the stages — writing ideas in your own words beats re-reading (generation effect, d ≈ 0.40 across 86 studies); testing yourself beats re-studying (retrieval practice, g ≈ 0.50–0.61) [reported].

Strong evidence for the pipeline’s individual stages — nothing about the whole system, or who benefits from it.

Platform cooperativism (Scholz 2014; Up & Go) — workers owning the platform itself; a New York cleaning co-op platform run by ~40 workers across three cooperatives.

The ownership principle, proven for labour marketplaces — never applied to a personal knowledge pipeline.

What’s still missing

  • A definition. No published work frames “the worker is the beneficiary” as a design object with parts, rules and failure modes.
  • A benefit metric. There is no standard measure for what the worker gains — time-to-recall, reuse rate, value captured. Without one, the idea can’t be tested.
  • A controlled test. The flagship method’s own reviewers state it flatly: no controlled study of the Zettelkasten as a whole exists. Same for every capture-to-context system since.
  • The comparison. Nobody has run the self-hosted pipeline head-to-head against the extractive platform baseline it’s defined against.
What being first looks like

A controlled comparison: a self-hosted capture → context → recall pipeline against a platform baseline, with the worker’s benefit — time-to-recall, reuse, value captured — as the headline outcome. Whoever runs that study defines the field.

All 33 case studies for this area
The Memex lineage — personal knowledge systems
  1. Bush (1945), “As We May Think” — the memex: a personal device storing all your books, records and communications, “an enlarged intimate supplement to his memory,” with trails of association [reported]. Gap: the founding vision is explicitly individual-benefiting — but a vision, not a measured pipeline. Source
  2. “Still Building the Memex,” CACM — defines the Personal Knowledge Base and surveys personal systems with a taxonomy [published]. Gap: who gains from the pipeline is never the evaluation criterion. Source
  3. Bell & Gemmell, MyLifeBits (2001–2007) — total lifelogging into a searchable personal database; ~160 GB including 100,000 photos [reported]. Gap: one person, own data, own benefit — never generalised or framed as a pipeline design principle. Source
  4. SenseCam — wearable camera snapping photos automatically on light/heat/movement triggers, feeding MyLifeBits [reported]. Gap: capture-side technology; beneficiary framing absent. Source
  5. Bell & Gemmell (2009), Total Recall — the book of the MyLifeBits project; Bell called it “an aid to bio-memory for life and afterlife” [reported]. Gap: as MyLifeBits. Source
  6. Luhmann’s Zettelkasten (1951–1996) — ~90,000 slips, ~30,000 cross-references, a keyword register of ~3,200 entries; ~600 publications including 40+ books [reported]. Gap: the canonical self-benefiting pipeline — analogue, n=1, benefit never measured. Source
  7. Schmidt (2018), the Luhmann archive — near-total cross-referencing in the first box; weak spot: unreferenced cards get lost; tending the file took a great deal of Luhmann’s time [reported]. Gap: the only close description of a lifelong self-benefiting pipeline; cost/benefit never quantified. Source
  8. Ahrens (2017/2022), How to Take Smart Notes — fleeting → literature → permanent notes, one idea per note [reported]. Gap: practitioner method, no controlled evaluation. Source
  9. Forte (2022/2023), Building a Second Brain / PARA — organise by project and actionability, with progressive summarisation [reported]. Gap: the user benefits — but no published measurement of the benefit. Source
  10. Pauk (1962), Cornell notes — page layout with cue column and summary, built for recitation [reported]. Gap: a technique, not a pipeline. Source
Learning science — evidence for the pipeline’s stages
  1. Bertsch et al. (2007), generation effect — meta-analysis of 86 studies: generating material yourself beats reading it, d ≈ 0.40 [reported]. Gap: validates “write in your own words” as a stage; says nothing about who benefits from the system. Source
  2. McCurdy et al. (2020) — 126 papers, 310 experiments: generated material wins on every test type; recognition 76.7% vs 62.2% for read-only [reported]. Gap: as above. Source
  3. Roediger & Karpicke (2006), testing effect — tested students beat re-readers after two days and a week; re-reading only won at five minutes [reported]. Gap: validates retrieval stages; not a pipeline study. Source
  4. Retrieval-practice effect sizes — testing g = 0.50–0.61, “2–3× more effective than highlighting or re-reading”; pretesting d = 0.35 [reported]. Gap: component evidence only. Source
  5. Whittaker et al. (2011), email refinding — 345 users, over 85,000 refinding actions; elaborate folders were relied on but costly to prepare [reported]. Gap: measures the cost side of personal organisation; no beneficiary framing. Source
  6. “No controlled study of the Zettelkasten as a whole” (2026 review) — states outright that no controlled study found has tested the method as a whole against other ways of studying [reported]. Gap: direct evidence the flagship self-benefiting pipeline has never been evaluated as a system. Source
  7. Mannheim (2022), note-taking claim — advanced note systems linked to 23% more novel relationships between ideas [reported — weak source, treat as unknown quality]. Gap: included as an example of how thin the evidence base is; the underlying study could not be located. Source
  8. JARHE (2024), thesis-time claim — digital Zettelkasten linked to 31% less thesis-chapter writing time [reported — weak source, treat as unknown quality]. Gap: as above. Source
  9. AERA (2023), doctoral-delay claim — 68% of doctoral candidates said to delay or abandon theses over trouble structuring and linking ideas [reported — weak source, treat as unknown quality]. Gap: as above. Source
Today’s tools — capture-to-context loops you can buy
  1. Recall app — one-click capture of articles, video, podcasts and PDFs; automatic summaries, knowledge graph, spaced-repetition quizzes; “never trains AI on your content”; $10/month [reported]. Gap: a commercial capture-to-context loop whose user benefits — cloud-backed, subscription-priced, unevaluated as a pipeline. Source
  2. NotebookLM (Google) — question-and-answer over your own source set, with inline citations [reported]. Gap: user-benefiting retrieval, but hosted on Google’s cloud — part of the pipeline’s value accrues to the platform. Source
  3. Obsidian — local-first notes with plugins; free, Sync $4/month [reported]. Gap: the closest commercial tool to a self-hosted worker-benefiting pipeline — still a product, not a studied system. Source
  4. Mem / Reflect / Readwise — cloud AI-organised notes ($9/mo), encrypted synced notes ($10/mo), reading and highlights ($9.99/mo) [reported]. Gap: the user benefits, but rent is paid to the platform. Source
  5. Handzel personal learning system — a fully specified pipeline: CAPTURE → TRIAGE → ENRICH → LEARN → REVIEW → TRANSFER, with an AI triage threshold of ≥ 0.7 [reported]. Gap: a complete worker-benefiting pipeline — documented in a personal blog post, n=1, no outcome measurement. Source
  6. WUSTL Learning Center Zettelkasten guide — university guidance on fleeting/permanent/literature notes and digital tools [reported]. Gap: instructional, not evaluative. Source
The extractive contrast — what this design is defined against
  1. Scholz (2014), Platform Cooperativism — the internet’s infrastructure organised under cooperative ownership instead of venture-backed corporations [reported]. Gap: the ownership-level version of the worker-beneficiary principle — about platforms, not personal data pipelines. Source
  2. Up & Go (New York) — three cooperative cleaning companies, ~40 workers, running their workflows on a co-op-owned platform [reported]. Gap: workers own the pipeline — the closest real-world instance of the principle — but a labour marketplace, not a knowledge pipeline. Source
  3. Platform Co-op Development Kit — $1M Google.org grant to help co-ops go digital; ~240 platform co-op projects estimated at the time [reported]. Gap: infrastructure for worker-benefiting platforms; no knowledge-work pipeline. Source
  4. Extractive take-rates — Uber takes 25–30% of each fare; Airbnb 3% host plus up to 14.2% guest; TaskRabbit 15%; Uber driver take-home fell from ~80% of fare (2014) to 55–65% (2022) [reported]. Gap: quantifies the extraction the worker-beneficiary pipeline is defined against. Source
  5. “Own This!” (Public Seminar, 2018) — platform-capitalism critique: users keep little control over their data and no say over platforms; risk is pushed onto workers [reported]. Gap: a diagnosis of the extractive model; no pipeline design. Source
  6. Platform cooperative (overview) — cooperatively owned two-sided markets; categories include “data cooperatives” [reported]. Gap: “data cooperatives” is the nearest named relative of a worker-owned data pipeline — undeveloped in the source. Source
  7. Remotasks / Scale AI worker conditions — repeated investigative reporting (2023–2025) on crowdworker pay and conditions on the annotation platform [reported]. Gap: the extreme case — annotation workers generate the data value and capture almost none of it; the exact inverse of the target pipeline. Source
  8. MyLifeBits privacy stance (2008) — “We’re not life loggers because we’re not publicly disclosing… This was built to be entirely personal, to aid the individual” [reported]. Gap: the beneficiary principle stated explicitly, in 2008 — and never developed into a research programme. Source
AREA 04 · PLACEMENT-ERROR METRICS + SYNTHESISED-HANDWRITING LAW No metric · legal question unaddressed

The ruler that doesn’t exist

Software guesses where fields sit on real paper. There is no standard way to score how wrong the guess is — and a legal question about AI-copied handwriting that nobody, anywhere, has answered.

In plain English — part one: the missing ruler

When software reads a form, every benchmark grades the same thing: did it read the right words? Almost nothing grades position — how far the box it drew sits from where the field actually is on the paper. A guess can be 2 millimetres off (fine) or 2 centimetres off (the value lands in the wrong box entirely) and today’s scores can’t tell the difference. There is no agreed ruler for placement error. Building one — a metric plus a benchmark of guessed-vs-true field positions on real forms — would be a first.

In plain English — part two: is it your signature?

AI has been able to copy a person’s handwriting style since 2013, well enough that samples “cannot always be told from real ones.” Now the unasked question: if you use it to fill in your own form, in your own handwriting, with your own permission — is that legally your signature? E-signature law (the ESIGN Act) judges by intent: a mark you adopt intending to sign can count. Forgery law only punishes deceptive use. Honest use of your own synthesised hand falls exactly between the two — and no case, paper or authority found addresses it. Even the companies selling AI signature generators disclaim that their output is “not automatically legally valid.”

the paper form the field, where it really is the system’s guess how wrong is this? no standard score exists
Every benchmark can tell you whether the words were read. None can tell you how far off the box was.
0.847 vs 0.320
best vs worst layout score (Adjusted F1) across 18 document systems on RealDocBench — position scored as box overlap, not field error [reported]
0.8420
FUNSD score (F1) of LayoutLMv2 — the field’s standard benchmark grades label correctness, not distance from the true position [published]
2013
year handwriting synthesis was demonstrated (Graves) — the legal question it raises is still unaddressed over a decade later
0
published sources found addressing the legal validity of a person’s own synthesised handwriting on their own form

Closest anyone has come

RealDocBench (2026) — 1,500 human-verified pages of real regulated documents; layout scored with a smart matcher that recovers from split and merged boxes; 18 systems ranked.

The nearest thing to a placement metric — but it scores a parser’s layout detection, not the error of an inferred field position against the paper original.

FUNSD / SROIE / CORD benchmarks — the field standards for forms and receipts, scored by entity F1 (did the right words get the right label?).

Labels, not positions. One dataset paper even notes flat F1 can’t score structure — the gap’s own complaint, without a fix.

Signature-verification research — decades of work detecting forged signatures; synthetic signatures studied as attacks; skilled-forgery detection improved 17.76–28.00% in the best recent work [published].

Verification is not validity: it can flag a fake, but it never asks when a synthesised signature legitimately counts.

Robosigning / font signatures — computer-font signatures are authorised for most court filings; the accepted issue is whether a human reviewed the document, not the mark itself [reported].

The nearest precedent — a non-handwritten mark accepted — but it concerns fonts, not synthesised handwriting, and rests on a Q&A answer, not scholarship.

What’s still missing

  • A geometric error metric — mean displacement, overlap-weighted field error, correction cost. The field’s own 2025 survey confirms the metric palette is F1 / exact-match / accuracy / string-similarity only.
  • A benchmark of inferred-vs-true positions on real paper forms — guessed box against the field’s actual location.
  • Any legal treatment at all of self-synthesised handwriting: ESIGN’s intent test points one way, forgery law covers only deception, and the honest middle case belongs to neither.
  • A test case. No court, regulator or law review found has taken the question up.
What being first looks like

Two firsts, either one publishable: (a) a placement-error metric — say, average normalised displacement plus overlap-weighted field error — released with a benchmark of inferred-vs-true field positions on real forms; (b) the first rigorous legal analysis (or test case) of self-synthesised handwriting under ESIGN/UETA and their international equivalents.

All 34 case studies for this area
Benchmarks & metrics — placement-adjacent
  1. RealDocBench (2026) — two-track benchmark on real regulated documents: 1,356 field-level questions over 581 documents; layout track of 1,500 human-verified pages with a Hungarian matcher and split/merge recovery; 18 systems [published]. Gap: the closest thing to a placement-error metric — it scores layout detection, not the error of inferred field positions against a paper original. Source
  2. RealDoc-Bench leaderboard (Extend) — layout Adjusted F1: Extend 0.847, Reducto 0.759, AWS Textract 0.709, Azure 0.687, DotsOCR 0.320; per-field Q&A accuracy up to 95.7% [reported]. Gap: vendor-run; placement measured as block overlap, not field-position error. Source
  3. SCORE-Bench (Unstructured) — 224 expert-annotated real documents, including 54 forms and 33 with mixed handwriting/print; distinguishes hallucinations from omissions [reported]. Gap: evaluates parsing, not placement error of inferred fields. Source
  4. LlamaIndex parser benchmark (2026) — Google Document AI 88/100 (OCR 97.2%), AWS Textract 85, Adobe 82, Azure 80; simple-form OCR up to 98.1% [reported]. Gap: aggregate scores; no geometric placement-error metric. Source
  5. ParseBench — standardised document-level evaluation for parsing systems [reported]. Gap: document-level, not placement-level. Source
  6. LayoutLMv2 (Xu et al. 2021) — FUNSD F1 0.8420, SROIE 0.9781, DocVQA 0.8672, Kleister-NDA 0.8520, RVL-CDIP 0.9564 [published]. Gap: entity-level F1 measures label correctness — not how far a predicted box sits from the true field position. Source
  7. LayoutLM (Xu et al. 2020) — the baseline LayoutLMv2 improved on (FUNSD 0.7895) [published]. Gap: as LayoutLMv2. Source
  8. Donut (Kim et al. 2022) — OCR-free document understanding; FUNSD 87.6, SROIE 97.0 [reported]. Gap: generates answers from pixels; no placement error measured. Source
  9. DocLLM — FUNSD 91.2, SROIE 98.3 [reported]. Gap: as LayoutLMv2. Source
  10. UDOP — unified document vision model, among 20+ architectures surveyed [reported]. Gap: as LayoutLMv2. Source
  11. LayoutLMv3 — the “Document AI” successor line, OCR-free or layout-aware [reported]. Gap: as LayoutLMv2. Source
  12. Pix2Struct — screenshot-to-structure model [reported]. Gap: as LayoutLMv2. Source
  13. FUNSD dataset (Jaume et al. 2019) — 199 scanned forms with word-level labels; semantic entity labelling [published]. Gap: the standard form benchmark — scored by entity F1, not positional error. Source
  14. CORD dataset — receipt understanding with nested groups and tables [published]. Gap: its own discussion notes flat field-F1 can’t score hierarchy — the gap’s complaint, without a fix. Source
  15. SROIE dataset — receipt extraction; LayoutLMv2 scored 0.9781 excluding OCR mismatch, 0.9661 including it [published]. Gap: as FUNSD. Source
  16. Kleister-NDA — key-information extraction from NDAs; F1 0.8520 [published]. Gap: as FUNSD. Source
  17. RVL-CDIP — document classification at 95.64% accuracy [published]. Gap: classification, not placement. Source
  18. DocVQA (Mathew et al. 2021) — document question-answering scored by ANLS (string similarity), 0.8672 with LayoutLMv2 [published]. Gap: answer-text similarity, not position. Source
  19. InfographicVQA — multi-span and computed answers over infographics [reported]. Gap: as DocVQA. Source
  20. Structured Document Understanding survey (2025) — 20+ architectures across 10+ datasets; the standard metrics are F1, exact match, accuracy and ANLS [published]. Gap: the survey itself confirms the field’s metric palette contains no placement-error metric. Source
  21. LayoutLMv2 coordinate scheme — positions normalised to integers in [0,1000] relative to page size, fed in as model features [published]. Gap: positions are model inputs — error in predicted positions is never the evaluation target. Source
  22. GAIDA stage-effort metrics (2026) — measures human annotation effort per stage rather than accuracy alone [published]. Gap: effort metrics, not placement-error metrics — but the only source treating the human correction as a measurable quantity. Source
Handwriting synthesis & signature verification
  1. Graves (2013/14), handwriting synthesis — a neural network synthesises handwriting in a real writer’s style from text; samples “cannot always be told from real ones” (the author’s informal claim, based on informal viewing) [reported]. Gap: synthesising a person’s handwriting was demonstrated a decade ago; the legal-validity question has never been addressed. Source
  2. IAM-OnDB database — online handwriting from 221 writers: pen position and state sequences [reported]. Gap: the data that makes style imitation possible. Source
  3. “Computers can now accurately forge human handwriting” (ETeknix, 2015) — press coverage of Graves’ demo converting up to 100 characters in 5 handwriting styles; forgery potential noted [reported]. Gap: the forgery implication was flagged in the press in 2015; no legal scholarship found addressing it. Source
  4. Bromley et al. (1988), signature verification — early neural network on signature images: ~2% of genuine signatures rejected, 2–4% of forgeries accepted [published]. Gap: verification, not validity — it detects forgery; it doesn’t say when a synthesised signature counts. Source
  5. Tolosana et al. (2022), biometric signature verification — recurrent networks over 400 users; 17.76–28.00% relative improvement in detecting skilled forgeries [published]. Gap: as Bromley. Source
  6. Presentation attacks in signature biometrics (2023) — a taxonomy of attacks on signature verification, with synthetic forgeries as a research line [published]. Gap: the field studies synthetic signatures purely as attacks — the closest framing to the legal question, and purely technical. Source
The legal side — the gap itself
  1. US ESIGN Act (§§101/106) — an e-signature is a symbol or process “executed or adopted by a person with the intent to sign”; a signature can’t be denied effect solely for being electronic [reported]. Gap: intent-based validity — a synthesised signature adopted with intent might count, but no case law or scholarship found tests it on a filled form. Source
  2. UETA — the state-level companion to ESIGN; generator-created signatures covered subject to state rules [reported]. Gap: as ESIGN; vendor guidance, not legal analysis. Source
  3. New York forgery statutes (PL 170.05 / 170.10) — forgery turns on intent to defraud, deceive or injure; the serious grade covers contracts, deeds and wills [reported]. Gap: forgery law covers deceptive use — not a person synthesising their own handwriting on their own form. Source
  4. Hong Kong ETO / Singapore ETA / Malaysia DSA 1997 — reliability and signer-identification requirements for electronic signatures [reported]. Gap: identity and authentication requirements; synthesised handwriting as a signing method is not discussed. Source
  5. DaVinci AI signature generator (vendor FAQ) — “An AI-generated image of a signature is not automatically a legally valid electronic signature. Legal validity depends on the signing process and jurisdiction” [reported]. Gap: the vendors themselves disclaim; no authority found settles it. Source
  6. Robosigning / font signatures (Avvo, 2013) — computer-generated font signatures authorised for court filings in most courts; the issue is review, not the mark [reported]. Gap: the nearest precedent — but fonts, not synthesised handwriting, and a Q&A answer rather than scholarship. Source
AREA 05 · HUMAN-ANNOTATION ADJUSTMENT DIFFERENCING The diff is discarded everywhere

Learning from the nudge

A system guesses where something goes. A human drags it to the right place. The difference between the guess and the fix is a pattern — and every existing system throws it away.

In plain English

Picture software that fills in a paper form — or places parts on a board. It makes its best guess at where each item belongs. You look, and you nudge: a bit left, a touch down, slightly rotated. That nudge is not random. If the system always guesses a few millimetres too low on this kind of page, the nudges will all point the same way — and a smart system could learn the pattern and stop making that mistake.

Learning from corrections is a huge, mature field. But in every system found, the correction is swallowed as a brand-new answer, or counted as “time saved.” The diff — the actual vector between guess and fix — is never kept, never modelled, never learned from. The most informative part of the interaction is the part everyone deletes.

a form, a page, a board — the workspace the system’s best guess where you moved it THE DIFF — direction + distance + rotation today: thrown away · the idea: learn from it
The correction is kept. The correction’s shape — the part that teaches — is not.
+9 pts
accuracy gain (mIoU) when human corrections were fed back into image segmentation — 12–15% relative, with 3–4× less effort [published]
−80.9%
region-drawing and transcription time when AI pre-filled a document and humans corrected it (GAIDA) [published]
>85%
of 13,595 form fields auto-accepted without human touch in a two-AI cross-check system; final word error rate 0.003 [published]
0
systems that learn a placement model from the difference between guess and adjustment

Closest anyone has come

Culotta et al. (2006), corrective feedback — the conceptual twin, twenty years old: a system that asks users to correct its extractions and keeps learning from the corrections, steering people to the mistakes that teach the most.

Corrections as a training signal — but to field values, not positions. No diff of an adjustment is ever modelled.

Human-in-the-loop segmentation (Shaeri et al. 2025) — corrections treated as interventions and propagated to similar images; +9 accuracy points, 3–4× less annotation effort [published].

The corrections here are genuinely spatial — the closest published analogue. But the diff is used to spread edits around, not to learn why placements were wrong.

GAIDA (2026) & DocSpiral (2025) — humans correcting AI pre-filled documents; work cut by up to 80.9% and at least 41% respectively [published].

Real document workflows with real placement corrections — measured purely as effort saved. The corrections teach the system nothing.

What’s still missing

  • The diff as data. Nobody logs (guess, adjusted) pairs and studies the gap between them as the object of learning.
  • A placement-error model. Systematic offsets, scale errors and drift — the repeatable structure inside people’s nudges — are discarded instead of fitted.
  • The head-to-head. No study tests “learn from the diff” against the standard approach of retraining on corrected answers.
  • The infrastructure gap. The big annotation platforms (Label Studio, Labelbox, Scale) all collect corrections at industrial scale — none of them diff an adjustment.
What being first looks like

A system that records every (guess placement, human-adjusted placement) pair, computes the diff, fits a model of the systematic error — and then proves, on documents it has never seen, that learning from the diff beats retraining on the corrected answers alone.

All 31 case studies for this area
Correction-based learning — the closest family
  1. Culotta, Kristjansson, McCallum & Viola (2006) — corrective feedback plus persistent learning for information extraction: solicit corrections, guide users to the most informative mistakes, fold corrections into inference [published]. Gap: the conceptual twin — but corrections are to values and labels, not placements, and no adjustment-diff is modelled. Source
  2. Nallapati, Surdeanu & Manning, CorrActive Learning — learns from noisy labels by showing users only the examples most likely mislabelled, then learning from their corrections [published]. Gap: corrections to labels, not placements; no spatial diff. Source
  3. Shaeri, Woo & Mohammadpour (2025), HITL segmentation — human corrections as interventional signals, propagated across similar images: up to +9 mIoU points (12–15% relative), 3–4× less annotation effort [published]. Gap: corrections are spatial — the closest analogue — but the diff propagates edits; it doesn’t train a placement model. Images, not documents. Source
  4. Mesabah & Elhosary (2026), GAIDA — AI pre-annotation plus human validation for documents: OCR help cut region/transcription time up to 80.9%; reviewing AI labels cut labelling time up to 73.7%; pre-annotation F1 78.86% (FUNSD) / 86.98% (WildReceipt) [published]. Gap: humans do correct placements — but corrections are measured as effort, never diffed or learned from. Source
  5. DocSpiral (2025) — “human-in-the-spiral” annotation: models need progressively less intervention; ≥41% annotation-time reduction, up to 75% [published]. Gap: the loop learns from annotations as labels — not from the diff between guess and correction. Source
  6. Yi (2026), Double Triangle Annotation — two independent AIs annotate; agreement is auto-accepted, disagreement goes to a human jury; word error rate 0.003; >85% of 13,595 fields auto-accepted [published]. Gap: humans adjudicate rather than adjust placements; nothing is learned from an adjustment. Source
  7. Klie et al. (2018), INCEpTION — annotation platform whose recommenders learn from user annotations and rejections; active-learning mode [published]. Gap: the correction signal is accept/reject — not a spatial diff. Source
  8. WebAnno (Yimam et al. 2014) — automation mode learns from user annotations and offers suggestions [reported]. Gap: as INCEpTION. Source
  9. GATE Teamware (Bontcheva et al. 2013) — pre-annotations corrected by annotators in a managed workflow [reported]. Gap: corrections consumed as labels. Source
  10. COCO Annotator (Brooks 2019) — annotation tool without model pre-annotation [reported]. Gap: baseline tool; no learning loop at all. Source
RLHF & preference learning — corrections as a score, not a position
  1. Ouyang et al. (2022), InstructGPT — human rankings train a reward model that steers the AI; the human annotators themselves agreed only ~70% of the time [reported]. Gap: human judgement as training signal — over text preferences, not placement adjustments. Source
  2. Ziegler et al. (2019) — introduced the RLHF pipeline for language models [reported]. Gap: as InstructGPT. Source
  3. Liu (2025), Explainable RLHF — finds which training data caused bad responses and unlearns it, improving outputs [published]. Gap: correction of causes, not of placements. Source
  4. DPO / Constitutional AI / RLAIF — the successor family to RLHF: direct preference optimisation and AI-generated feedback [reported]. Gap: as InstructGPT. Source
  5. “Human-in-the-loop NLP: how to design effective feedback” (2026) — feedback cycles only work when the feedback itself is designed, not merely collected [reported]. Gap: general doctrine; no placement-diff mechanism. Source
Annotation platforms — the correction infrastructure
  1. Label Studio (HumanSignal) — open-source labelling with ML-assisted pre-labelling and active-learning triggers; v1.23.2 (Nov 2026) [reported]. Gap: the infrastructure for collecting placement corrections exists — the platform does not diff adjustments or learn placement models from them. Source
  2. Prodigy (Explosion AI) — scriptable annotation with continuous active learning; $490 one-time licence [reported]. Gap: active learning picks examples; corrections are labels, not diffs. Source
  3. Argilla — open-source annotation for NLP/LLM fine-tuning and RLHF data [reported]. Gap: as Prodigy. Source
  4. Labelbox Model-Assisted Labelling — imports model predictions as pre-labels for human correction [reported]. Gap: pre-label → correct is exactly the interaction — but the correction delta is never modelled as a learnable placement signal. Source
  5. Scale AI Data Engine / Remotasks — enterprise human-feedback at scale with a managed workforce [reported]. Gap: industrial correction collection; learning from it is the customer’s problem. Source
  6. V7 Darwin — vision annotation with active learning [reported]. Gap: as Prodigy. Source
  7. Humanloop — LLM evaluation and RLHF pipelines; from $450/month [reported]. Gap: as Prodigy. Source
  8. CVAT — open-source, self-hosted video annotation [reported]. Gap: geometric annotation (boxes, masks) with no learning from correction diffs. Source
  9. Ango Hub — model-assisted pre-labelling, QA and fine-tuning workflows [reported]. Gap: as Labelbox. Source
  10. Keylabs.ai — enterprise annotation platform, human-verified ground truth [reported]. Gap: as Labelbox. Source
  11. Surge AI — fast human-in-the-loop for RLHF and LLM evaluation [reported]. Gap: as Scale AI. Source
The evidence — corrections are valuable, and permanent
  1. LLM-assisted labelling & active learning (review) — reviewing a pre-existing label is 2–5× faster than labelling from scratch, even with errors in it; active learning cuts annotation costs 40–70% [reported]. Gap: quantifies the value of correct-don’t-create — but measures effort saved, not what the diff teaches. Source
  2. Pre-labelling throughput claims (2026) — “10× faster” claims hold up only on clean, high-contrast image data [reported]. Gap: the correction burden is permanent — which is exactly why learning from the diff matters — but no source frames it that way. Source
  3. AraReceipt dataset — 100 Arabic receipt images, 25 classes, produced through human correction [published]. Gap: a dataset built by correction — the corrections themselves were never published as diffs. Source
  4. FUNSD / WildReceipt benchmarks — the standard form/receipt benchmarks used to score pre-annotation quality (78.86% / 86.98% Micro F1) [published]. Gap: benchmarks score the guess, not the correction diff. Source
  5. “Human review doesn’t go away, it changes shape” — the annotation industry’s own caveat that AI pre-labelling shifts human work rather than removing it [reported]. Gap: the field knows corrections are the durable human contribution; nobody has built the science of learning from them as diffs. Source
AREA 06 · PREDICTING SPECIFIC UNREAD FINDINGS Exact task never tested

Guessing what a paper found — before reading it

Forecasters can bet on whether a known study will hold up. Nobody has tested whether a person can predict the specific findings of a paper they have never read.

In plain English

There’s a well-studied party trick in science: show experts a published study and let them bet on whether it will replicate — whether doing the experiment again gives the same result. Prediction markets get that right about 73% of the time. Impressive — but notice the two cushions: the forecasters read the paper first, and the bet is only yes or no.

The untested version removes both cushions. You get a paper’s title and field — nothing else — and you must predict its specific findings: which hypothesis won, which direction the effect went, roughly how big it was. No study, anywhere, has run that test. And the warning signs say it would be humbling: in one study, cancer experts forecast a 75% chance that certain results would replicate — and zero out of five did.

TODAY’S TESTED TASK forecaster reads the full paper predicts one yes/no question: “will it replicate?” — 73% accurate THE UNTESTED TASK forecaster sees title + field only predicts the actual findings: which idea won · direction · size two cushions removed at once — neither removal has ever been tested
The known results are real. They just aren’t results about this.
73%
accuracy of prediction markets across 103 published findings — binary replicate/fail, papers read first [published]
84%
accuracy of repliCATS, the best structured expert method (AUC 0.94), over 3,000 claims [published]
75% → 0/5
cancer experts’ forecast chance of replication vs how many of the first studies actually replicated [published]
~30%
how much better the top 60 “superforecasters” were than career intelligence analysts — on questions they could fully read [reported]

Closest anyone has come

Replication prediction markets (Dreber 2015; Gordon 2021) — pooled across four projects and 103 findings, markets called replicate/fail at 73% accuracy; market prices tracked outcomes with r = 0.581 [published].

The flagship result of the neighbouring field — with both cushions firmly in place.

repliCATS (Fraser et al. 2023) — structured expert elicitation (a disciplined Delphi-style process) scoring claims at 84% accuracy, AUC 0.94; used on 3,000 claims in 18 months.

The elicitation machinery an unread-findings test would need — built for known claims.

Superforecasters (Tetlock; Good Judgment Project) — the top 2% of forecasters beat intelligence analysts by ~30%; an hour of training improved average forecasters ~10% [reported].

Forecasting specified, fully-stated questions. In the target task, the forecaster doesn’t even see the question the paper asked — only its title.

Benjamin, Mandel & Kimmelman (2017) — the cautionary tale: experts predicted 75% replication probability for preclinical cancer findings; none of the first five replicated on either criterion; more topic expertise made forecasts worse.

Suggests unread-findings prediction would be poor — but that’s an inference, not a measurement. Nobody has measured it.

What’s still missing

  • The unread condition. Every forecasting study found hands the forecaster the paper, the claim, or at minimum the full question.
  • Specific findings. Every target is binary (replicates / doesn’t) or a probability. Nobody scores “which hypothesis won, which direction, what size band.”
  • A dataset. Binary replication forecasting has a public dataset of 103 findings. No unread-findings dataset exists.
  • Even the base rates. One critique showed markets weren’t significantly better than a “bet everything fails” rule (p = .177) — the easy version is shakier than its headlines.
What being first looks like

A preregistered study: give participants only a paper’s title and field metadata, ask them to predict its specific findings — which of several hypotheses was supported, the effect’s direction and size band — then score the predictions against the actual paper, against a chance baseline. First run of that experiment defines the task.

All 31 case studies for this area
Replication prediction — forecasters see the paper, bet yes/no
  1. Dreber et al. (2015), PNAS — prediction markets over 41 completed psychology replications: market prices called 71% correctly (29/41); a survey of the same people managed 58%, indistinguishable from chance [published]. Gap: forecasters knew the studies and predicted only replicate/fail. Source
  2. Gordon et al. (2021), PLoS ONE — pooled four forecasting projects, 103 findings: markets 73% accurate; markets r = 0.581, surveys r = 0.564 against outcomes [published]. Gap: as Dreber, at scale. Source
  3. Camerer et al. (2016), EERP — 18 experimental-economics papers; 11/18 (61%) replicated; both surveys and markets “predicted” 61% — but since all beliefs sat above 50%, neither ever forecast a failure [published]. Gap: forecasters saw the papers; binary only; headline accuracy equalled the base rate. Source
  4. Camerer et al. (2018), SSRP — 21 social-science experiments from Nature/Science; 13/21 replicated; average replicated effect was 46% of the original size [published]. Gap: as Dreber. Source
  5. Open Science Collaboration (2015) — 100 psychology studies rerun; 36 of 97 significant results replicated (37%) [published]. Gap: the base rate that makes binary forecasting look easy; not itself a prediction task. Source
  6. Many Labs 2 — large-scale replication project; one of the four pooled forecasting projects [reported]. Gap: as Dreber. Source
  7. Benjamin, Mandel & Kimmelman (2017) — cancer experts forecast 75% probability of replicating significance and 50% for effect size; none of the first studies replicated on either criterion; higher h-index helped, more topic expertise hurt [published]. Gap: experts overconfident about known studies; the unread-findings task is harder and untested. Source
  8. RP:CB (Reproducibility Project: Cancer Biology) — 29 direct replication studies planned [published]. Gap: as Dreber. Source
  9. Chandrashekar et al. (2026) — the newest replication-forecasting study: 16/28 (57%) replicated, against predictions of 70% (markets) and 65% (surveys); market r = 0.43, survey r = 0.26 [published]. Gap: still binary, still known papers. Source
  10. Fraser et al. (2023), repliCATS — structured expert elicitation for replicability: 84% classification accuracy, AUC 0.94; applied to 3,000 claims over 18 months [published]. Gap: assesses known claims; its elicitation machinery is the closest tool an unread-findings test would need. Source
  11. JSSR (2026), markets vs surveys — 103 findings, 206 forecasts: market error 0.384 vs survey 0.423; the difference was not significant (p = 0.135) [published]. Gap: as Dreber. Source
  12. Replication Index critique (Schimmack) — market performance varied across projects and was not significantly better than a “bet everything fails” rule (p = .177) [reported]. Gap: even binary replication forecasting is fragile — the unread-findings task would be harder still. Source
  13. Dreber market designs — how the markets were run: e.g. SSRP ~200 participants over two weeks; EERP 97 participants over ten days [reported]. Gap: design details for the nearest task family. Source
Expert forecasting — the question is always fully shown
  1. Mellers et al. (2014), Good Judgment Project — training, teaming and tracking improved forecasters’ calibration and resolution; the top 2% were moved into elite teams [published]. Gap: forecasters see the full question; in the target task they don’t see the paper. Source
  2. Tetlock & Gardner (2015), Superforecasting — across 2,200–3,900 tournament forecasters, the top 60 “superforecasters” ran ~30% more accurate than career intelligence analysts [reported]. Gap: as Mellers. Source
  3. GJP results summary — the Good Judgment team beat its control group by 60% (year 1) and 78% (year 2); teams beat individuals by 23%; superforecasters beat prediction markets by 15–30% [reported]. Gap: as Mellers. Source
  4. GJP training experiment — a ~60-minute training module improved average forecasters’ accuracy ~10%; superforecasters scored ~60% better than average [reported]. Gap: as Mellers. Source
  5. Tetlock (2005), Expert Political Judgment — the foundational track record of expert forecasting [reported]. Gap: as Mellers. Source
  6. IARPA ACE programme (2010–2015) — five teams forecasting hundreds of geopolitical, economic and military events over four years [published]. Gap: the infrastructure for forecasting known questions; no unread-findings analogue. Source
Structured elicitation — how expert judgement gets extracted
  1. Dalkey & Helmer (1963), the RAND Delphi — iterated written questionnaires with controlled feedback [reported]. Gap: an elicitation method, not a prediction-of-findings task. Source
  2. Gustafson et al. (1973), estimate-talk-estimate — head-to-head test found the talk, not the paper feedback, carried the accuracy gain [reported]. Gap: as Delphi. Source
  3. IDEA protocol (Hanea 2017) — Investigate–Discuss–Estimate–Aggregate: the structured protocol repliCATS builds on [reported]. Gap: as Delphi. Source
  4. Woudenberg (1991), Delphi evaluation — no evidence Delphi beats other judgement methods; its consensus comes mainly from group pressure to conform [published]. Gap: cautions that elicitation method matters less than assumed. Source
  5. “The Intelligence Game” (2012) — Delphi groups in the IARPA tournament: US groups beat the baseline by 10%, a team prediction market by 47%, Australian Delphi groups by 51%; diversity was the suspected factor [published]. Gap: as Mellers. Source
  6. First Delphi experiment (1948) — Delphi was first used to improve betting scores at horse races [reported]. Gap: historical. Source
  7. “Return to Delphi” (2026) — flagged, not counted — claims a forecasting tournament, but its institutions (“Institute for Retrospective Forecasting,” “Dept. of Oracular Engineering”) mark it as apparent satire. Gap: recorded here only to show it was found and deliberately excluded from the evidence. Source
The prediction-market literature at large
  1. Majjha evidence index — an index of prediction-market accuracy studies from 1988 to today: elections, corporate forecasting, sports, films, macro data, replicability [reported]. Gap: confirms the breadth of forecasting research — none of it tests predicting unread findings. Source
  2. Iowa Electronic Markets (1988– ) — academic election markets; the founding prediction-market literature [reported]. Gap: known-question forecasting. Source
  3. Corporate prediction markets (Google, Ford) — internal markets forecasting company outcomes, reviewed in the Review of Economic Studies [reported]. Gap: as Iowa. Source
  4. DARPA SCORE programme — the funder behind repliCATS [published]. Gap: even the funder of the closest task family scopes itself to replicability of known claims. Source
  5. pooledmaRket dataset — the public dataset of forecasts for 103 findings [published]. Gap: the data asset for binary replication forecasting; no unread-findings dataset exists. Source
AREA 07 · SELF-ENCAPSULATING MESSAGES + HUMAN–AI GROUNDING No published work

Messages that carry their own context

Write the message so it survives on its own — background, definitions and edge cases packed inside — because whoever reads it may never see the conversation it came from.

In plain English

When two people talk, they quietly build common ground — shared understanding of what words mean, what’s been decided, what “it” refers to. Psychologists call the building process grounding, and between humans it’s well understood. Between a human and an AI, it mostly fails: studies show today’s AI systems assume common ground instead of building it — and, remarkably, the training that makes them follow instructions (RLHF) made the problem worse, because it teaches them to act on commands rather than check understanding.

The fix nobody has tried in print: stop relying on the conversation. Deliberately write self-encapsulating messages — messages built like sealed capsules that carry their own prerequisites, definitions and edge cases, designed to be re-read cold, later, by a reader who wasn’t there. Researchers searched for this idea under every name they could think of. It does not exist in the published record. Neither does its cousin: deliberately pre-loading shared understanding before a human–AI task begins.

A PLAIN MESSAGE “do the same thing as last time, but watch out for the usual problem” meaningless to anyone who wasn’t there A SELF-ENCAPSULATING MESSAGE the task + its background + definitions + edge cases + what “done” means still works read cold, weeks later the left pattern is universal practice · the right pattern has no published study at all
Everyone writes the left kind. The right kind — designed, tested, measured — is the gap.
10%
chance two separate conversations pick the same word for the same object (Brennan 1996) — why cold reading fails [reported]
2–4 yrs
age of the human–AI grounding field — it is still at the stage of diagnosing and measuring the problem
20
participants in the Google Home study where the assistant’s commonest reply — “Sorry, I’m not sure” — gave users nothing to build understanding on [published]
0
published studies of self-encapsulating messages, or of pre-loading shared understanding before a human–AI task

Closest anyone has come

Shaikh et al. (2024), “Grounding gaps” — measured it directly: language models ground less than humans, presuming common ground; instruction tuning and preference optimisation reduce grounding behaviour [published].

The key diagnosis of the problem this area addresses. It diagnoses; it doesn’t design the message-side remedy.

Poelitz et al. (2026), common-ground benchmark — the first instrument for measuring human–AI common ground; with GPT-4.1 it found one-shot overlong answers and the repair burden dumped asymmetrically on the human [published].

Measures the deficit precisely. Doesn’t test pre-loaded or self-encapsulating messages as the cure.

Entrainment systems (Shi et al. 2023; Kumar & Dušek 2024) — teaching AI to mirror the user’s word choices (“entrainment”); trained systems entrain substantially better than base models.

The nearest implemented remedy — but it’s reactive, turn by turn, inside a live conversation. The target works before and without one.

Clark’s common-ground theory (1986–1996) — the human–human foundation: grounding as presentation + acceptance; “communal” common ground assumed from shared group membership; the principle of least collaborative effort.

Communal common ground is the nearest existing concept to pre-loaded understanding — but it’s assumed from membership, never deliberately constructed per message. And self-encapsulation deliberately spends more effort up front to save effort later — a trade the least-effort principle never evaluates.

What’s still missing

  • The artefact itself. No paper designs, tests or measures a message built as a self-contained, re-readable unit carrying its own prerequisites and edge cases. Searched under every name found: nothing.
  • Pre-loading. No study tests deliberately loading shared understanding before a human–AI task — every tested intervention is reactive and turn-based.
  • A proactive remedy. The field’s fixes so far train the model to ground better mid-conversation. The target designs the message so it survives without the conversation.
  • Human–AI entrainment as a solved behaviour. Position papers state plainly that language models don’t adapt to individual users — listed as an open problem.
What being first looks like

One clean experiment: give matched content to human–AI pairs two ways — as a plain message, and as a self-encapsulating message (prerequisites, edge cases, disambiguation packed in). Then remove the context and test comprehension and task success. Success = the encapsulated message still works cold; the plain one doesn’t.

All 31 case studies for this area
Human–AI grounding — the core literature
  1. Shaikh, Gligorić & Khetan (2024), “Grounding gaps” — language models generate language with less conversational grounding than humans, presuming common ground; instruction tuning and preference optimisation reduce grounding acts [published]. Gap: the key finding — AI doesn’t ground — is the problem the target addresses; the paper diagnoses, it doesn’t design self-encapsulating messages. Source
  2. Poelitz, Doshi-Velez & Lindley (2026), common-ground benchmark — a collaborative puzzle task requiring joint action, coordination and repair; validated with GPT-4.1, revealing single-turn overlong responses, asymmetric grounding burden and shallow grounding [published]. Gap: the first benchmark for human–AI common ground — measures the deficit; doesn’t test the message-design remedy. Source
  3. Cho & Rader (2020), CSCW — 20 participants using Google Home; the commonest response, “Sorry, I’m not sure how to help,” gave nothing to build on, while even tangentially related answers helped grounding more [published]. Gap: grounding failure modes in human–AI dialogue; the message-design remedy untested. Source
  4. Wu et al. (2025) — when models are adapted to perform more grounding interaction patterns, human–AI collaboration improves [reported]. Gap: grounding behaviour is trainable — adjacent to, but not, message design. Source
  5. Wray, Kirk & Folsom-Kovarik (2022) — a taxonomy of common-ground components for human-machine teaming; common ground named an “identified gap” for the field [published]. Gap: a gap analysis of exactly this problem — with no message-design solution. Source
  6. Schelble et al. (2022), “Let’s think together!” — human-agent teams developed team cognition similarly to all-human teams but trusted agents less and showed inconsistent mental-model similarity [published]. Gap: shared mental models measured — the intervention was team composition, not message design. Source
  7. Bidirectional human-robot value alignment (Science Robotics, 2022) — robots inferring user values from in-situ feedback and explaining decisions; real-time mutual understanding shown achievable in cooperative tasks [published]. Gap: bidirectional grounding with robots — pre-loading shared understanding is not the mechanism. Source
  8. Ouyang et al. (2022), InstructGPT — the instruction-following paradigm hypothesised as the cause of the grounding gap: models trained to act on commands rather than build common ground [reported]. Gap: identifies the training cause of the deficit the target work designs around. Source
Entrainment — AI learning to talk like you, turn by turn
  1. Ostrand & Berger (2024) — position paper: language models do not adapt their language user-specifically; two named gaps, including building models that adapt to users [published]. Gap: the entrainment half of this area, stated as an open problem — no solution tested. Source
  2. Shi, Sen & Lipani (2023), MULTIWOZ-ENTR — current response-generation models don’t adequately do lexical entrainment; new dataset and measure provided [published]. Gap: entrainment operationalised for dialogue systems — the nearest implemented work — but reactive (reuse the user’s words), not pre-loaded understanding. Source
  3. Kumar & Dušek (2024), LEEETs-Dial — a task-dialogue system trained for entrainment; all three techniques gave substantially better entrainment than the base model [published]. Gap: entrainment trained into a system — still turn-by-turn adaptation, not self-encapsulating messages. Source
  4. Koulouri et al. (2014) — humans align their word choices with a robot in a route-finding task (the “bendy road” effect) [reported]. Gap: human-to-system alignment shown; system-to-human pre-loading not studied. Source
  5. Coulston et al. (2002) — children modify their speech volume in response to animated characters — louder with an “extrovert,” quieter with an “introvert” [reported]. Gap: as Koulouri. Source
  6. Lopes et al. (2013/2015) — entrainment in modular dialogue systems [reported]. Gap: early entrainment work, pre-neural. Source
  7. Hu et al. (2014) / Dušek & Jurčíček (2016) — entrainment in modular dialogue systems [reported]. Gap: as Lopes. Source
  8. Ostrand & Chodroff (2021) — speakers synchronise speech rate and phonetic patterns with partners [reported]. Gap: human–human entrainment evidence. Source
  9. Nenkova et al. (2008) — entrainment correlates with dialogue success [reported]. Gap: the payoff for entrainment — human–human. Source
  10. Babel (2012) / Kim et al. (2011) — phonetic alignment evidence [reported]. Gap: human–human alignment levels. Source
  11. Fernández (2016), NASSLLI lecture — survey of alignment at phonetic, lexical, syntactic and semantic levels; exploiting alignment in interface design sketched [published]. Gap: pre-loading shared understanding is not among the mechanisms considered. Source
Human–human grounding theory — the well-studied baseline
  1. Clark & Schaefer (1989) — grounding as a two-phase process: presentation, then acceptance, with evidence of understanding [reported]. Gap: the theory the human–AI work imports — designed for symmetric human partners. Source
  2. Clark (1996), Using Language — common ground split into communal (from community membership) and personal (from shared experience) [reported]. Gap: communal common ground is the nearest concept to pre-loaded understanding — assumed, not deliberately constructed per message. Source
  3. Clark & Brennan (1991) — the costs of grounding; misalignment accumulates and the human bears the repair burden [reported]. Gap: the cost model that motivates self-encapsulating messages — the front-loading remedy is not in Clark. Source
  4. Clark & Wilkes-Gibbs (1986) — the principle of least collaborative effort: minimise total grounding effort across both parties [reported]. Gap: self-encapsulating messages deliberately spend more effort up front to save it later — a trade-off this principle doesn’t evaluate. Source
  5. Brennan (1996) — in referential communication, two conversations chose the same term for the same object only ~10% of the time [reported]. Gap: quantifies why out-of-context reading fails — the problem self-encapsulation would solve. Source
  6. Garrod & Anderson (1987) — low word variability within a conversation: partners converge on shared terms [reported]. Gap: as Brennan. Source
  7. Pickering & Garrod (2004) — interactive alignment through priming [reported]. Gap: a human–human alignment mechanism; language models don’t prime the same way. Source
  8. Traum (1994) — a computational model of grounding acts (acknowledgements, repairs, requests for repair) as a state machine [reported]. Gap: the classical computational grounding model — turn-based, not message-based. Source
  9. Giles (2008), communication accommodation — speakers negotiate social distance through alignment [reported]. Gap: the social motivation for alignment. Source
  10. Graesser et al. (1995) — ineffective grounding in tutoring leads to misunderstanding the student and irrelevant feedback [reported]. Gap: the stakes of grounding failure. Source
The absences — recorded as findings in their own right
  1. Self-encapsulating messages — absence finding — searches for self-contained, re-readable messages carrying their own prerequisites and edge cases found no published work under any name. Nearest concepts: Clark’s communal common ground (assumed, not constructed), the grounding-gap diagnosis, and least collaborative effort (which prices the trade-off without testing front-loading) [unknown — absence of evidence after targeted search, not proof of nonexistence].
  2. Pre-loading shared understanding — absence finding — no study found tests deliberately pre-loading shared understanding before a human–AI task. The closest tested interventions are entrainment training and grounding-behaviour adaptation — both reactive and turn-based [unknown].

What couldn’t be verified

The research behind this page flagged its own weak spots rather than smoothing them over. They are: