Research dossier · 2026-11-18

Filling a scanned form like a person would — what 100 case studies say

The mission: build a tool that takes a noisy scanned paper form — not a native digital form — fills it accurately in a handwriting style, and converts flat or scanned PDFs into truly fillable AcroForm forms. This is the evidence base: 100 documented case studies — peer-reviewed papers, primary docs, and vendor benchmarks, every figure tagged by how much to trust it — gathered at a 50 / 40 / 10 ratio of on-topic, closely related, and deliberately novel work.

100 case studies Numbers tagged [published] [reported] [estimated] [unknown] Researched 2026-11-18 via web index
50 ON-TOPIC
40 CLOSELY RELATED
10
On-topic — form detection, filling, OCR, dewarp, handwriting Related — layout, registration, HITL, coordinates Kaleidoscope — AR, medical, photogrammetry, servoing…
01

The verdict

The pieces for a scanned-form filler exist and are mostly open — but nobody has assembled them into one calibrated system, and nobody publishes placement-error numbers for filled output. That gap is the opening.

The winning architecture is a two-layer system — best-effort perception proposes, a deterministic engine disposes — with homography registration against the form’s own printed furniture, per-field confidence routing to human review, and a render → measure → nudge → re-render servo loop that logs before/after coordinates and per-field error, so the system learns from every fill.

Confidence: medium-high on the architecture direction (it converges across many independent sources) · medium on specific accuracy figures (benchmarks are template-leaky, vendor numbers unverified) · low on any claim about end-to-end filled-form placement accuracy — because no source publishes it.

The house style — handwriting on the line, not type in a box
Applicant name
Filled by hand, placed by measurement
Date
Nov 18, 2026

The font is the easy part — a legible hand like Caveat reads better than Arial ever will. The research below is about the hard part: landing it on the line, every time, and proving it.

02

Key numbers

The quantitative anchors the whole design hangs on. Tags show how much to trust each one.

0.29–0.34 F1

OpenCV line-detection F1 on AutoFormBench, vs YOLOv11 as the best detector. Classical heuristics are fine for checkboxes, hopeless for lines and boxes — learned detection wins by ~2×.

OT-03 · AutoFormBench [published]
+19 F1

What layout awareness is worth: text-only BERT scores 60.3 F1 on FUNSD; layout-aware models reach 79.3, and LayoutLMv3 reaches 92.08. Position is signal, not decoration.

OT-01 / OT-14 · FUNSD [published]
−40 pts

The cost of flattening a document to text before understanding it: receipt extraction drops from 87.46 to 47.00 accuracy when layout is thrown away first.

CR-14 · Berghaus benchmark [published]
28.5% → 0.9%

Character error rate at 4° of skew, before and after deskew. Geometry correction comes before everything else — even 1.5° of skew costs ~5.6 points of CER.

OT-29 · Radon/Hough deskew [reported]
68–73%

Real-world handwriting extraction accuracy in a business IDP benchmark — and cursive narrative reads at only ~45%. Verification-by-re-reading fails exactly where handwriting matters most.

CR-15 / CR-16 [reported]
120 → 6.5

Handwriting-synthesis quality (FID on IAM, lower is better): GANwriting in 2020 vs FW-GAN in 2025, which matches a writer’s hand from a single sample. One-shot synthesis is now practical.

OT-30 / OT-33 [published]
0.12–0.67 px

Reprojection error that photogrammetry treats as normal — Pelican-2 band registration under 0.12 px, USGS historical SfM at 0.67 px RMS. The registration fields publish their residuals; form filling doesn’t. Yet.

K-06 / K-07 [published]
81 → 42 F1

RE2’s collapse when a form-understanding model moves to a new domain. Cross-template generalization is the norm’s weak point — plan for per-template fine-tuning.

OT-18 · RE2 + DIVERSEFORM [published]

Layout is worth ~32 points of F1

FUNSD form-understanding scores as models gained access to 2D position. Same dataset, same task — the only thing that changed is whether the model can see where words sit.

0 25 50 75 100 60.3 79.3 92.08 BERT · text only Layout-aware LayoutLMv3
Source: OT-01 / OT-14 · FUNSD (2019) progression · all figures [published]
03

The ranked steal list

Fifteen moves worth taking, ranked by expected impact on fill accuracy × ease of adoption. References point into the study index below.

  1. 1

    Two-layer architecture: perception proposes, a deterministic engine disposes

    The model never writes the PDF. It produces a confidence-scored field manifest; a deterministic builder — AcroForm injection or overlay stamp — executes it. Every guess carries a confidence below 1.0. OT-06 · OT-09

  2. 2

    Homography registration against the form’s own printed furniture

    Detect 4+ stable anchors — corner rules, title block, bold lines — solve a homography, and report mean anchor reprojection error in pixels as the per-page registration score. Above threshold → human review. CR-28 · CR-29 · CR-32 · K-01 · K-06

  3. 3

    Median-stack template + variance map to discover data cells

    Align N samples of the same form and median-stack them for clean printed furniture; per-pixel variance separates furniture (low) from filled data (high). Field boxes become objective — template authoring drops from hours to minutes. CR-31

  4. 4

    Render → measure → nudge → re-render servo loop

    Treat placement like visual servoing: render the filled page, measure each value’s pixel error against its field box and baseline, apply a proportional correction, re-render, iterate to convergence — logging before/after coordinates and per-field error every pass. K-09 · K-03

  5. 5

    Per-field confidence routing with risk bands

    ≥0.92 auto-pass, 0.75–0.92 human review, <0.75 quarantine — thresholds per field type, not global. Route by risk as well as confidence: signature and amount fields are always reviewed. CR-30 · CR-33 · CR-34

  6. 6

    Edge-tolerance + point-in-box placement metrics

    Score placement by edge tolerance (5/10/20%) and point-in-box accuracy, and report success-within-X — within 2pt, within 5pt — the way the servoing literature does, not IoU alone. OT-03 · CR-37 · CR-38

  7. 7

    Deskew + dewarp before anything else, with a quality gate

    Cascade: affine → deformable → iterative. Use AAD (axis-aligned distortion) as the dewarp gate metric, score scan quality (DIQA), and route bad scans to enhancement or review. OT-27 · OT-28 · OT-29 · OT-35

  8. 8

    Multi-engine OCR with annotation-free selection

    No single OCR engine wins everywhere [published]. Run two engines and pick per page by agreement. Default: PaddleOCR for print, TrOCR/EasyOCR for handwriting verification, VLM OCR as the hard-page fallback. OT-46 · OT-47 · OT-48 · CR-24

  9. 9

    YOLO-class field detection at high resolution, fine-tuned per form family

    OpenCV is fine for checkboxes and hopeless for lines and boxes. Train on CommonForms, fine-tune per template family — cross-domain transfer collapses (81 → 42 F1 [published]). OT-02 · OT-03 · OT-18

  10. 10

    Label ↔ field association as graph assignment, not nearest-neighbor

    Geometry-aware models — graph edges, MIP assignment — beat text-order heuristics. Layout alone is worth ~+19 F1. OT-17 · OT-14 · CR-28

  11. 11

    Cap-height / baseline typography for the renderer

    Align filled text to the form’s baseline grid; center by cap height, not em-box; use kerning and tracking to fit fixed-width fields; optical sizing by field height. K-08

  12. 12

    One-shot handwriting synthesis with a font fallback

    FW-GAN-class models (FID 6.5 [published]) can match a user’s hand from one sample. Fall back to a handwriting font when synthesis is unavailable; the long-term target is vector Bézier stroke output for resolution-independent embedding. OT-33 · OT-32 · OT-34

  13. 13

    Template-disjoint evaluation + benchmark skepticism

    Split evals by template, never by page. Discount published key-information-extraction numbers ~5–10 F1 for leakage, and run a 200–500-document blind pilot on our own forms before trusting anything. OT-05 · OT-21 · CR-02

  14. 14

    Checkboxes: detect-then-VLM-read

    YOLO detection plus a vision model reading the mark’s state beats OCR + string matching by ~7 points and is robust to faint marks. OT-22

  15. 15

    Correction logging as training data

    Every human correction — logged against field, page, document type, and before/after coordinates — is the dataset for the calibration loop. The system must get better the longer it runs. CR-34 · CR-35

04

Recommended architecture

Scanned-form → filled-form pipeline. Input: a scanned, image-only, noisy PDF plus a data record of field values. Output: a flattened filled PDF for distribution, a fillable AcroForm PDF for reuse, the field manifest, an error-ledger entry, and an audit trail.

0
INTAKE

Intake & triage

Detect the document kind first. AcroForm already present? Fill natively with pypdf, flatten, done. XFA? Treat as flat — it’s deprecated in PDF 2.0 (2017) [published spec fact]. Flat or scanned → render each page at 300 DPI, read /Rotate and CropBox per page, and run a DIQA quality gate: low sharpness/shadow/crease score → enhance or review.

1
GEOM

Geometry normalization — the registration stack

  • Deskew (Radon for dense text, Hough for ruled forms) + 4-way orientation.
  • Dewarp cascade: global affine → deformable → iterative refine. Enforce a fold-free warp; record the warp field per page.
  • Template ID: embedding shortlist (top-3) → homography proof by inlier count, spatial spread, and per-anchor reprojection residual.
  • Known template → register the scan to it via homography from 4+ printed-furniture anchors. Unknown → proceed template-free.
  • Gate: mean anchor reprojection error (px) + AAD score → pass / review / quarantine.
2
DETECT

Field detection — perception proposes, never writes

A YOLO-class detector (FFDNet, CommonForms-trained, at 1216px) proposes Text / Choice / Signature boxes. In parallel, a classical pass: OpenCV contours for checkboxes (precision 0.82 [published]), line detection for underlines, TableFormer-class models for grid forms. Where multiple samples of a template exist, the median-stack variance map discovers data cells. Output: candidate fields, each with confidence < 1.0.

3
READ

Understanding — labels, values, and the manifest

Multi-engine OCR with per-word boxes and per-page engine selection; a layout model for regions; label ↔ field association by graph/MIP assignment; checkbox state by detect-then-VLM-read. Output is the field manifest — {name, type, page, rect in PDF points, label, confidence, evidence} — and the manifest is the contract between perception and construction.

4
REVIEW

Human review — confidence + risk routing

≥0.92 auto · 0.75–0.92 review · <0.75 quarantine · signature and amount fields always reviewed. Side-by-side UI: scan vs manifest overlay, click-drag to fix boxes. Borrow the music-alignment shortcut: the human marks anchors and structure; the machine does the dense work. Every correction is logged with before/after rects.

5
FILL

Fill — the deterministic construction layer

  • AcroForm path: inject real fields at manifest rects, set values, flatten on export. Widget rects: a 16pt text box sitting 1pt above the rule [reported].
  • Overlay path (scanned forms): handwriting-style text via a ReportLab stamp page merged with pypdf, or PyMuPDF insert. Coordinate mapping: pixels → PDF points via 72/DPI scale + y-flip + CropBox/Rotate correction, composing the deskew rotation.
  • Baseline-align to the form’s printed baseline grid; center by cap height; fit by tracking before shrinking the font.
  • Handwriting: one-shot synthesis from the user’s sample when available, handwriting-font fallback, vector strokes as the long-term target. Checkboxes get a centered mark; signatures are fitted to their box, never overflowing.
6
LEARN

Calibration / measurement loop ★ — the learning system

  • Servo loop, per fill: render the filled page → measure per-field placement error (dx/dy in pt and px, baseline offset, overflow beyond the box, rotation residual; re-OCR the rendered value and compare, ANLS; checkbox mark inside its box?) → apply a proportional nudge (k ≈ 0.7) → re-render → iterate until error < tolerance (e.g. 1pt) or 3 iterations. Log every iteration.
  • Per-field error ledger: for each (template, field) — before/after coordinates, error vector, correction applied, final error, reviewer overrides. Aggregated into per-template bias vectors and per-scanner distortion profiles.
  • Learning: apply learned per-field offsets on the next fill of the same template; fields with high error variance get more review and become detector fine-tuning data; confidence thresholds recalibrate from observed error rates; the median-stack template sharpens with each aligned sample.
  • Dashboard metrics: straight-through-processing rate, per-field mean/p95 placement error, success-within 1pt/2pt/5pt, reviewer override rate, error trend over time. Evaluation stays template-disjoint, with a 200–500-doc blind pilot before any release.

Perception ≠ construction

Separate processes, manifest as the only interface — so a bad detection can never corrupt a PDF, and every output is auditable. OT-06 · OT-09

Homography first, deformable second

A homography carries an error certificate — reprojection residuals. Dense warps don’t, and can silently translate cells. CR-31

One coordinate home

All coordinates live in PDF points with a recorded transform chain — render DPI → deskew → dewarp → homography → CropBox/Rotate → y-flip — so any coordinate traces back to its source pixel and any error can be blamed on a stage.

Close the loop on the render

The servo loop measures the pixels of the actual filled page, not the model’s prediction — the render is the only ground truth that matters. K-09

05

Kaleidoscope patterns

What emerges when all 100 studies are viewed together — including the ten borrowed from AR, medical imaging, photogrammetry, typography, robotics, and music.

PATTERN 01

Everything is registration

Form filling, medical imaging, satellite mapping, SfM, AR, and music alignment are the same problem: put two things in one coordinate frame and measure the residual. The best field — photogrammetry, at 0.12–0.67 px — treats the residual as a first-class output. Our filler should too.

PATTERN 02

Anchors beat content

Every robust system anchors to stable, high-contrast, geometrically regular features — AprilTags, title blocks, corner rules — never to the thing being measured. Anchor to the form’s printed furniture; never to the data.

PATTERN 03

Coarse-to-fine is universal

Affine → deformable → iterative. Global → local. Shortlist → proof. Detect → zoom-in refine. No domain solves alignment in one shot.

PATTERN 04

A little human structure beats full automation

Music alignment gained +49 points from a human clicking repeat signs [published]. Form filling will be the same: human-marked anchors + machine dense alignment beats either alone. Design review for structural input, not pixel-pushing.

PATTERN 05

Measure error in domain units

Half a measure (music), millimeters at the end-effector (servoing), percent of sheet diagonal (blueprints), points on the page (forms). Raw pixels aren’t comparable across scans — normalize.

PATTERN 06

Confidence is routing, not a quality claim

Every production system routes on confidence bands; none claims the model is “accurate enough” outright. The honest product is a calibrated router plus a fast review UI.

PATTERN 07

Layout is information

Removing layout costs ~40 accuracy points; adding 2D position is worth +19 F1. Any pipeline stage that flattens to text destroys the signal the next stage needs.

PATTERN 08

Synthetic data bootstraps small fields

Blender-rendered warped documents, copy-paste checkbox augmentation, synthetic handwriting for training recognizers. We can synthesize warped, filled forms faster than we can collect them.

PATTERN 09

Benchmarks lie by template

75% template replication in SROIE’s test set, 16% in FUNSD, ~10 mAP from page-splitting. Read every headline number in this dossier as an upper bound on real-world performance.

PATTERN 10

The output format is the product

Vector strokes, real AcroForm fields, schema-first manifests — the systems that last are the ones whose output is editable, auditable, and standards-compliant, not a rendered image.

06

The 100 studies

The full evidence base, one line each: what it found, and where it came from. Filter by band or search by name, method, or result.

07

Open questions — what nobody has solved

The honest frontier. Several of these are the product opportunity; a few are warnings.

  1. Q1

    Placement-error benchmarks for filled forms do not exist

    No source in this research publishes “our filled text lands within X pt of the target.” We have to create the metric — edge tolerance + baseline offset + success-within-X — and the gold set ourselves.

  2. Q2

    Handwriting synthesis at field scale

    Every surveyed synthesizer works at word level. Nobody demonstrates fitting synthesized handwriting into a fixed-width, fixed-height field with correct baseline, wrapping, and overflow behavior.

  3. Q3

    End-to-end error attribution

    When a fill lands 3pt low, is it the dewarp, the homography, the detector, or the renderer? No surveyed system attributes placement error to pipeline stages. Transform-chain logging is designed to make it answerable — but it’s unproven.

  4. Q4

    Dewarp error certificates

    Dense dewarpers can silently translate content, and no dewarping paper provides per-pixel uncertainty on the warp. VoxelMorph-style uncertainty has not been transferred to document dewarping.

  5. Q5

    Cross-template generalization

    RE2’s 81 → 42 F1 cross-domain collapse is the norm, not the exception. Whether one field detector can generalize across unseen form families — or per-template fine-tuning is mandatory — remains open.

  6. Q6

    Reading cursive back

    At ~45% accuracy on cursive narrative [reported], verification-by-re-OCR fails exactly where handwriting-style filling is most used. Reading our own filled output back is an unsolved verification path for cursive.

  7. Q7

    Multi-page form consistency

    Joint, bundle-adjustment-style registration across the pages of one form is proposed in photogrammetry but not demonstrated for forms; per-page independent processing can drift.

  8. Q8

    Legal validity of synthesized handwriting

    No surveyed source addresses whether a synthetically rendered handwritten signature or filled value is legally equivalent to a wet-ink one. E-signature law (ESIGN/UETA) covers intent and attribution, not glyph provenance. Legal review required before shipping signature synthesis.

  9. Q9

    What “good enough” placement actually is

    No user study quantifies the placement-error threshold at which a filled form gets rejected by a human reviewer or a downstream system. The 1pt tolerance in the architecture is a guess [estimated], not a measured requirement.

08

Couldn’t verify

Stated plainly, per the research standard — accredited, posted, documented, and honest about the seams:

  • Accuracy tables on low-authority pages (OT-23 checkbox guide, OT-49 Surya 97% claim, CR-24 DeepSeek-OCR >95% claim) are listed as [reported, unverified] — do not rely on them.
  • Vendor pricing figures (Textract / Azure / Document AI per-1k-page prices) are as reported by secondary comparisons in November 2026 and change frequently; verify against vendor pricing pages before budgeting.
  • Several papers were read via secondary summaries or search-result snippets rather than the full primary PDF; where the primary PDF was the retrieved source, it’s the one linked.
  • No live-browser verification was performed for this dossier, and no tool was installed or run. Library and API claims (acroforge, pypdf, PyMuPDF, Docling) are as documented in the cited pages.
  • Exact mAP/F1 for some models (FFDNet-L, WriteViT, IVGocr/direct) were not stated in the retrieved text and are marked [unknown].