Research digest · 12 Nov 2026 · Literature only — no new experiments

The last centimetre is a contact problem, not a vision problem.

How real systems place into a stack without knocking it apart, check stability before they let go, and recover when a cup leans — translated into fixes for your simulator. Every number below carries its source; anything the literature does not settle is flagged as unknown, not filled in.

Arm 5-DOF · 35 / 30 / 12 cm links Gripper parallel-jaw Cups 10 × plastic · 7.5 cm rim · 5.5 cm base · 9 cm tall · 150 g Target 4–3–2–1 pyramid Sensing wrist + overhead camera · no F/T sensor World physics simulation

The blunt message: every system that stacks well either adds force sensing, replaces force with geometry + a physics rollout, or learns contact from massive demonstration data. You have no force sensor — but you do have the simulator itself. That is your force sensor. Use its contact events and cloned rollouts as ground truth, and use vision/geometry as the policy-facing signal.

43% → 78%

What adding wrist / joint force-torque observations did on RGB-Stacking Triplet 2 (CHEF). The single clearest price tag on flying blind at contact.

Source: CHEF summary — alphaxiv.org
0/10 → 4/10

Vision-only ACT vs. touch-predicting ViTacFormer on separating two nested paper cups and stacking them. Vision alone lifted both cups as one and toppled them.

Source: ROBOTIS demo via report — indicative, not peer-reviewed (see caveats) — youtube.com
80% · 17.9 h

AIRoA-0, the best cup-pyramid result found — and it needed 17.9 hours of task demonstrations, the most of any of its ten tasks, on top of 16,151 h of pre-training.

Source: AIRoA-0 technical report — arxiv.org · unreplicated, no limitations section
Start here

Your three failures, mapped to the literature

These are not quirks of your build. Each one is the documented default behaviour of a geometry-only pipeline — which is good news: the fixes are known.

Failure 1 · The shove

Descent pushes row-2 cups out of their V-seats

MoveIt / MoveIt Task Constructor has no contact detection at all. Its place sequence descends a straight Cartesian segment to a pose computed from scene geometry, opens the gripper, and retreats. If the geometry is slightly wrong, the arm pushes through the target. Your shove is the standard stack working as designed.

Steal: compute seat height from the two measured supporter poses + cup geometry (a V-seat sinks below rim height — never assume nominal stack height), descend vertically and slowly, and stop on the simulator's contact event.
Sources: PickNik MoveIt Pro docs · Lerner et al., arXiv 2404.17668
Failure 2 · The delayed topple

Looks upright after placement, tips minutes later

A cup on a flat base topples at a critical angle of ≈37° (derived — see lab). A row-2 cup does not rest on its base: it rests on two contact points, so its support “polygon” collapses to a line. It can look perfectly upright while its centre of mass sits millimetres from the edge — metastable, until the next placement vibrates it over.

Steal: stop checking tilt for V-seat cups. Check lateral CoM offset from the seat centreline instead — and re-verify the whole stack before every subsequent placement, not just the cup you just placed.
Sources: Cholewiak, Fleming & Singh, J. Vision 2015 — docksci.com · COBRA-PPM — arXiv 2403.14488
Failure 3 · No feel

No force feedback, so contact is invisible

Real systems without an F/T sensor fake it proprioceptively: FANUC's collision detection compares expected vs. actual motor torque (SRVO-050); the research standard is a generalised momentum observer on joint efforts. A 2025 Sensors formalisation cites a joint-1 threshold of 0.5 Nm with detection in ~5–6 ms.

Steal: in sim you don't need to fake it — engine contact points, normals and forces are exact. Add two vision proxies on top: track the gap-to-seat during descent, and run a slow-release probe that turns “force” into visible tip/slide before you commit.
Sources: FANUC SRVO-050 explainer · Sensors 2025, PMC · Momentum-observer review, AIMS Press
Ranked for your system

The steal list — top 10, in build order

Ranked in the report by expected impact × implementability for a vision-only, no-F/T, 5-DOF arm in simulation. Filter by what you're working on. #1 is the highest-leverage change in the whole report.

1

Pre-release sim rollout — clone, release, step, check

Before releasing for real, clone the sim state, release the cup in the clone, step the physics, and confirm it stays put. COBRA-PPM's sim-based stability prediction hit 88.6% accuracy and lifted stacking success from 74.4% to 94.2% in sim — and in your own simulator the prediction is exact, not 88.6%.

Q2 · StabilityCOBRA-PPM, arXiv 2403.14488
2

Compute the V-seat from measured supporters — never nominal height

Seat z comes from the two supporter rim circles plus cup geometry (7.5 / 5.5 / 9 cm), because the upper cup's base sinks between the rims. Correct the target for the cup's in-gripper offset seen by the wrist camera — grasp offset is a first-order source of V-seat misses.

Q1 + Q3MoveIt/MTC practiceWrench-aware admittance, arXiv 2604.19469
3

Slow-release probe before full release

Partially open the gripper while still caging the cup, watch overhead + wrist cameras for tip or slide over ~1 s, then fully release — or re-seat. This is Lerner et al.'s human-inspired future work, and it converts the force signal you lack into a motion signal you have.

Q2 · StabilityLerner et al., arXiv 2404.17668
4

Slow, vertical, quasi-static descent — stop on sim contact

Cap the final approach at 7 cm/s (RGB-Stacking's published Cartesian cap), slower in the last centimetre, straight down, and stop at first contact. Zero relative velocity at contact is the conservative norm in impact-aware work; a fast descent physically cannot stop in time once contact is detected.

Q1 + Q3RGB-Stacking, 0.07 m/s capLerner et al.
5

Verify after full withdrawal — and re-verify the whole stack every step

Only credit a placement once the gripper is fully clear: RGB-Stacking's stack-leave rule requires the TCP ≥10 cm from the object's CoM, because a touching gripper masks instability. Then re-check all cups before the next placement — delayed topples are cumulative uncertainty, per COBRA-PPM. CoRe's rule: re-observe after release, retry once on failure.

Q2 · StabilityRGB-Stacking / CHEFCoRe (2026)
6

For V-seats, check lateral offset — not tilt

Support is a line segment between two rim contacts, so the margin is millimetres of lateral CoM offset from the seat centreline. Require a margin of a few mm and re-seat otherwise. (The “few mm” is the report's engineering recommendation — no published V-seat number exists. See Unknowns.)

Q2 · StabilityCholewiak et al. 2015 + derivation
7

Retract straight up ≥10 cm — open the gripper slowly, while lifting

MTC's standard retreat is a 10 cm straight-up segment, and for good reason: gripper opening itself displaces light objects (VREgo), and finger “stickiness” drags objects on release (Lerner et al.). Peel the fingers away upward, never sideways, and re-check the cup's pose once fully clear.

Q3 · DisturbanceMTC 10 cm retreatVREgo, arXiv 2611.07272
8

Recovery ladder: nudge → re-place → backtrack top-down

Small lean, topmost cup only → nudge (Recova's template presses dz = −0.02 m, then re-verify). Leaning topmost cup → remove and re-place from a fresh seat computation. Leaning lower cup → remove everything above it first, top-down, back to the last known-good state. Never leave a leaning cup and build on it.

Q5 · RecoveryRecova, arXiv 2610.01178Lerner backtracking
9

Go compliant on final approach — let the arm yield, not the stack

Low-gain Cartesian control with gravity compensation (“minimum impedance”) achieves low contact forces with no external force sensing, given a good model. In sim this is nearly free: unexpected contact deflects the arm instead of shoving the row-2 cups.

Q3 · DisturbanceMinimum-impedance control, PA-10MERL TR2023-090
10

Momentum-observer cross-check on joint efforts

Compare expected (free-space) vs. actual joint effort during descent; a residual spike is a contact signal independent of the contact list. Cheap in sim, redundant with engine contacts — and it's the one technique here that transfers directly to a real arm later.

Q1 · ContactGMO review, AIMS PressDLR thesis 2023

Also worth stealing, from Q3: fix the seat, not just the cup. CHEF's policies learned to push the supporter object into a stackable orientation before placing on it. If a row-1 cup is misaligned, correct it before you ever descend with row 2. Source: CHEF / RGB-Stacking, Triplet 2 — alphaxiv.org.

Interactive · Why tilt is the wrong check

The V-seat lab

Drag the placed cup sideways. On a flat base you'd have ~37° of tilt to spare. In a V-seat the cup hangs on two rim contact points — so what matters is how far its centre of mass sits from the line between them.

supporter supporter support line — the whole “polygon” CoM
Centred — full marginCoM projection sits on the support line with margin on both sides. This is the only placement worth building on.
  • Flat base: critical angle ≈ atan(2.75 / 3.6) ≈ 37° — base radius 2.75 cm, CoM ≈3.6 cm up (≈40% of the 9 cm height, truncated cone). Forgiving.
  • V-seat: two contact points → support degenerates to a line segment. Stability margin is lateral distance in millimetres, not degrees.
  • Why cups “look fine” then fall: small tilt + near-edge offset = metastable. The next placement's vibration finishes the job.
  • What to log instead: after every placement, record each cup's CoM offset from its seat centreline and final CoM position — Lerner et al. store each placed object's final CoM as the initial guess for the next placement.

Honesty note: the 37° figure and this diagram are the report's derivation from Cholewiak et al.'s critical-angle principle, not a published cup measurement. The stable / marginal / toppling bands on the slider are illustrative — the report found no published V-seat offset threshold, and recommends deriving yours empirically in sim: sweep lateral offset, measure topple probability over subsequent placements.

Principle: Cholewiak, Fleming & Singh (2015) — docksci.com
Reality check

What published systems actually score

Calibrate expectations: a hand-coded geometric pick-and-place scored 51% on RGB-Stacking's diverse shapes. Verification + recovery is what closes the gap — not a better nominal trajectory.

SystemTaskResultWhat made it work / fail
AIRoA-0 (Nov 2026)Cup pyramid, bimanual Franka, real80% · 15 trials, partial credit16,151 h handheld-demo pre-training + 17.9 h task demos; wrist cameras only; stage prediction + motion memory. Closest win vs. rivals (π0.5: 71%). Unreplicated; no limitations section. arXiv
DexImit (RSS 2026)Six-cup pyramid, bimanual, sim52%Video→4D reconstruction at near-metric scale; subtask decomposition; force-closure grasp synthesis. Baselines failed entirely; errors propagate along the sequential pipeline. 52% is a sim data-quality eval, not a real-robot result. arXiv 2602.10105
Learned Speed Prior (SNU)Three-cup pyramid, real Franka29/30 (97%) · 3.50 sSpeed prior learned from demos retimes policies to human pace; 52% less jerk. The SAIL naive speed-up baseline on the same task: 0.40 success, 14.71 s — where the robot is slow matters more than overall speed. arXiv
CHEF / RGB-Stacking (DeepMind, CoRL 2021)Stack red on blue, 5 shape triplets, real96% CHEF · 51% scripted · 54% unseen shapesIterative offline RL + sim-to-real. Triplet 2 needed F/T observations (43%→78%) and learned to re-orient the supporter before stacking on it. alphaxiv · DeepMind blog
Lerner et al. (Berkeley)Rock / puck stacking, UR5 + F/T sensor100% basic (16 trials) · 17% needing adjustmentZero-net-wrench release test; iterative CoM-over-contact-point shifts. Fails on small offsets near stable poses (sensitivity limit), non-horizontal surfaces, and gripper stickiness on release. arXiv
COBRA-PPM (Oxford / Microsoft)Block stacking, Gazebo sim + real demo94.2% vs 74.4% baseline · 88.6% prediction accuracyCausal Bayesian network + online PyBullet sim at inference; models perception + actuation noise. Sim prediction is good, not perfect. arXiv
ViTacFormer / ROBOTIS (demo, Oct 2026)Separate 2 nested paper cups + stack0/10 vision-only · 1/10 tactile · 4/10 ViTacFormerTouch must be predicted, not just sensed (cross-attention fusion + autoregressive tactile prediction). Numbers from a demo video / secondary summaries, not the paper itself — indicative only. Demo video
MIT Jenga robot (2019)Jenga extraction + placement, ABB IRB 120QualitativeSoft-pronged gripper + force-sensing wrist cuff; gentle probing; learned push-vs-move from ~300 attempts. Lesson: it cannot judge a block from vision alone — it probes. MIT BCS
The five questions, in full

Playbooks Q1–Q5

Condensed from the report — open each for the techniques, numbers, and sources behind the steal list.

Q1 · Contact-aware placement without an F/T sensor

The standard stack doesn't detect contact. MoveIt/MTC: pre-place pose → straight Cartesian descent (tutorial uses 10 cm approach / lift / retreat segments in z) → open gripper → straight retreat. Contact is assumed at the computed pose.

  • Geometry proxy (primary): compute contact height from measured supporter poses + known object geometry. For a V-seat, derive z from both supporter rim circles — the cup sinks between them.
  • Depth / gap proxy: track distance from held cup to seat continuously during descent; stop at zero gap. Pre-touch sensing “reduces effective latency, since a control loop can respond to a contact before or as it occurs”; miniature ToF sensors report <8 ms latency (ToFTouch, arXiv 2611.03489).
  • Held-pose proxy: if the held cup stops tracking the gripper's commanded motion, contact has occurred. This is the vision analogue of Lerner et al.'s “lowered quasi-statically until we detect significant resistance.” No paper isolates this as a standalone technique — flagged in the report as an implicit signal.
  • Sensorless proprioception (real-arm route): momentum observers / motor-current disturbance torque. FANUC stops the servo on torque deviation (SRVO-050); Sensors 2025 cites 0.5 Nm joint-1 threshold, ~5–6 ms detection.
  • Vision-tactile proxies (context): TacFinRay reports 0.1 mm depth and 2 mm location accuracy and improved placement accuracy under uncertain picks — evidence the field treats deformation vision as a legitimate contact proxy.
  • In sim: use engine contact events as the primary detector; validate the vision proxies against that ground truth.
Sources: PickNik MTC docs · MoveIt 2 tutorial · ToFTouch · TacFinRay · ProTac
Q2 · Verifying stability — before and after release

Before release:

  • Zero-net-wrench test (gold standard, force-based): lower quasi-statically until resistance; when the net wrench on the grasped object is zero, the stack alone supports it — release. If torque is non-zero, shift the pose so the CoM sits above the inferred contact point and retry. Protocol: wait 0.1 s after motion stops, then average 0.5 s of samples (25 Hz). Vision-only analogue: run the same test on the sim engine's contact wrench, or geometrically against the support polygon.
  • Sim rollout: COBRA-PPM runs PyBullet at inference time — 88.6% prediction accuracy, 94.2% vs 74.4% task success.
  • Contact-patch estimation: estimating the extrinsic contact patch during contact formation predicts stability on release (demonstrated stacking irregular pieces on an unstable tower, arXiv 2309.14552). Geometric analogue: compute the expected patch from seat geometry.
  • Placeability scoring: a generalised metric scores stability + graspability + clearance directly from noisy point clouds, no CAD, with “CAD-comparable accuracy in predicting stability loss” (arXiv 2510.14584).
  • Slow-release probe: partially release, watch behaviour, then commit or re-seat (see steal #3).

After release: re-observe and retry once on failure (CoRe); only judge stability after full withdrawal (TCP ≥10 cm, RGB-Stacking); re-verify the whole stack before each subsequent placement (COBRA-PPM, cumulative uncertainty).

Sources: Lerner et al. · COBRA-PPM · Contact-patch estimation · Placeability metric · To Fall Or Not To Fall (2016) · CoRe (2026)
Q3 · Placing without disturbing the stack
  • Speed: RGB-Stacking caps Cartesian velocity at 0.07 m/s at 20 Hz control — the best published “how slow” number found. Impact-aware work treats zero relative velocity at contact as the conservative norm (faster impacts, up to 0.9 m/s, only for soft contacts — Dehio & Kheddar, ICRA 2021).
  • Release height: zero. Every geometric pipeline releases at contact. No published non-zero drop height exists — dropping converts potential energy into stack-disturbing impact. (See Unknowns.)
  • Gravity settle vs. driving to pose: neither pure approach wins. Published middle path: quasi-static descent to the computed pose, stop at first resistance, then small corrective shifts (raise, shift laterally, re-lower) — with the light-release probe letting gravity reveal the true equilibrium while the gripper still cages the cup.
  • Compliance: impedance control “minimises the transfer of energy, momentum and force” at contact (MERL). Minimum-impedance (low gain + good model + feedforward) gets low interaction forces with no external sensing.
  • Grasp-offset correction: estimate payload CoM offset during transport and correct the horizontal waypoint so the object, not just the TCP, lands at equilibrium (wrench-aware admittance, UR5e). Vision analogue: wrist-camera in-gripper offset.
  • Disturbance budget: RGB-Stacking terminates an episode if wrist force exceeds 20 N — a usable abort threshold analogue in sim (contact impulse).
  • Retract: straight up; approach vertically from directly above the seat with fingers oriented along the valley; check the gripper's swept volume against measured cup poses. No paper quotes a clearance number — general practice, per the report.
Sources: RGB-Stacking docs (DeepWiki) · Dehio & Kheddar · Robotiq gripper force · MERL impedance control · Wrench-aware admittance
Q4 · Cup-pyramid benchmarks — what actually made them work
  • Data scale beats cleverness for learned policies: AIRoA-0's 80% = 17.9 h of cup demos on 16,151 h of pre-training. DexImit's 52% = one video per task reconstructed into 100 augmented demos. Much of the 52→80 gap is that data gap.
  • Contact sensing is the difference-maker: CHEF 43%→78% with F/T observations; ViTacFormer 0/10→4/10 by predicting touch. The last centimetre is underdetermined by vision alone.
  • Speed kills unless timing is learned: naive speed-up 0.40 success vs. learned retiming 29/30. Slow near contact, fast in transit.
  • Expect a ~50% geometric baseline: RGB-Stacking's scripted baseline is 51%. Budget for verification + recovery to close the rest.

Full numbers, tasks, and caveats are in the benchmark table above.

Q5 · Recovery — nudge, re-place, or rebuild?

Recova (NVIDIA) treats recovery as a distinct objective with three skill roles — restorative (return the scene to a workable state), enabling (prepare the next step), adaptive (change the approach instead of repeating the failure). Recovery skills raised real-robot mean success from 77.5% to 87.5%, and human intervention fell from 87.5% to 0% over four collection rounds. Key line: “Simply retrying the task policy often leaves the underlying obstruction unresolved.” Loop: monitor → detect → recover → verify restoration → resume.

  • Minimal restoration (CoRe): plan recovery virtually from a recent viable state, physically restore only what's needed — up to +85 percentage points success, 42.2% fewer physical restorations; placement verified by re-observation, one retry.
  • Re-plan to the last correct state (neuro-symbolic): a scene-graph discriminator detects and localises failures; recovery re-plans to the last correct state. Motivating example: a collision collapses the tower → re-stack from the current state.
  • Backtracking (Lerner): if the placement loop won't converge, remove the last object from the tower using its stored pose and try a new sequence — backtracking further down if needed.

The leaning-cup ladder — the report is explicit: no paper states a definitive rule. This ladder is its synthesis of the pattern across sources, flagged as such:

  • Small lean, topmost cup, within margin → nudge (template: press dz = −0.02 m), re-verify. Only if it cannot cascade.
  • Leaning topmost cup, at/past margin edge → remove and re-place from a fresh seat computation.
  • Leaning lower cup → remove everything above it first, then re-place it. Never pull a load-bearing cup out from under a pyramid.
  • Full collapse → re-sense all poses, rebuild from the current state.
  • Never leave a leaning cup in place and continue building on it.
Sources: Recova, arXiv 2610.01178 · Neuro-symbolic recovery · CoRe · Lerner et al.
Pin this above the bench

Numbers to set

Every value the report supports, with its status. “Recommendation” means the report proposes it from the evidence — it is not a quoted figure. Don't promote a recommendation to a citation.

Placement spec — from the literature

Final-descent speed cap≤ 7 cm/s Published
Speed in the last centimetre1–2 cm/s Recommendation
Release height (drop)0 — at contact Std. practice
Approach / retreat segments (MTC)10 cm, straight Published
Stability credited only after TCP clears≥ 10 cm Published
Settle before judging a reading0.1 s wait + 0.5 s avg Published
Vision settle check after release1–2 s Recommendation
Disturbance abort budget20 N Published
Corrective nudge (topmost only)≤ 2 cm press Published
Post-release verification failuresRetry ×1 Published
Do not guess these

What the research could not verify

Open questions — stated plainly, per the report