Introducing TacK-JEPA: JEPA Model for Robot Tactility

TacK-JEPA gives a robot hand model the exact 3D position of every one of its tactile sensors, computed directly from joint geometry — no vision model required to guess where a touch happened.

System, in numbers

1e-5
forward-kinematics agreement with the underlying physics solver
53 / 53
unit tests passing — kinematics, force conservation, model components
18–25×
more embedding variance retained than architecture variants without the full training objective

Built for a different class of tactile hardware

Most tactile sensing today comes from one or two optical sensors (like GelSight or DIGIT) mounted at a fingertip, imaged like a tiny camera. TacK-JEPA is built for a structurally different kind of hardware: an articulated hand covered in a distributed array of independent force-sensing elements ("taxels") spread across every finger and the palm, each tied to a joint-tracked link — the sensing approach used by devices like OSMO's open-source tactile glove. Because every taxel's position is pinned to a joint encoder reading, its exact 3D location is known outright rather than inferred from an image, which is the property this model is built to exploit directly.

What does kinematic grounding offer?

Controlled comparison against an otherwise-identical model with the exact 3D taxel geometry removed, evaluated on grasps the model never trained on.

Per-taxel force estimation

31% lower error
than the same model without kinematic grounding.

Contact area estimation

3.2× lower error
than the same model without kinematic grounding.

Try it — step through real grasp episodes

Per-taxel contact force, computed by the actual pipeline

Playing automatically — every frame is rendered directly from the model's contact-sensing pipeline on a real simulated grasp. Drag the slider to take control.

Per-taxel force heatmap
contact forming auto-playing
Grasp force at this step
— N
Active taxels at this step
—
Simulation step
—
Never seen during training

The hand modeled here is based on the Allegro Hand, which has an opposable thumb. Each black point is one of its tactile sensors, plotted at its exact 3D position — computed from the hand's joint angles, not estimated. The cyan dot marks the object being grasped. As the hand closes around it, the sensors that make contact light up on a black-to-yellow scale showing how much force each one is reading at that instant — brighter means more force.