Introducing TacK-JEPA: JEPA Model for Robot Tactility

Try it — step through real grasp episodes

Per-taxel contact force, computed by the actual pipeline

Playing automatically — every frame is rendered directly from the model's contact-sensing pipeline on a real simulated grasp. Drag the slider to take control.

Episode 1 Episode 2 Episode 3 Episode 4 Episode 5

contact forming auto-playing

Grasp force at this step

— N

Active taxels at this step

—

Simulation step

—

Never seen during training

The hand modeled here is based on the Allegro Hand, which has an opposable thumb. Each black point is one of its tactile sensors, plotted at its exact 3D position — computed from the hand's joint angles, not estimated. The cyan dot marks the object being grasped. As the hand closes around it, the sensors that make contact light up on a black-to-yellow scale showing how much force each one is reading at that instant — brighter means more force.

System, in numbers

Built for a different class of tactile hardware

Most tactile sensing today comes from one or two optical sensors (like GelSight or DIGIT) mounted at a fingertip, imaged like a tiny camera. TacK-JEPA is built for a structurally different kind of hardware: an articulated hand covered in a distributed array of independent force-sensing elements ("taxels") spread across every finger and the palm, each tied to a joint-tracked link — the sensing approach used by devices like OSMO's open-source tactile glove. Because every taxel's position is pinned to a joint encoder reading, its exact 3D location is known outright rather than inferred from an image, which is the property this model is built to exploit directly.

What does kinematic grounding offer?

Controlled comparison against an otherwise-identical model with the exact 3D taxel geometry removed, evaluated on grasps the model never trained on.