Daily AI Paper

Dexterous robotic hands are one of the most exciting frontiers in robotics because they promise something closer to human manipulation: not just picking things up, but moving them precisely inside the hand, reorienting them, and handling awkward objects in tight spaces. But that same richness makes them hard to control. There are many joints, many possible contact points, and very little margin for error. If the hand slips, the object can fall. If the wrist turns, the whole problem changes. And unlike simpler robot tasks, safety is not a side issue here. When a robot is handling a fragile or hazardous object, the difference between a successful maneuver and a drop matters a lot. This paper is about pushing that boundary. The task the authors study is called in-hand object pose reaching. In plain terms, the robot has to move an object to a desired position and orientation relative to the hand, and it has to do this not just once, but over long sequences of goals, without resetting between them. That is much harder than the more common in-hand rotation tasks you often see in the literature, where the object is just spun around a single axis, usually with the palm supporting it from below. Here, the robot must also translate the object within the hand, which means it often has to hold it at the fingertips, in awkward configurations, sometimes with no palm support at all. The authors also want the hand to work at arbitrary wrist orientations, not just one convenient pose like palm-up. That makes the task a good stress test for dexterity, robustness, and safety all at once. To understand why this matters, it helps to look at the landscape of prior work. Reinforcement learning and imitation learning have produced impressive dexterous controllers in simulation, and there are real-robot demonstrations of in-hand reorientation, grasping, and tool use. But most of those systems are still narrow. They often assume a single wrist orientation, they focus on repetitive motions, and they tend to report success over short trials. Reliability over long runs, and explicit safety metrics like how often the object is dropped, are often missing. That is a problem if we want dexterous hands to leave the lab. In the real world, a robot may need to keep running for a long time, deal with unexpected disturbances, and avoid dropping something that could break, spill, or pose a hazard. The authors’ core idea is to treat this as both a control problem and a safety problem. They build a full sim-to-real training recipe for a 16-degree-of-freedom LEAP hand, trained entirely without human data. The recipe combines several pieces. First, they use large generated grasp datasets to create diverse starting states and goals. That matters because if the policy only ever sees easy, similar configurations, it won’t learn the broad range of grasps needed for real dexterity. Second, they introduce an adaptive friction curriculum. Friction is a huge factor in in-hand manipulation: too much or too little can completely change whether the object sticks, slides, or slips. By varying friction during training in a structured way, they help the policy gradually become robust to harder contact conditions. Third, they keep the reward relatively simple: a few dense terms that encourage the object to reach the goal, plus regularizers. The emphasis is not on a complicated reward design, but on making the training distribution and the simulation itself realistic enough for transfer. A major practical ingredient is system identification. Rather than simulating each servo as an idealized position controller, they fit a more realistic motor model to data from a single-servo test bench, and then use that model for all 16 actuators in the hand. They also calibrate latency by matching real and simulated trajectories, including delays in joint readings, commands, and the wrist camera. This is important because dexterous manipulation is sensitive to timing. If the simulation is too optimistic or too responsive compared with the real hand, policies can fail badly when transferred. With these identified actuator dynamics and delays, the simulated joint motion and depth images track the real system much more closely. The hardware setup itself is fairly practical. The hand is mounted on a UR5 arm, but the policy controls only the hand, not the arm. The arm mainly sets the wrist orientation, which changes gravity relative to the hand. The hand uses hard 3D-printed fingertips covered with high-friction tape rather than soft fingertips, which is notable because some prior work has relied on softer materials to make manipulation easier. The objects include a lightbulb, a wineglass, a knife, and a rubber duck, with some of them highly nonconvex or otherwise awkward. For state-based training and evaluation, the authors use motion capture to get the object’s 6D pose. For the vision-based version, they use a single wrist-mounted stereo depth camera looking at the palm and fingers, and they compute depth from stereo images rather than the camera’s own depth output because it is cleaner. That means they can run a policy without external cameras or motion capture, which is much more portable. The task formulation is carefully defined. A goal can be a full 6D pose, or, for rotationally symmetric objects like the lightbulb and wineglass, it can be a position plus a target axis direction. That makes sense because for symmetric objects, rotation around the object’s own axis may not matter or may not even be observable from vision. A goal counts as reached when both the position error and the orientation error are within tolerance. They also define safety very concretely: the robot should not drop the object. In practice, if the object moves too far from the hand root, beyond a threshold that is well outside any valid goal, it is considered dropped. This makes safety measurable instead of vague. On top of the task policy, the authors add a learned residual safety filter. This is adapted from prior work called ResSafe, but here it is used for dexterous manipulation. The idea is simple and appealing: keep the main policy fixed, and train a second module that makes corrective changes to the action when the system is drifting toward unsafe behavior. Because it is residual, it does not replace the policy; it nudges it. And because it has a single correction weight, the user can tune how conservative it is at deployment. More conservative means more protection against drops, but potentially less task performance. That tradeoff is exactly what you want if you care about both capability and safety. The evaluation is extensive by dexterous manipulation standards. They report 13.3 hours of real-robot runtime over 4,800 unseen goals, across four objects and six wrist orientations, with long runs of 50 consecutive goals. The state-based policy, which uses motion-capture object pose, reaches 95.2 percent of goals within 2 cm and 0.4 rad. When the safety filter is added, drops are reduced by 75 percent, from 16 to 4, while still reaching 91.5 percent of goals. That is a very strong result because it shows the filter does not just make the robot safer in a trivial way by freezing it; it preserves most of the task performance while substantially reducing failures. The depth-based policy is harder, as expected, because it has to infer object state from a single wrist camera. Even so, it still reaches 72 percent of goals overall, and does especially well on the lightbulb and rubber duck. The safety filter also helps here, nearly halving the number of drops. That is important because it shows the approach is not limited to privileged state input. It can work as a portable skill that only needs onboard sensing. The paper also includes several qualitative stress tests that help interpret the numbers. The system can spin a lightbulb held in difficult, arbitrary poses. It can reach goals that are out of distribution. It can withstand fast arm motion and physical perturbations. And because the depth policy does not depend on motion capture, it can be used in longer, more practical arm-and-hand tasks, such as screwing in a lightbulb until it lights up. These demonstrations matter because they show the policy is not just overfit to a benchmark. It can support actual manipulation sequences that look like useful robot behavior. There are still limitations, though. The best results depend on careful system identification and a well-designed simulation pipeline, so this is not a plug-and-play recipe for any hand. The state policy relies on motion capture, which is excellent for research evaluation but not always available in deployment. The depth policy removes that dependency, but with a noticeable performance drop. The safety filter improves reliability, but it is still learned and does not come with formal guarantees in the way that some control-barrier or Hamilton-Jacobi methods aim to. And while the paper stresses diverse wrist orientations and long goal sequences, the evaluation is still on a specific hand, a specific set of objects, and a specific lab setup. Even with those caveats, the main message is strong. The paper shows that in-hand object pose reaching can be pushed much farther than the usual rotation-only benchmarks, and that safety can be treated as a first-class objective rather than an afterthought. The combination of diverse simulation training, realistic actuator modeling, a vision-based deployment path, and a learned safety filter gives a surprisingly capable system. More broadly, the work provides a concrete platform for studying not just whether dexterous policies work, but how reliably and safely they work when the task gets genuinely hard.