It is imperative that robots can understand natural language commands issued by humans. Such commands typically contain verbs that signify what action should be performed on a given object and that are applicable to many objects. We propose a method for generalizing manipulation skills to novel objects using verbs. Our method learns a probabilistic classifier that determines whether a given object trajectory can be described by a specific verb. We show that this classifier accurately generalizes to novel object categories with an average accuracy of 76.69% across 13 object categories and 14 verbs. We then perform policy search over the object kinematics to find an object trajectory that maximizes classifier prediction for a given verb. Our method allows a robot to generate a trajectory for a novel object based on a verb, which can then be used as input to a motion planner. We show that our model can generate trajectories that are usable for executing five verb commands applied to novel instances of two different object categories on a real robot.
Dataset generated and used for this paper is available upon request.
Please contact the author at rachelm8@mit.edu.
@article{ma2023skill,
title={{Skill Generalization with Verbs}},
author={Ma, Rachel and Lam, Lyndon and Spiegel, Benjamin A. and Ganeshan, Aditya and Patel,
Roma and Abbatematteo, Ben and Paulius, David and Tellex, Stefanie and Konidaris, George},
journal={2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year={2023},
}