Emre Ugur

Ph.D.

Special Topics in CMPE: Robot Learning (CMPE58Y)
2018-2019 Spring


Instructor: Emre Ugur (contact)
Textbook: No textbook, but please see the list of readings below.
Lectures: Monday 12:00-15:00
Location: Computer Engineering Dept, A5
Mailing-list: Send email if not automatically registered.
Note: Machine Learning and Robotics background is desired but not a must. Send cconsent request.

Grading:

  • 50%: Homeworks (Coding: GMM, HMM, Q-Learning, Policy gradient, Affordance learning):
  • 25%: Final Exam
  • 20%: Project
  • 5%: Presentation


Aim:
This course is about (i) "general" approaches that aim development of robot intelligence, and (ii) "more focused" advanced learning methods that endow robots with a particular set of sensorimotor skills. To get a grasp of general approaches, we study the general framework of developmental robotics, active learning and intrinsic motivation, bottom-up skill development and symbol acquisition. These approaches are formulated mostly in an interdisciplinary manner, in relation to the findings from infant development, human information processing, experimental and ecological psychology. For the latter one, we will focus on particular methods that are effective in learning manipulation skills and sensorimotor representations such as learning by demonstration, grasp learning, and probabilistic modeling. Rather than detailed analysis of the Machine Learning methods, we will focus on their exploitation for different robot learning problems.


Schedule:
Introduction To Robotics
Reinforcement Learning, Tabular Q Learning,
Reinforcement Learning, Function Approximation,
Policy Gradient RL, Causality, Deep RL BootCamp,
Learning from Demonstration
Dynamic Movement Primitives
LfD - GMM
Jacobian Concept
Developmental Robotics
Affordance Learning,
Intrinsic Motivation,
Data for HW2,
IM cont'd, signal->symbol,
Learning from Demonstration, DMP,
DMP cont'd, probabilistic methods in LfD,

Expected Outcome:
  • Breath: Through introductory material presented by the instructor, and paper presentations by students: An overview of the state-of-the-art methods in robot learning, particularly in developmental robotics, learning by demonstration, policy search methods and reinforcement learning, probabilistic modeling of sensorimotor experience, grasp synthesis algorithms.
  • Depth: Through implementing and extending many of machine learning methods in robotic problems and one of the advanced methods/papers in the final project, an in-depth knowledge on the corresponding method and the robot learning task.

Final Project: Students might suggest implementing a completely novel approach or choose replicating and extending one high-impact paper. A final project report written in conference/workshop paper format is expected. The topics are not limited to the ones discussed in the lectures as long as they are related to robot learning.
Tentative reading material (to be extended)
Developmental robotics:
  • Asada, M., MacDorman, K. F., Ishiguro, H., and Kuniyoshi, Y. (2001). Cognitive developmental robotics as a new paradigm for the design of humanoid robots. Robotics and Autonomous Systems, 37(2), 185-193.
  • Weng, J., McClelland, J., Pentland, A., Sporns, O., Stockman, I., Sur, M., and Thelen, E. (2001). Autonomous mental development by robots and animals. Science, 291(5504), 599-600.
  • Zlatev, J., and Balkenius, C., 2001, Introduction: why epigenetic robotics? In C. Balkenius, J. Zlatev, H. Kozima, K. Dautenhahn and C. Breazeal (eds), Proceedings of the First International Workshop on Epigenetic Robotics, Vol. 85, Lund University Cognitive Studies, pp. 1-4.
  • Ugur, E., Dogar, M. R., Cakmak, M., and Sahin, E. (2007, July). Curiosity-driven learning of traversability affordance on a mobile robot. In Development and Learning, 2007. ICDL 2007. IEEE 6th International Conference on (pp. 13-18). IEEE.
  • Max Lungarella, Giorgio Metta, Rolf Pfeiffer and Giulio Sandini, Developmental robotics: a survey. Connection Science 15(4): 151-190, 2003.
  • M. Asada, K. Hosoda, Y. Kuniyoshi, H. Ishiguro, T. Inui, Y. Yoshikawa, M. Ogino, and C. Yoshida. Cognitive developmental robotics: a survey. IEEE Trans. Autonomous Mental Development 1(1): 12-34, 2009.
  • Chapters 1-4, Cangelosi, A. and Schlesinger, M. Developmental robotics: From babies to robots. MIT Press. 2015
  • Pierce, David, and Benjamin J. Kuipers. "Map learning with uninterpreted sensors and effectors." Artificial Intelligence 92.1 (1997): 169-227.
  • P-Y. Oudeyer, and F. Kaplan. "What is intrinsic motivation? a typology of computational approaches." Frontiers in neurorobotics 1 (2007).
  • P.-Y. Oudeyer, F. Kaplan, and V.V. Hafner. "Intrinsic motivation systems for autonomous mental development." Evolutionary Computation, IEEE Transactions on 11.2 (2007): 265-286.

Affordances:
  • L. Jamone, E, Ugur, A, Cangelosi, L. Fadiga, A. Bernardino, J. Piater and J. Santos-Victor, Affordances in psychology, neuroscience, and robotics: a survey, IEEE Transactions on Cognitive and Developmental Systems, under review.
  • E. Ugur, E. Oztop and E. Sahin, Goal emulation and planning in perceptual space using learned affordances, Robotics and Autonomous Systems, 59 (7-8), pp. 580-595, 2011.
  • E. Ugur and E. Sahin, Traversability: A case study for learning and perceiving affordances in robots, Adaptive Behavior, 18(3-4), pp. 258-284, 2010.

Reinforcement Learning:
  • Konidaris, George, Scott Kuindersma, Roderic A. Grupen, and Andrew G. Barto. "Autonomous Skill Acquisition on a Mobile Manipulator." In AAAI. 2011.
  • Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." Nature 518.7540 (2015): 529-533.
  • Kober, Jens, J. Andrew Bagnell, and Jan Peters. "Reinforcement learning in robotics: A survey." The International Journal of Robotics Research (2013)

Deep learning
  • Lenz, Ian, Honglak Lee, and Ashutosh Saxena. "Deep learning for detecting robotic grasps." The International Journal of Robotics Research 34.4-5 (2015): 705-724.
  • Sigaud, Olivier, and Alain Droniou. "Towards Deep Developmental Learning.", IEEE Transactions on Cognitive and Developmental Systems (2016).
  • Arunkumar Byravan and Dieter Fox, "SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks", IEEE International Conference on Robotics and Automation (ICRA), 2017
  • Levine, S., Pastor, P., Krizhevsky, A., Quillen, D. (2016). Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. arXiv preprint arXiv:1603.02199.
  • Noda, K., Arie, H., Suga, Y., Ogata, T. (2014). Multimodal integration learning of robot behavior using deep neural networks. Robotics and Autonomous Systems, 62(6), 721-736.

Policy search / Reinforcement Learning:
  • Kober, J., Oztop, E., & Peters, J. (2011, July). Reinforcement learning to adjust robot movements to new situations. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence (Vol. 22, No. 3, p. 2650).
  • Peter Pastor et al. "Skill learning and task outcome prediction for manipulation." Robotics and Automation (ICRA), 2011 IEEE International Conference on. IEEE, 2011.
Inverse Reinforcement Learning:
  • Ng, A. Y., Russell, S. J., et al. (2000). Algorithms for inverse reinforcement learning. In Icml, pages 663–670.
  • Abbeel, P., & Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning (p. 1). ACM.

Learning from Demonstration:
  • Argall, Brenna D., et al. "A survey of robot learning from demonstration." Robotics and autonomous systems 57.5 (2009): 469-483.
  • Pastor, P., Hoffmann, H., Asfour, T., & Schaal, S. (2009, May). Learning and generalization of motor skills by learning from demonstration. In Robotics and Automation, 2009. ICRA'09. IEEE International Conference on (pp. 763-768). IEEE.
  • Calinon, Sylvain, Florent Guenter, and Aude Billard. "On learning, representing, and generalizing a task in a humanoid robot." Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on 37.2 (2007): 286-298.

Grasping:
  • Miller, A. T., & Allen, P. K. (1999). Examples of 3D grasp quality computations. In Robotics and Automation, 1999. Proceedings. 1999 IEEE International Conference on (Vol. 2, pp. 1240-1246). IEEE.
  • Kao, I., Lynch, K., & Burdick, J. W. (2008). Contact modeling and manipulation. In Springer Handbook of Robotics (pp. 647-669). Springer Berlin Heidelberg.
  • Prattichizzo, D., & Trinkle, J. C. (2008). Grasping. In Springer handbook of robotics (pp. 671-700). Springer Berlin Heidelberg.
  • Roa, M. A., & Suárez, R. (2015). Grasp quality measures: review and performance. Autonomous robots, 38(1), 65-88.

Grasp learning:
  • Coelho Jr, J. A., & Grupen, R. (1997). A control basis for learning multifingered grasps
  • Detry, R., Ek, C. H., Madry, M., Piater, J., & Kragic, D. (2012, May). Generalizing grasps across partly similar objects. In Robotics and Automation (ICRA), 2012 IEEE International Conference on (pp. 3791-3797). IEEE.
  • Detry, R., Kraft, D., Kroemer, O., Bodenhagen, L., Peters, J., Krüger, N., & Piater, J. (2011). Learning grasp affordance densities. Paladyn, Journal of Behavioral Robotics, 2(1), 1-17.

Probabilistic Approaches:
  • Koppula, H. S., Gupta, R., & Saxena, A. (2013). Learning human activities and object affordances from rgb-d videos. The International Journal of Robotics Research, 32(8), 951-970.
  • T. Nakamura, T. Nagai, and N. Iwahashi. "Grounding of word meanings in multimodal concepts using LDA." In Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on, pp. 3943-3948. IEEE, 2009.
  • Celikkanat, H., Orhan, G., & Kalkan, S. (2015). A probabilistic concept web on a humanoid robot. Autonomous Mental Development, IEEE Transactions on, 7(2), 92-106.


Human-in-the-loop learning:
  • Babič, Jan, Joshua G. Hale, and Erhan Oztop. "Human sensorimotor learning for humanoid robot skill synthesis." Adaptive Behavior (2011).
  • Peternel, L., Petrič, T., Oztop, E., & Babič, J. (2014). Teaching robots to cooperate with humans in dynamic manipulation tasks based on multi-modal human-in-the-loop approach. Autonomous robots, 36(1-2), 123-136.

Symbol/skill acquisition:
  • Mugan, Jonathan, and Benjamin Kuipers. "Autonomous learning of high-level states and actions in continuous environments." Autonomous Mental Development, IEEE Transactions on (2012): 70-86.
  • Konidaris, G., Kuindersma, S., Grupen, R. A., Barto, A. G. (2011, April). Autonomous Skill Acquisition on a Mobile Manipulator. In AAAI.
  • E. Ugur and J. Piater, Bottom-Up Learning of Object Categories, Action Effects and Logical Rules: From Continuous Manipulative Exploration to Symbolic Planning, IEEE International Conference on Robotics and Automation (ICRA), Seattle, Washington, 2015, pp. 2627-2633
  • Konidaris, L. P. Kaelbling, and T. Lozano-Perez, “Constructing symbolic representations for high-level planning,” in 28th AAAI Conf. , 2014.
  • Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, and Hideki Asoh, Symbol Emergence in Robotics: A Survey, Advanced Robotics, Vol.30, (11-12) pp. 706-728 .(2016)