OpenDR

Recent advances in deep learning led to a number of spectacular applications, ranging from self-driving cars, and robots that outperform humans in various tasks, to building algorithms that can understand and answer questions by fusing information from many modalities. The application of deep learning in robotics leads to very specific learning, reasoning and embodiment problems and research questions that are typically not addressed by the computer vision and machine learning communities. Moreover, the available deep learning open frameworks are not easily employed in robotics since they have a steep learning curve and radically different methodology (end-to-end learning, from sensing to acting, etc.) than conventional robotics and thus, relatively few robotics laboratories are including deep learning methodologies to their research. Furthermore, even though deep learning was indeed able to successfully tackle these challenging problems and open an exciting new field of novel and high-impact applications, it comes with an additional significant drawback. Deep learning models are becoming more and more complex, with some recent state-of-the-art techniques using more than 1000 neural layers (e.g., ResNets), requiring vast amounts of computational power and energy both for training and inference purposes. These requirements are becoming especially limiting in many robotics applications, where significant energy and computational power constraints exist, restricting the speed and the accuracy of the deployed models. Additionally, the already available deep learning software modules are implemented in order to be deployed to large and expensive Graphics Processing Units (GPUs) and they rarely perform in real-time even for low-resolution input. Thus, they are usually inappropriate for deployment on embedded systems such as those usually found on robotics systems. The need for an open deep learning toolkit that contains easy to train and deploy real-time, lightweight, Robot Operating System (ROS) compliant deep learning models for robotics is evident.

The core objective of OpenDR is to provide real-time (as defined by the specific use case requirements), lightweight (to be deployed in use-case specified embedded CPUs/GPUs like the NVIDIA TX1/2 and Xavier) and ultralightweight (to be deployed in embedded architectures like Raspberry Pi, FPGAs, microcontrollers, etc.) deep learning architectures for performing inference and control in various robotics tasks. State of the art deep learning architectures will be considered for raw sensor signal (speech, image, video, Lidar, GPS, etc.) analysis that will be trained using supervised, semi-supervised, unsupervised and reinforcement learning both in real data and simulation in order to provide easy to train and deploy models for robotics. A library with a collection of ROS nodes and the necessary tools for training and deployment will be developed that will enable any ROS-based robotic architecture to integrate them for improving its technical capabilities. OpenDR will also build and share open datasets, as well as appropriate simulation environments, for training and evaluating the developed tools on selected robotic applications. Three real use-cases on agri-food, healthcare and agile production will be developed in OpenDR to demonstrate the functionalities of the toolkit.