Key words : C++, OpenCV, Point Cloud, ROS, Artifical Vision, Machine learning algorithm
Introduction
As robot enter novel environments, they are able to navigate these environments successfully and autonomously. However to be efficient, the robot should be able to manipulate their environment to gain access to new spaces, such as by opening doors. This remains a challenging and actual problem because a robot will likely encounter doors and handles it has never seen before.
The main part of this work is to develop a versatile algorithm which will be able to detect a door, detect the door handle and classify the handle type. In order to achieve this goal, several tools will be used. All the programs and algorithm are developped on the ROS environment in C++. We will also use point cloud computation and filtering, image analysis with OpenCV as well as machine learning algorithms.
 |
| REEM Robot http://pal-robotics.com/ |
Problematic
- How to recognize door handles ?
- Door handles have different shapes, colors...
- How to be sure the object is a handle ?
 |
| Different types of handles |
First step : Detect the position of the handle
- ROS + Point Cloud Library
- Camera on the robot's head
- A : We run several transformation on the original point cloud (downsampling, base transformation, passthrough...)
- B : Thanks to a plane segmentation we know the points that belong to the door plane
- C : With several functions we can isolate the points that are in front of the door plane, among these points, there are the points that represent the handle
At this step we have a list of points. Next an Euclidian cluster function is applied in order to identify objects.
We obtain a list of objects. Then we sort these objects (size, height...) and we get a list of "potential handles". We represent them with a colored point :
Second step : Contruction of a ROS topic
In order to communicate with the classification node, we build a ROS message that contains the handle image (and other information for future actions).
Knowing the position of the handle, we can extract a cropped image that represents the handle. After several transformations we add the image(s) to the message and publish it.
The transformations are made to simplify the image. It improves the classification. Examples of transformations :
 |
| Normal color images, bad results |
 |
| Binary images, good results |
 |
| Depth image, good results |
Third step : Classification
This node subscribes to the topic published by the detector and classifies the handle's image. The output of the node is the type of the handle :
- Regular handle
- Knob handle
- Not a handle
What is a classifier ?
Machine learning, a branch of artificial intelligence, is about the construction and study of systems that can learn from data. For example, a machine learning system could be trained on email messages to learn to distinguish between spam and non-spam messages. After learning, it can then be used to classify new email messages into spam and non-spam folders.
The core of machine learning deals with representation and generalization. Representation of data instances and functions evaluated on these instances are part of all machine learning systems. Generalization is the property that the system will perform well on unseen data instances; the conditions under which this can be guaranteed are a key object of study in the subfield of computational learning theory.
There is a wide variety of machine learning tasks and successful applications. Optical character recognition, in which printed characters are recognized automatically based on previous examples, is a classic example of machine learning.
Machine learning algorithms can be organized into a taxonomy based on the desired outcome of the
algorithm or the type of input available during training the machine.
- Supervised learning generates a function that maps inputs to desired outputs (also called labels, because they are often provided by human experts labeling the training examples). For example, in a classification problem, the learner approximates a function mapping a vector into classes by looking at input-output examples of the function.
- Unsupervised learning models a set of inputs, like clustering. See also data mining and knowledge discovery. Here, labels are not known during training.
- Semi-supervised learning combines both labeled and unlabeled examples to generate an appropriate function or classifier. Transduction, or transductive inference, tries to predict new outputs on specific and fixed (test) cases from observed, specific (training) cases.
- Reinforcement learning learns how to act given an observation of the world. Every action has some impact in the environment, and the environment provides feedback in the form of rewards that guides the learning algorithm.
- ....
Image classification steps
There are three main steps to run this kind of classifier.
- The first step is to create a huge database. Thousands of positives images (images representing the object we want to detect) and thousands of negatives images (images representing everything but the object) are collected.
- The second step is the features extraction (for images), these features will be trained, this is the second step. There are a lot of image features. Basically features describe images. Some of them are dedicated to machine learning and other are more common (like corners).
- The final step is the Training. In that step the images (in our case) are trained. We tried a lot of different features and training methods to obtain the best results with our project. A big work was to write a node containing functions and method that allowed us to run several features extraction and training methods with different configurations, for a given training set in order to select the best one.
Examples of image features used with machine learning :
 |
| HAAR |
 |
| HOG |
Summary :
ROS : http://wiki.ros.org/
PCL : http://pointclouds.org/
OpenCV : http://opencv.org/