Diego – the humanoid biped

After proving that the robot’s arm articulation is precise enough and that the vision system works well in finding and tracking a ball, I decided to take it a step further and combine sensing and actuating. Actually a few steps further, but before explaining what are all the steps the robot does, let’s see the video:

The robot performs the following steps in this video:

1. Speech recognition of all commands (SAPI 5.1)
2. Looking for the red ball with it’s stereoscopic cameras.
3. Finding the ball’s image and stereo-matching it using the sum of absolute differences algorithm.
4. Calculating the distance of the ball using triangulation.
5. Calculating the Cartesian (x,y,z) position of the ball using trigonometry.
6. Performing inverse kinematics (IK) to determine joint angles to get to the ball.
7. Actuating the servos to take the ball.

Let’s focus on some of the new algorithms.

The red ball is found in the left image by locating the “most red” area in it. Pure red is found in the image by the following formula calculated for each pixel:

diff(x,y)=red(x,y)-blue(x,y)-green(x,y)

the ‘diff’ signal will have its highest values for areas which have only the red color component. Then, an adjustable threshold is applied to cut off other color areas.

After this is done, we commence with stereo matching. This is done along so called epipolar lines. Since the cameras’ optical axes in this system are parallel, the epipolar lines are horizontal. This means that the ball’s image will appear in the right image at the same y-coordinate as on the left image, but translated along the x axis.  The search along the x is done using the 2D sum of absolute differences between the left and right images. The area in the right image that most resembles the area in the left image is declared stereo-matched.

Next, knowing the camera focus lengths and positions of the center of the ball in both images we can calculate the angles under which the cameras see the object. By also knowing the inter-camera distance, we can triangulate the distance of the object.

After the direct line distance is calculated, trigonometry is used to figure out the Cartesian distances in the x,y and z directions. These are fairly complex calculations, since the head’s tilt and yaw angles must be accounted for too.

Knowing the x,y and z coordinates of the ball compared to the origin of the hand in the shoulder, we need to figure out the angles in the arm’s joints to reach the object. This problem is called inverse kinematics. It is an iterative process in which the angles are adjusted until the hand reaches the ball.

Finally, when the angles are calculated the robot performs these and grabs the ball.

In the below video my robot performs a couple of things. First he recognizes speech. With the limited grammar it works great. Then he visually recognizes objects of the chosen color in its field of view. Finally he turns his head towards where the recognized object is moving.

As mentioned in an earlier post, speech recognition is done using Windows’ Speech API, while generation is done using Windows Text To Speech (TTS).

Object recognition is based on detection of an area with the color given by speech recognition. There is no stereoscopy involved in this case. Video acquisition and image processing is done using the famous OpenCV library. It provides all the necessary routines that make it easy to implement any computer vision idea. Here I separate the given color from the image, find the area’s center and then call servo control functions of the Roboard to turn the robot’s head in order to keep the red or blue ball in the center of the field of view.

Since the new processing unit for my robot, the Roboard RB-100, has USB ports, I can start using standard PC devices to fulfill different robot functions. This is so much easier than writing the device driver code myself. In this light I got hold of a very interesting off-the-shelf PC stereoscopic camera system: the Minoru 3D. I must thank the company, Promotion and Display Technologies, for taking this brave step and marketing such a device. This is how it comes when you buy it:

Minoru 3D consists of two web cameras in one sleek housing. For my purposes, I needed only the electronics, that would allow me to connect two webcams to a single USB port. This is why I stripped down the housing and came up with this:

Finally, here is how it looks like when the camera system is installed on the robot’s head on a pair of pan and tilt servo motors:

I found the Minoru 3D to be a great robotics component. It works very well under Windows and delivers good image quality. Very conveniently, it also contains a very well amplified microphone, which can be very useful in speech recognition applications. As a hint for anyone who might want to use this device in a similar fashion: when trying to take the plastic housing apart, there is no need for breaking it. The whole thing is held together with a concealed screw on the back of the neck of the Minoru housing. In future posts I will talk more about how I used this newly acquired vision system to add to my robot’s sensing capabilities.

Recently I got a couple of miniature on-chip digital cameras: Toshiba TCM8230MD. They can be bought for example on the SparkFun website. These are really small 0.3 megapixel VGA cameras intended for use in mobile phones. This is how the cameras look like installed on Diego:

As I said, they are REALLY small (6mm x 6mm). They are not yet operational. I have two of them installed in order to form a stereoscopic system. I will post more about them when I connect them to a processing unit.