Robotic Creatures

Cloud robotics is getting more and more attention recently. See for example the Google initiative presented at I/O 2011, or the RoboEarth project funded by the European Comission. The idea is to let robots share information on the cloud with the goal of generating a global knowledge base and ultimately accelerating robotic development.

We recently presented rostweet, a ROS package that lets your ROS-based robots post text to a twitter account. We now extend it and add support for posting pictures, by using the recent post_with_media twitter API. This enables the sharing of images among ROS-based robots through twitter, which could have interesting applications in ‘social’ object recognition, learning, or just communication among robots and between robots and humans.

This is how it works: by running rostweet, a ROS service is created that your node can call for posting a tweet with some text (string) and an optional image (sensor_msgs/Image). In addition, incoming tweets are published on a topic as they arrive, also in the format of string + sensor_msgs/Image. Just create a twitter account for your robot and start following other robots (or humans!). For now, rostweet only supports the image media type, although support for other types of media can be easily added.

Please check our first post for installation instructions, although there are some changes in this last version:

• Posting a tweet is now done though a service call. An example node called ‘post’ has been included in the package. Use it as rosrun rostweet post "<text>" [<path_to_picture>]
• The rostweet_msgs/IncomingTweet message now includes also an image. To see just the text of the incoming tweets, do rostopic echo /rostweet/incomingTweet/tweet

Hi, for those who may need it, this is a simple C++ class that loads a point cloud file (PCD format of the Point Clouds Library PCL) and creates an OpenSceneGraph geode that you can easily add to your OSG scene.

#ifndef OSGPCDLOADER_H_
#define OSGPCDLOADER_H_

#include <pcl/io/pcd_io.h>
#include <pcl/point_types.h>

#include <osg/Geometry>
#include <osg/Geode>

class osgPCDLoader {
public:
  pcl::PointCloud<pcl::PointXYZRGB> cloud;
  osg::ref_ptr<osg::Geode> geode;

  osgPCDLoader(std::string pcd_file) {
   if (pcl::io::loadPCDFile<pcl::PointXYZRGB> (pcd_file, cloud) == -1)  // load the file
   {
     std::cerr << "Couldn't read file " << pcd_file << std::endl;
   } else {
     std::cout << "Loaded " << cloud.width * cloud.height << " data points from " << pcd_file << std::endl;

     geode=osg::ref_ptr<osg::Geode>(new osg::Geode());
     osg::ref_ptr<osg::Geometry> geometry (new osg::Geometry());

     osg::ref_ptr<osg::Vec3Array> vertices (new osg::Vec3Array());
     osg::ref_ptr<osg::Vec4Array> colors (new osg::Vec4Array());

     for (int i=0; i<cloud.points.size(); i++) {
       vertices->push_back (osg::Vec3 (cloud.points[i].x, cloud.points[i].y, cloud.points[i].z));
       uint32_t rgb_val_;
       memcpy(&rgb_val_, &(cloud.points[i].rgb), sizeof(uint32_t));

       uint32_t red,green,blue;
       blue=rgb_val_ & 0x000000ff;
       rgb_val_ = rgb_val_ >> 8;
       green=rgb_val_ & 0x000000ff;
       rgb_val_ = rgb_val_ >> 8;
       red=rgb_val_ & 0x000000ff;

       colors->push_back (osg::Vec4f ((float)red/255.0f, (float)green/255.0f, (float)blue/255.0f,1.0f));
     }

     geometry->setVertexArray (vertices.get());
     geometry->setColorArray (colors.get());
     geometry->setColorBinding(osg::Geometry::BIND_PER_VERTEX);

     geometry->addPrimitiveSet(new osg::DrawArrays(osg::PrimitiveSet::POINTS,0,vertices->size()));

     geode->addDrawable (geometry.get());
     osg::StateSet* state = geometry->getOrCreateStateSet();
     state->setMode( GL_LIGHTING,osg::StateAttribute::OFF);
    }
  }

  ~osgPCDLoader(){}

};
#endif

This is an example of a point cloud loaded into UWSim:

Just for fun & profit I  have attached a head-massaging ‘tool’ to a mechanical structure with one servo-controlled joint. The servo is plugged to an arduino board that is integrated in ROS via the rosserial stack. On the other hand, a mobile phone running android executes a very simple app consisting of just a SeekBar. The value of the seekbar is published to the ROS network thanks to rosjava, and therefore the actuated joint is moved accordingly. The perfect gift for these christmas

The following video shows an Android application sending position references to a simulated robotic arm by using rosjava. Rosjava is a java port of ROS (Robot Operating System). It allows to run ROS nodes inside a java application, e.g. typical Android apps. The application running on the phone just displays 5 SeekBars, each one corresponding to one joint of a robotic arm (CSIP ARM5E in this case). Each time a bar thumb is moved, a ROS JointState message is published by the ROS node running on the phone, and received by other nodes that are subscribed to the topic. In this case, our underwater simulator UWSim was receiving the JointState messages and updating the simulated arm accordingly. The same could be done with the real arm that accepts the same JointState interface. More to come!

Rostweet is a twitter client for ROS. It allows your robots to send tweets to its followers, and to receive tweets from its following contacts. Rostweet can be applied in many different ways, these are some examples:

• Communicating progress. Robots running ROS can use rostweet for reporting feedback: task progress, failure detection, etc.
• Sharing new knowledge. After acquiring new information, a robot can use rostweet in order to share it with its followers, e.g. how to grasp an object, a new environment map, etc.
• Acquiring new knowledge. By listening to the others’ tweets, a new source of incoming information is created and can be used for enlarging the robot knowledge.
• Just fun! Let your robot read your tweets, etc.

In general, rostweet can be useful for instantaneous and decentralized robot-robot, robot-human and human-robot sharing of information.

Rostweet is delivered as a ROS stack. It can be downloaded from the uji-ros-pkg google code project repository:

svn co http://uji-ros-pkg.googlecode.com/svn/trunk/rostweet

After downloading and placing it in a folder inside your ROS_PACKAGE_PATH, it should compile with:

rosmake rostweet

and run with:

rosrun rostweet rostweet

This will ask for a twitter username and password and start the node afterwards (by running the node for the first time, you implicitly accept rostweet as a twitter application in your account, and allow it to read your timeline and publish tweets). Once the node is running, other nodes can subscribe to incoming tweets at /rostweet/incomingTweet (of type rostweet_msgs/IncomingTweet), and publish tweets by calling to the service /rostweet/postTweet (of type rostweet_msgs/postTweet). There is an example node for posting e.g.:

View incoming tweets:

rostopic echo /rostweet/incomingTweet/tweet

Send a tweet:

rosrun rostweet post "Hello world from rostweet"

Without any parameters, each time rostweet is executed, it will ask for username, password, and will get the OAuth tokens. If you want to skip some of these steps, you can set them with the following parameters via rosparam:

After the first execution, the username and token keys are stored in a file inside the launch/ folder in YAML format. With it you can easily write a launch file in order to set the parameters automatically:

<launch>
<node name="rostweet" pkg="rostweet" type="rostweet" output="screen">
<rosparam command="load" file="$(find rostweet)/launch/paramFile.yaml" />
</node>
</launch>

Please check also our more recent post about tweeting with pictures.
For support, please post to ROS-Answers.

This video shows a visual servoing application where an I-AUV has to first align with an object before starting a manipulation action. Green crosses represent the perceived corners of the observed object, whereas red crosses show the desired pose. A 2D Visual Servoing control law (using ViSP) is executed in order to minimize the error between the current and the desired projection of the object in the image. The UWSim simulator is used to visualize the execution, and to provide images from a virtual camera, everything running into a ROS middleware.

These are two videos of the autonomous black box recovery experiments carried out some weeks ago at Roses (Girona), using the Girona 500 underwater vehicle, the Lightweight ARM 5E manipulator and stereo vision. The first shows a summary of all the parts and the results obtained. The other was recorded by a local TV.

First the vehicle conducted an optical survey of the seabed. Next an Ortho-photomosaic was built and used for target selection (the black box mockup). Then the vehicle was launched again and searched for the target. When found, it switched to visual station keeping and recovered the target. All the mission was performed autonomously, only an ethernet umbilical was used for testing and monitoring.

Last week we were at Roses (Girona), doing experiments on autonomous underwater intervention in the harbour. The task was to find and recover a Flight Data Recorder (black box) at some 4m depth in the harbour. The only cable connected to the autonomous vehicle was an ethernet cable used for monitoring the task. Inside the vehicle, there were 3 PCs devoted to vehicle control, arm control and visual processing. The robot successfully performed a survey of the area, found the black box, kept its pose visually on top of it and recovered it using a hook.

Our last experiments have focused on using the 3D point clouds for grasp planning and synthesis. The approach is to let a human indicate the most suitable part for grasping, and then automatically plan a grasp on that part. In fact, in some underwater robotic applications such as archeology, the selection of the part where to grab an object is crucial in order to avoid any damages of the item. Hard to say, but leaving such decision to a robot may still be quite risky . Therefore, in our approach it is a human who indicates the most suitable part for grasping, and the robot just plans a grasp around that area. The user input is a pair of antipodal grasp points clicked on a 2D image. The projection of those 2D points define two 3D lines that are used to define a 3D volume (with a fixed width). The complete 3D point cloud is then filtered and only those points lying inside the user-clicked volume are kept. The next step is to remove outliers and to build a downsampled cloud. The highest point (farest to the floor) on that point cloud defines the final grasping point. Below some results on simulation (with UWSim) and with the real robot (the grasp was defined by the user on the rope, because the current size of the hand only permits grasping small objects).

We have finally placed the arm, laser and camera in a water tank, and have performed a 3D scan test of an amphora. The camera was placed on the top, looking downwards, towards the floor. The laser was attached on the forearm of the CSIP ARM5E underwater arm. A complete scan was performed by moving the elbow joint of the arm through a range that covered the whole image. The first captured image was used in order to assign a colour to each 3D point. Results are shown below. Still to do: remove outliers, mesh representation, textures, filling shadows, holes, grasp planning, etc. More to come!