Readme

Starbax

Functionality of package

This package is a collection of scripts and message definitions created to have a Baxter robot prepare a cup of coffee using a single serve keurig, a kcup, and a mug. The objects must be set at a specific height and the coffee maker must face Baxter, but objects can otherwise be placed arbitrarily so long as Baxter can easily reach them.

This package makes use of two features to identify whether the object is the kcup, mug, or the coffee maker and also to find the transform from Baxters base frame to the object. The first feature is a Computer vision algorithm that utilizes the CVBridge package to convert ROS images to the CV images we use to identify series of pink, blue, and yellow post-it-notes placed next to the kcup, cup, and keurig respectively. The other feature is the tracking of AR tags. The AR tags come from the ar_track_alvar package. What this packes does is it will use an external camera to identify a series of AR tags that can be generated using the packages generation function. When an AR tag is detected by the camera the package will begin publishing transforms from a selected base frame to the AR tag. We used these tags to find the transforms from Baxter to the items these transforms will eventually be used in the motion planning program.

Once the tags are identified a different algorithm takes the transformation data produced from the AR tags and matches the position of the tag with the position of the detected color it then determines what AR tag corresponds to which item. When the items and their positions with respect to Baxter are determined the data is sent to a motion planning program.

The motion planning program uses Baxter's in-built inverse kinematics service to move to desired locations. The objective is to approach the objects horizontally and keep them level to avoid spilling their contents, and then places the objects at the desired positions. To open and close the keurig two services are called in the middle of the motion planning program.

In total, there are three main components of this project: Color Segmentation, AR tag Recognition, and Inverse Kinematics to move Baxter's end effector.

Scripts

• The pic_cal_sw.py script launches the node pic_cal that subscribes to Baxter's left hand camera topic /cameras/left_hand_camera/image. While not actually run in the launch file, this node was used to obtain clear images of the blue, yellow, and pink post-it-notes that Baxter used to differentiate between the three different objects: the Keurig, cup, and K-cup.

• The table_cam_sw.py script launches the node table_cam that subscribes to Baxter's left hand camera topic /cameras/left_hand_camera/image. It then loads the three post-it-note images from the media folder and uses histogram-based color segmentation to create three masks - one for each color. Assuming the camera has a clear view of all three post-it-notes, it finds the centroids and compares them to find their relative positions. It then publishes a String message on the pos_items topic listing the objects from left to right as one faces Baxter.

• The starbax_sw.py script launches the node initial that positions Baxter's left arm so that it's camera has a top-down view of the table where the Keurig, cup, and K-cup are located.

• The artransforms.py script will launch the ar_info node and it will subscribe to the ar_pose_marker topic generated by the ar_track_alvar package. This script will then extract the AR tags id number, the x,y,z positions with respect to Baxter's base frame, and orientation data of each of the tags. This data is then republished on the topic ar_pose_id as a custom message type called ar_tag.

• The pose_and_item.py script launches the node pose_item that subscribes to the pos_items and ar_pose_id topics to get the relative positions of the objects and their pose in the world. The incoming pose is of the ar_tag message definition. It then sorts the three poses from least to greatest using their 'y' coordinates. The pose with the smallest 'y' coordinate is then paired with the left most object as one faces Baxter, and so forth. Finally, the node publishes a new ar_tagstr message containing the object name and pose to the topic pose_and_item.

• The movement.py Will subscribe to the topic pose_and_item to find the x and y positions of the items. The process of picking and placing of objects is broken down into several independent tasks, this node defines each of these tasks as a separate function. The main function calls all of the movement functions in the correct order after the x and y positions of the objects are obtained. The node also calls the open_service and close_service services to open and close the Keurig lid.

• The open_service.py script launches the opener_node node, which provides the /opener service. This service is intended to be called with a Pose message containing the position of the coffee maker relative to the base frame of Baxter. When called, the node will use the ExternalTools/left/PositionKinematicsNode/IKService to produce a series of joint angle solutions relative to the coffee maker, then command the left arm and gripper in a sequence intended to open the lid of the coffee maker.

• The close_service.py script launches the close_node node, which provides the /closer service. This service is intended to be called with a Pose message containing the position of the coffee maker relative to the base frame of Baxter. When called, the node will use the ExternalTools/left/PositionKinematicsNode/IKService to produce a series of joint angle solutions relative to the coffee maker, then command the left arm and gripper in a sequence intended to close the lid of the coffee maker.

Topics & Services

Services

• /opener: Receives a Pose message containing the position of the coffee maker. The node will plan and execute a series of joint movements, then return the Bool True. Uses the Open2.srv definition.

• /closer: Receives a Pose message containing the position of the coffee maker. The node will plan and execute a series of joint movements, then return the Bool True. Uses the Open2.srv definition.

Topics

• /pos_items: Publishes the names of what objects are being seen

• /ar_pose_id:Publishes the id number, position, and orientations of the AR tags

• /pose_and_item: Publishes the positions of the AR tags with what item each position corresponds to

Messages

• ar_tag.msg: contains the id number as a uint32 and the pose as a geometry_msgs/Pose
• ar_tagstr.msg: contains contains the item label as a string and the pose as a geometry_msgs/Pose
• Open2.srv: Contains the service definition used to communicate with open_service.py and close_service.py.

External packages

• ar_track_alvar: This package will tack AR tags.
• CVBridge: This package will convert ROS images into open CV images

Launch Files

• baxter_sw.launch: This launch file will start the ar_track_alvar AR tag tracking and will sets the camera it identifies tags with to be the left hand camera. It will then start the artransforms.py, starbax_sw.py, table_cam_sw.py, movement.py, pose_and_item.py,close_service.py, and open_service.py python scripts in that order. There have been some instances of the pose-and-item node crashing after startup. The suggested work-around for this is to either restart that node individually.