Of course! Using a Kinect with ROS and Python is a classic and powerful combination for robotics, computer vision, and 3D perception. This guide will walk you through the entire process, from setting up the hardware to writing Python scripts to access the Kinect's data.
Overview of the Tools
- ROS (Robot Operating System): Provides the communication framework (topics, services) and many standard tools for robotics.
- Kinect Sensor: The hardware that captures RGB images, depth data, and (in older models) body skeleton information.
- Python: The programming language we'll use to write our ROS nodes. It's excellent for rapid prototyping and leveraging libraries like OpenCV and NumPy.
- OpenNI & NITE (The "Old" Way): Historically, this was the primary driver for Kinect on Linux. It's largely deprecated but you might still find tutorials for it. We will NOT use this method.
- libfreenect2 (The "New" & Recommended Way): This is the modern, high-performance, and actively maintained open-source driver for the Kinect v2. It's the standard for ROS integration today.
Step 1: Hardware and Software Prerequisites
Hardware
- A Microsoft Kinect for Windows v2 sensor.
- A USB 3.0 port. The Kinect v2 requires the higher bandwidth of USB 3.0 and will not work with USB 2.0.
- A power supply for the Kinect (it comes with a special splitter cable that plugs into a wall outlet and the USB port).
Software
- Ubuntu 20.04 LTS (or 18.04). This guide is tailored for Ubuntu 20.04.
- ROS Noetic Ninjemys. This is the ROS version for Ubuntu 20.04.
- A working Python 3 environment (
python3,pip3).
Step 2: Installation
We'll follow the standard ROS installation process for the libfreenect2 driver.
Install System Dependencies
Open a terminal and install the necessary libraries for libfreenect2 and its ROS wrapper.
# Update your package list
sudo apt-get update
# Install required system libraries
sudo apt-get install -y \
libusb-1.0-0-dev \
pkg-config \
libglfw3-dev \
libgl1-mesa-dev \
libglu1-mesa-dev \
libtbb2 \
libtbb-dev \
libturbojpeg0-dev \
libjpeg-turbo8-dev
# Install the ROS driver package
# Replace 'ros-noetic-' with 'ros-melodic-' if you are on Ubuntu 18.04
sudo apt-get install -y ros-noetic-libfreenect2 ros-noetic-freenect2-launch
Install libfreenect2 from Source (Recommended for Performance)
The version in the Ubuntu repositories can be outdated. Compiling from the official source is highly recommended.
# Create a workspace for building mkdir -p ~/ros_kinect_ws/src cd ~/ros_kinect_ws/src # Clone the libfreenect2 repository git clone https://github.com/OpenKinect/libfreenect2.git cd libfreenect2 # Create a build directory mkdir build cd build # Compile the library cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local make -j4 sudo make install sudo make install_rules
Note: The sudo make install_rules step is important to set up the correct udev rules so your computer recognizes the Kinect.
Test the Driver (Before ROS)
It's a good idea to make sure the driver works on its own.
# Plug in your Kinect v2 to a USB 3.0 port and connect its power. # Run the viewer Protonect -l 1
You should see two windows pop up: one for the RGB image and one for the depth image. If you see this, the hardware and driver are working perfectly! Close the viewer with Ctrl+C.
Step 3: Running the ROS Node
The ros-noetic-freenect2-launch package we installed provides an easy way to launch the Kinect driver as a ROS node.
Source Your ROS Workspace
Make sure your ROS environment is active.
source /opt/ros/noetic/setup.bash
Launch the Kinect Node
The launch file will start the driver, which will publish the sensor data as ROS topics.
# Navigate to your workspace cd ~/ros_kinect_ws # Run the launch file roslaunch freenect2.launch
You should see output indicating that the node has started. The Kinect is now publishing data!
Check the Published Topics
In a new terminal, run:
rostopic list
You should see topics like:
/camera/rgb/image_raw(The RGB image)/camera/depth/image_raw(The depth image)/camera/ir/image_raw(The infrared image)/camera/depth/camera_info(Intrinsics for the depth camera)/camera/rgb/camera_info(Intrinsics for the RGB camera)/freenect2/device(System status of the device)
Let's inspect one of the image topics:
# See the message type rostopic type /camera/rgb/image_raw # Output: sensor_msgs/Image # View the image using rqt_image_view rqt_image_view
In the rqt_image_view window, you can select the /camera/rgb/image_raw or /camera/depth/image_raw topics to see the live feed.
Step 4: Writing a Python Subscriber Node
This is where we access the Kinect data in our own Python script. We will create a simple node that subscribes to the RGB and depth topics and displays them side-by-side using OpenCV.
Create a Python Script
Create a new file in your src folder.
cd ~/ros_kinect_ws/src touch kinect_subscriber.py
Write the Code
Paste the following code into kinect_subscriber.py. The comments explain each part.
#!/usr/bin/env python3
import rospy
import cv2
import numpy as np
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
class KinectSubscriber:
def __init__(self):
"""
Initializes the KinectSubscriber node.
"""
rospy.init_node('kinect_subscriber_node', anonymous=True)
# Create a CvBridge object to convert ROS images to OpenCV format
self.bridge = CvBridge()
# Subscribe to the RGB and Depth image topics
self.rgb_sub = rospy.Subscriber("/camera/rgb/image_raw", Image, self.rgb_callback)
self.depth_sub = rospy.Subscriber("/camera/depth/image_raw", Image, self.depth_callback)
# Initialize image storage
self.rgb_image = None
self.depth_image = None
rospy.loginfo("Kinect Subscriber Node Started. Waiting for images...")
def rgb_callback(self, data):
"""
Callback function for the RGB image topic.
Converts the ROS Image message to an OpenCV image and stores it.
"""
try:
# Convert ROS Image message to a NumPy array (OpenCV format)
self.rgb_image = self.bridge.imgmsg_to_cv2(data, "bgr8")
except Exception as e:
rospy.logerr(f"Could not convert RGB image: {e}")
def depth_callback(self, data):
"""
Callback function for the Depth image topic.
Converts the ROS Image message to an OpenCV image and stores it.
The depth image is a 16-bit single-channel image where each pixel is the depth in millimeters.
"""
try:
# Convert ROS Image message to a NumPy array
self.depth_image = self.bridge.imgmsg_to_cv2(data, "16UC1")
# For visualization, we often scale the depth image to 8-bit
# Normalize the depth values to 0-255 for display
if self.depth_image is not None:
# Clip values to a reasonable range (e.g., 0 to 5 meters = 5000 mm)
depth_clipped = np.clip(self.depth_image, 0, 5000)
# Scale to 8-bit
self.depth_image_display = cv2.convertScaleAbs(depth_clipped, alpha=255.0/5000.0)
except Exception as e:
rospy.logerr(f"Could not convert Depth image: {e}")
def run(self):
"""
The main loop of the node.
It displays the RGB and Depth images in a single window.
"""
rate = rospy.Rate(10) # 10 Hz
while not rospy.is_shutdown():
# Check if both images have been received
if self.rgb_image is not None and self.depth_image_display is not None:
# Resize images to the same size for display
h, w = self.rgb_image.shape[:2]
depth_resized = cv2.resize(self.depth_image_display, (w, h))
# Concatenate the images horizontally
combined_image = np.hstack((self.rgb_image, depth_resized))
# Display the combined image
cv2.imshow('Kinect RGB and Depth Feed', combined_image)
# Check for 'q' key press to exit
if cv2.waitKey(1) & 0xFF == ord('q'):
break
rate.sleep()
# Clean up
cv2.destroyAllWindows()
if __name__ == '__main__':
try:
subscriber = KinectSubscriber()
subscriber.run()
except rospy.ROSInterruptException:
pass
Make the Script Executable
chmod +x ~/ros_kinect_ws/src/kinect_subscriber.py
Create a Launch File for Your Node
It's best practice to launch your custom nodes with a launch file. Create kinect_v2.launch in your src folder.
<launch>
<!-- First, launch the official freenect2 driver -->
<include file="$(find freenect2_launch)/launch/freenect2.launch" />
<!-- Then, launch our custom Python subscriber node -->
<node name="kinect_subscriber_node" pkg="ros_kinect_ws" type="kinect_subscriber.py" output="screen" />
</launch>
Note: The pkg="ros_kinect_ws" assumes you have added your workspace to your catkin workspace list. If not, you can use type="$(find ros_kinect_ws)/src/kinect_subscriber.py".
Build and Run
Your workspace needs to be built so ROS can find your new script.
# Go to the root of your workspace cd ~/ros_kinect_ws # Build the workspace (catkin_make is for catkin workspaces) catkin_make # Source the workspace setup file source devel/setup.bash
Now, you can run everything with a single command:
roslaunch ros_kinect_ws kinect_v2.launch
A window titled "Kinect RGB and Depth Feed" should appear, showing the live feed from your Kinect on the left and the corresponding depth map on the right. Press q in the window to stop the node.
