Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Next »

Overview

This application note explains how to use the ZED camera with the Emcraft software distribution for the Renesas RZ/V2H SoC.

High-level Overview

The ZED camera support may be seen as a part of the ROS2 support layer. Information on the ROS2 support layer is available from Using ROS2 Layer and APIs .

The ZED camera support is adapted from the TI’s Robotic SDK https://software-dl.ti.com/jacinto7/esd/robotics-sdk/latest/docs/source/README.html, which is based on the OpenCV library.

ZED Camera Support Packages

The Emcraft software distribution installs to the target image the following ZED camera support packages:

ROS2 Package

Version

Summary

Notes

ros-humble-zed-capture

0.0.1-1

A ZED camera driver ROS node based on OpenCV VideoCapture API for publishing left and right raw images.

 

NOTE: More ROS2 demo nodes from the TI’s SDK are going to be adapted soon.

Understanding ZED Packages Build Process

Emcraft makes use of a separate build environment to build the ZED camera support packages integrated into the Emcraft software distribution for the RZ/V2H SoC.

The build environment is based on the GitHub - ros/meta-ros: OpenEmbedded Layers for ROS 1 and ROS 2 Yocto meta-layer and is configured to build packages for the ARM64 and AMD64 architectures. The built binary packages are then published via a private Debian package archive. The Emcraft software distribution for the RZ/V2H SoC has the PPA included to the APT source list.

Verifying ZED Camera Support

Running ZED Capture ROS2 Node

Using ROS2 Interface to the ZED Camera

Step through the following procedure to demonstrate use of the ROS2 interface to the ZED camera:

  1. Open a terminal window and login onto the RZ/V2H EVK board using ssh:

    <terminal 1>$ ssh user@<evk IP>

  2. Connect the ZED camera to the EVK USB 3.x interface.

  3. Run the ROS2 camera node wrapper:

    <terminal 1>$ ros2 launch zed_capture zed_capture_launch.py zed_sn_str:=SN10026720

    The node will start with providing logs and information on the detected ZED camera:

    [INFO] [launch]: All log files can be found below /home/user/.ros/log/2023-12-18-11-24-52-897616-rzv2hevkalpha-2902
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [zed_capture-1]: process started with pid [2945]
[zed_capture-1] [INFO] [1702898698.084998416] [zed_capture]: Initialize the ZED camera
[zed_capture-1] [ WARN:0] global ./modules/videoio/src/cap_gstreamer.cpp (2075) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module source reported: Could not read from resource.
[zed_capture-1] [ WARN:0] global ./modules/videoio/src/cap_gstreamer.cpp (1053) open OpenCV | GStreamer warning: unable to start pipeline
[zed_capture-1] [ WARN:0] global ./modules/videoio/src/cap_gstreamer.cpp (616) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
[zed_capture-1] [INFO] [1702898699.271579125] [StereoCamera]: YUV422
[zed_capture-1] [INFO] [1702898699.271702458] [StereoCamera]: Stereo Camera Mode HD, width 1280, height 720
[zed_capture-1] [INFO] [1702898699.432589500] [zed_capture]: Loading camera info from yaml files
[zed_capture-1] [INFO] [1702898699.435826125] [zed_capture]: camera calibration URL: file:///opt/ros/humble/share/zed_capture/config/SN10026720_HD_camera_info_left.yaml
[zed_capture-1] [INFO] [1702898699.439227625] [zed_capture]: camera calibration URL: file:///opt/ros/humble/share/zed_capture/config/SN10026720_HD_camera_info_right.yaml
[zed_capture-1] [INFO] [1702898705.495579435] [zed_capture]: Successfully found the ZED camera
...

  4. Open another terminal window and login onto the EVK:

    <terminal 2>$ ssh user@<evk IP>

  5. Verify that the camera related topics are present:

    <terminal 2>$ ros2 topic list
/camera/left/camera_info
/camera/left/image_raw
/camera/right/camera_info
/camera/right/image_raw
/parameter_events
/rosout

  6. Capture the camera information via the camera/left/camera_info topic:

    <terminal 2>$ ros2 topic echo /camera/left/camera_info

    The application will start receiving the camera info message:

    header:
  stamp:
    sec: 1695145271
    nanosec: 503948931
  frame_id: left_frame
height: 720
width: 1280
distortion_model: plumb_bob
d:
- -0.174814
- 0.0277134
- 0.0
- 0.0
- 0.0
k:
- 701.695
- 0.0
- 645.375
- 0.0
- 701.695
- 351.9755
- 0.0
- 0.0
- 1.0
r:
- 0.9999991771672719
- 0.00021264049636184653
- -0.0012650884548001328
- -0.00021565273732970637
- 0.9999971412291467
- -0.0023813919103813186
- 0.001264578457844275
- 0.0023816627706823587
- 0.999996364255276
p:
- 686.7164431988998
- 0.0
- 643.5791931152344
- 0.0
- 0.0
- 686.7164431988998
- 342.6826591491699
- 0.0
- 0.0
- 0.0
- 1.0
- 0.0
binning_x: 0
binning_y: 0
roi:
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: false
---
...

  7. Capture the camera information via the camera/right/camera_info topic:

    <terminal 2>$ ros2 topic echo /camera/right/camera_info

    The application will start receiving the camera info message:

    header:
  stamp:
    sec: 1695145324
    nanosec: 367362789
  frame_id: right_frame
height: 720
width: 1280
distortion_model: plumb_bob
d:
- -0.174312
- 0.0273345
- 0.0
- 0.0
- 0.0
k:
- 700.655
- 0.0
- 636.98
- 0.0
- 700.655
- 343.9995
- 0.0
- 0.0
- 1.0
r:
- 1.0
- 2.1923516186951437e-14
- -1.2767118909117024e-13
- -2.1624428776507077e-14
- 0.9999971641585754
- 0.002381527830432345
- 1.2772216711642694e-13
- -0.002381527830432344
- 0.9999971641585754
p:
- 686.7164431988998
- 0.0
- 643.5791931152344
- -30357.52616045848
- 0.0
- 686.7164431988998
- 342.6826591491699
- 0.0
- 0.0
- 0.0
- 1.0
- 0.0
binning_x: 0
binning_y: 0
roi:
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: false
---
...

  8. Capture the camera information via the camera/left/camera_info topic:

    <terminal 2>$ ros2 topic echo /camera/left/image_raw

    The application will start receiving the camera info message:

    header:
  stamp:
    sec: 1695145357
    nanosec: 770892888
  frame_id: left_frame
height: 720
width: 1280
encoding: yuv422
is_bigendian: 0
step: 2560
data:
- 132
- 11
- 134
- 12
- 132
- 11
- 134
- 11
- 132
- 11
- 134
- 11
- 132
- 11
- 134
- 11
- 132
- 12
- 134
- 11
- 132
- 12
- 134
- 11
- 132
- 12
- 134
- 12
- 132
- 12
- 134
- 12
- 132
- 11
- 134
- 11
- 132
- 12
- 134
- 12
- 132
- 12
- 134
- 12
- 132
- 12
- 134
- 12
- 132
- 12
- 134
- 12
- 132
- 12
- 134
- 12
- 132
- 12
- 134
- 13
- 132
- 12
- 134
- 12
- 132
- 12
- 135
- 12
- 132
- 13
- 135
- 12
- 133
- 13
- 135
- 14
- 133
- 19
- 136
- 18
- 133
- 23
- 137
- 24
- 133
- 28
- 137
- 25
- 133
- 33
- 138
- 29
- 134
- 46
- 137
- 90
- 120
- 162
- 135
- 170
- 106
- 153
- 135
- 105
- 121
- 57
- 133
- 29
- 131
- 37
- 133
- 40
- 130
- 54
- 133
- 49
- 130
- 36
- 132
- 26
- 131
- 27
- 133
- 29
- 131
- 32
- 134
- 25
- '...'
---
...

  9. Capture the camera information via the camera/right/camera_info topic:

    <terminal 2>$ ros2 topic echo /camera/right/image_raw

    The application will start receiving the camera info message:

    header:
  stamp:
    sec: 1695145404
    nanosec: 252655285
  frame_id: right_frame
height: 720
width: 1280
encoding: yuv422
is_bigendian: 0
step: 2560
data:
- 131
- 29
- 134
- 25
- 131
- 25
- 134
- 25
- 132
- 29
- 134
- 27
- 132
- 22
- 134
- 14
- 132
- 13
- 134
- 13
- 132
- 14
- 134
- 13
- 132
- 13
- 134
- 14
- 132
- 13
- 134
- 13
- 132
- 13
- 134
- 13
- 132
- 13
- 134
- 14
- 132
- 13
- 134
- 12
- 132
- 13
- 134
- 14
- 132
- 13
- 134
- 13
- 132
- 13
- 134
- 13
- 132
- 13
- 134
- 12
- 132
- 13
- 134
- 13
- 132
- 13
- 134
- 13
- 132
- 13
- 134
- 13
- 133
- 13
- 134
- 13
- 133
- 13
- 134
- 13
- 133
- 13
- 135
- 13
- 133
- 13
- 135
- 13
- 133
- 13
- 135
- 13
- 133
- 13
- 135
- 13
- 132
- 13
- 135
- 13
- 132
- 13
- 135
- 13
- 132
- 14
- 135
- 13
- 132
- 13
- 134
- 13
- 132
- 12
- 134
- 12
- 132
- 13
- 134
- 12
- 132
- 12
- 134
- 12
- 132
- 12
- 134
- 12
- '...'
---
...

Recording Capture and Playing Back the Captured Data in RVIZ

The ZED camera video capture can be played back on a host with installed rviz2. The capture is streamed by the ROS2 layer using the RTPS protocol as being captured, hence it can be played back online. Also, in the absence of an Ethernet connection between the RZ/V2H EVK and the host running rviz2, the ROS2 data can be captured into a bag and then played back by the host.

The capturing can be performed by the ros2 bag command and requires the ros-humble-ros2bag package to be installed:

user@rzv2hevkalphaemc:~$ sudo apt-get install ros-humble-ros2bag

The capture can be started by the following command:

ros2 bag record --regex /camera*

The command will start writing bagged data:

[INFO] [1695145640.224397231] [rosbag2_recorder]: Press SPACE for pausing/resuming
[INFO] [1695145640.259529398] [rosbag2_storage]: Opened database 'rosbag2_2023_09_19-17_47_20/rosbag2_2023_09_19-17_47_20_0.db3' for READ_WRITE.
[INFO] [1695145640.262090065] [rosbag2_recorder]: Listening for topics...
[INFO] [1695145640.262104523] [rosbag2_recorder]: Event publisher thread: Starting
[INFO] [1695145640.268865856] [rosbag2_recorder]: Subscribed to topic '/camera/right/image_raw'
[INFO] [1695145640.271986231] [rosbag2_recorder]: Subscribed to topic '/camera/right/camera_info'
[INFO] [1695145640.274928106] [rosbag2_recorder]: Subscribed to topic '/camera/left/image_raw'
[INFO] [1695145640.279502231] [rosbag2_recorder]: Subscribed to topic '/camera/left/camera_info'
[INFO] [1695145640.279967106] [rosbag2_recorder]: Recording...

The stream bandwidth is rather high, so consider stopping the capture by Ctrl-C after a while (say, 30 seconds).

[INFO] [1695145653.805510113] [rosbag2_cpp]: Writing remaining messages from cache to the bag. It may take a while
[INFO] [1695145653.811707821] [rosbag2_recorder]: Event publisher thread: Exiting
[INFO] [1695145653.812272321] [rosbag2_recorder]: Recording stopped
[INFO] [1695145653.822919904] [rosbag2_recorder]: Recording stopped

The data is written to the folder. Use tar to compress it:

user@rzv2hevkalphaemc:~$ tar czf bag.tgz rosbag2_2023_09_19-17_47_20/

Then copy file to the host with rviz2.

On the host, start rviz2 . 

user@ros2build:/opt/host$ rviz2 &
[1] 46
[INFO] [1703144653.332480044] [rviz2]: Stereo is NOT SUPPORTED
[INFO] [1703144653.332564044] [rviz2]: OpenGl version: 4.5 (GLSL 4.5)
[INFO] [1703144653.418620391] [rviz2]: Stereo is NOT SUPPORTED

Add two Image visualizations, and input topic names: /camera/left/image_raw and /camera/right/image_raw as depicted on screenshots below:

Repeat this sequence for the left and right camera topics.

Switch back to the terminal window and in-tar the data bag:

user@ros2build:/opt/host$ tar xzf bag.tgz

And then start the playback:

user@ros2build:/opt/host$ ros2 bag play rosbag2_2023_09_19-17_47_20/
[INFO] [1703000255.469821687] [rosbag2_storage]: Opened database 'voxel/vgslam_ros/rosbag2_2023_09_19-17_47_20/rosbag2_2023_09_19-17_47_20_0.db3' for READ_ONLY.
[INFO] [1703000255.469868576] [rosbag2_player]: Set rate to 1
[INFO] [1703000255.474512300] [rosbag2_player]: Adding keyboard callbacks.
[INFO] [1703000255.474549801] [rosbag2_player]: Press SPACE for Pause/Resume
[INFO] [1703000255.474565200] [rosbag2_player]: Press CURSOR_RIGHT for Play Next Message
[INFO] [1703000255.474581401] [rosbag2_player]: Press CURSOR_UP for Increase Rate 10%
[INFO] [1703000255.474605457] [rosbag2_player]: Press CURSOR_DOWN for Decrease Rate 10%

rviz2 would start visualizing the captured data.

The image below depicts the image visualization:

  • No labels