# 4. ROS2-Depth Camera Basic Course
## 4.1 Depth Camera Configuration
:::{Note}
* This document uses Ubuntu 20.04 and ROS2 Foxy as examples for installation. The system and ROS versions should not be lower than Ubuntu 20.04 and ROS2 Foxy, as software versions below these are outdated. Using outdated software means you will not receive security and maintenance updates, and compatibility issues will pose significant challenges, potentially leading to unsolvable problems.
* Before starting the configuration, please install virtual machine software and import Ubuntu 20.04.
* Next, install ROS Foxy. Refer to "**Install and Configure ROS2**" in the same path as this document. This document will not provide further details on the aforementioned installations.
:::
### 4.1.1 Install and Configure ROS2
:::{Note}
The virtual machine system image provided already has the ROS2 environment configured. This step is for reference only.
:::
(1) Before starting this course, please refer to "**Linux Basic Course-\>3. Linux Installation and Source Replacement**" to install the virtual machine client and import the ROS2 Ubuntu image (the official image is located in the "[**Apppendix ->Virtual Machines->ubuntu_ros2_humble**](resources_download.md)" ).
(2) The image file we provide is in the "[**Appedix->ubuntu_ros2_humble**](resources_download.md)" folder, and the ROS2 environment is already configured. Users can refer to this section for a brief overview of the ROS2 system installation and environment setup.
This tutorial must be used in a virtual machine. The camera functionality package is already configured on the robot, so please do not reconfigure it on the camera.
### 4.1.2 Version Options
Different versions of ROS2 are installed on different versions of Ubuntu. In this section, we will use the virtual machine with Ubuntu 20.04, which corresponds to ROS2 Foxy.
### 4.1.3 Check Ubuntu's Software and Update Sources
(1) Locate '**System&Update**'.
(2) Make sure to check the option for internet downloads and change the download source to the local region's source.
### 4.1.4 Set Encoding Format
(1) Start the virtual machine, and press '**Ctrl+Alt+T**' key to start the terminal.
(2) Input
```bash
sudo locale-gen en_US en_US.UTF-8
```
Set the Ubuntu locale to en_US.UTF-8. After entering the command, press Enter, then enter the virtual machine's password, "**hiwonder**" (users should enter the password they have set for their own virtual machine; in our case, the password is "**hiwonder**").
(3) Run the command below and press Enter key to update the language environment.
```bash
sudo update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
```
(4) Enter the command, press Enter, and export the environment variables to take effect immediately.
```bash
export LANG=en_US.UTF-8
```
### 4.1.5 Set the Download Source for ROS2
* **Enable the Ubuntu Universe Repository**
(1) Enter the command to check if the Universe repository is enabled:
```bash
apt-cache policy | grep universe
```
(2) If the output is as follows, it indicates that it has been enabled, and you can proceed to configure it further in [4.1.4 Set Encoding Format](#anchor_4_1_4).
(3) If you see the following prompt, it means it has not been enabled.
(4) Enter the command to install the software package "**software-properties-common**":
```bash
sudo apt install software-properties-common
```
(5) Enter the command to add the PPA to the Universe repository and automatically import the public key:
```bash
sudo add-apt-repository universe
```
(6) You can enter the command again to check if it has been enabled:
```bash
apt-cache policy | grep universe
```
* **Update the software packages and install the curl and gnupg tools**
(1) Enter the command to update the software installation packages:
```bash
sudo apt update
```
(2) After the update is complete, it will appear as shown in the following image.
(3) Enter the command to install the curl and gnupg tools:
```bash
sudo apt install curl gnupg2 lsb-release
```
(4) During the installation process, if you see the prompt as shown in the following image, simply enter "**y**" and press Enter.
(5) After the installation is complete, it will appear as shown in the following image:
* **Set the public key**
Enter the following command:
```bash
curl -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc | sudo apt-key add -
```
If the following error appears, re-enter the command:
```bash
curl http://repo.ros2.org/repos.key | sudo apt-key add -
```
* **Set Download Source**
Enter the following command:
```bash
sudo sh -c 'echo "deb [arch=$(dpkg --print-architecture)] http://packages.ros.org/ros2/ubuntu $(lsb_release -cs) main" > /etc/apt/sources.list.d/ros2-latest.list'
```
### 4.1.6 Install ROS2
(1) Enter the below command to update the software feature pack.
```bash
sudo apt update
```
(2) After the update is complete, it will appear as shown below.
(3) Run the below command:
```bash
sudo apt install ros-foxy-desktop
```
Press Enter. When prompted whether to install, type "**y**" and press Enter again. Wait for the download and installation to complete. Depending on your network conditions, this process may take about 10 minutes, so please be patient.
(4) After the installation is complete, it will appear as shown below.
### 4.1.7 Install Additional Dependency
(1) Enter the command, press Enter, and input the virtual machine password "**hiwonder**" to install the autocomplete tool:
```bash
sudo apt install python3-argcomplete
```
(2) After the installation is complete, it will appear as shown below.
(3) Input the following command and hit Enter key to install the compilation tool. Then enter '**y**' to confirm the installation.
```bash
sudo apt install python3-colcon-common-extensions
```
(4) After the installation is complete, it will appear as shown below.
### 4.1.8 Set Environment Variables
Sequentially enter the following commands, pressing Enter after each one, to complete the environment variable setup:
```bash
echo "source /opt/ros/foxy/setup.bash" >> ~/.bashrc
```
```bash
source ~/.bashrc
```
### 4.1.9 Test ROS2 System
(1) Enter the command to run a talker.
```bash
ros2 run demo_nodes_cpp talker
```
(2) Press "**Ctrl+Alt+T**" to open a new terminal, then enter the command to run a listener.
```bash
ros2 run demo_nodes_py listener
```
(3) Press "**Ctrl+C**" to stop the execution. If stopping fails, you can repeat pressing "**Ctrl+C**" multiple times until it stops.
### 4.1.10 Network Configuration
(1) Open virtual machine 
.
(2) Click-on **Edit** and choose **Virtual Network Editor**.
(3) Tick '**Bridged**', and choose the corresponding wireless network card.
(4) Select the imported virtual machine, then click on '**Power on this virtual machine**'.
(5) Click on 
to open terminal. Or press the short-cut '**Ctrl+Alt+t**' to open the command-line terminal.
(6) Enter command to check whether the network connection is successful.
```bash
ping www.baidu.com
```
If the terminal prints following content, that means network connection is successful.
If it shows that "**Name or service not known**", that means virtual machine fails to connect to the network.
Execute this step only when network connection is failed. Enter command "**ip a**" to check the ID of network card.
(1) Execute this step only when network connection is failed. Enter command and press Enter. "**ens33**" is the ID of network card. You need to modify the entered command based to suit the specific case at hand.
```bash
sudo dhclient ens33
```
Repeat step (6). If the terminal prints following content, that means network connection is successful.
### 4.1.11 Install Dependency
(1) Run the command below to update apt library.
```bash
sudo apt update
```
After update, the following messages will occur.
(2) Enter the command to install the necessary dependencies for loading the depth camera:
```bash
sudo apt install libgflags-dev ros-$ROS_DISTRO-image-geometry ros-$ROS_DISTRO-camera-info-manager ros-$ROS_DISTRO-image-transport ros-$ROS_DISTRO-image-publisher libgoogle-glog-dev libusb-1.0-0-dev libeigen3-dev -y
```
## 4.2 Construct Workspace
(1) Execute the command to build the workspace.
```bash
mkdir -p ~/ros_ws/src && cd ~/ros_ws/src
```
(2) Execute the command to compile the new workspace. Ensure that the compilation is performed within the '**ros_ws**' workspace.
```bash
colcon build
```
(3) Run the command to set a new workspace.
```bash
. ./install/setup.bash
```
(4) Enter the command to append the initialization action of environment variables to the terminal initialization file.
```bash
echo "source $HOME/ros_ws/install/setup.bash" >> ~/.bashrc
```
## 4.3 Install Depth Camera ROS SDK
(1) Copy the file [OrbbecSDK_ROS2.zip](resources_download.md) to the virtual machine.
(2) Right-click the package and select '**Open**'.
(3) Click on the folder and then click "**Extract**" to extract the folder into the workspace.
The green box represents the workspace path. Click "**Extract**" to extract.
(4) Run the command to access the feature pack:
```bash
cd ~/ros_ws/src/OrbbecSDK_ROS2/orbbec_camera/scripts
```
(5) Run the following command and hit Enter key, then input the password.
```bash
sudo bash install_udev_rules.sh
```
(6) Enter the below command to reload the udev rule.
```bash
sudo udevadm control --reload-rules && sudo udevadm triggers
```
(7) Run the command to navigate to the workspace:
```bash
cd ~/ros_ws/
```
(8) Enter the command to recompile the workspace.
```bash
colcon build --event-handlers console_direct+ --cmake-args -DCMAKE_BUILD_TYPE=Release
```
(9) Enter the command to set the new workspace environment variables.
```bash
. ./install/setup.bash
```
(10) So far, we have completed the installation of the Depth Camera ROS2 SDK.
## 4.4 Usage of Depth Camera
### 4.4.1 Enable Camera Service
(1) Attach the depth camera to the computer and establish its connection with the virtual machine.
(2) Run the command to start the camera.
```bash
ros2 launch orbbec_camera dabai.launch.py
```
We offer two approaches for image visualization. One option involves utilizing **rqt_image_view** to observe 2D images, while the alternative is to employ rviz for visualizing 3D images. Users are free to select the method that aligns with their particular needs. It is crucial to emphasize that during image viewing, the camera service should remain active and not be closed.
### 4.4.2 View Image Using rqt_image_view Tool
(1) Open a new terminal, and execute the command below to start the image viewing tool.
```bash
rqt
```
When the below interface appears, it means that the tool is started successfully.
Select "**Plugins -\> Visualization -\> Image View**".
(2) To view the infrared image, please select the corresponding topic.
(3) When viewing the depth map, the red box position needs to be set as shown in the following figure:
### 4.4.3 View Image Using rviz
(1) Execute the command to initiate the tool.
```bash
ros2 run rviz2 rviz2
```
When the below interface appears, it means that the tool is started successfully.
(2) To access the camera feed, start by configuring the "**Fixed Frame**" to "**camera_link**." Then, click on "**Add**," select "**By topic -\> camera -\> color -\> Image_raw -\> Image**," and finally, click "**OK**."
When the interface below appears, it indicates that the tool has been successfully launched.
(3) To view the depth map, set '**Fixed Frame**' as '**camera_link**', then click-on '**Add**' to select '**By topic -\>camera-\>depth-\>Image_raw-\>Image**', and click-on '**OK**'.
When the interface below appears, it indicates that the tool has been successfully launched.
### 4.4.4 LDP Protection
LDP protection typically refers to '**Lens Data Protection**' in the context of cameras. It is designed to protect the camera lens from accidental damage or impact. With the DaBai camera, if an object gets too close, the infrared camera will automatically shut down. The shutoff distance depends on the intensity of reflected light. We can enable or disable LDP protection according to our needs.
(1) Run the command to turn on the camera.
```bash
ros2 launch orbbec_camera astraproplus.launch.py
```
(2) Open a new terminal, and execute the command to initiate the image viewing tool.
```bash
rqt
```
The appearance of the interface in the following image indicates successful opening.
Choose '**Plugins\>Visualization\>ImageView**'.
Set the red box location to the content in the image below:
When the object reaches a certain distance from the camera, the depth map will transition to black.
(3) Open a new command-line terminal, and execute the command to close LDP protection.
```bash
ros2 service call /camera/set_ldp_enable std_srvs/srv/SetBool data:\ false
```
At this stage, the image remains visible even as an object approaches the camera.
(4) Execute the command to initiate LDP protection.
```bash
ros2 service call /camera/set_ldp_enable std_srvs/srv/SetBool data:\ true
```
## 4.5 Data Type and Point Cloud
### 4.5.1 Introduction to Point Cloud
The things we observe with our eyes are three-dimensional, but in computer vision, using regular cameras only provides two-dimensional information about an object, which cannot achieve the same observation effect as human eyes. In order for computers to represent three-dimensional objects, computer vision needs to transition into three-dimensional vision, and point clouds are the way to express three-dimensional vision.
### 4.5.2 Understanding Point Cloud Data Types
Point cloud data refers to a collection of vectors in a three-dimensional coordinate system, typically expressed in the X, Y, and Z axes of the three-dimensional coordinate system. It is commonly used to represent the surface shape of an object. Point cloud data can also represent attributes such as RGB colors, grayscale values, depth, etc., of an object's surface.
Robot point cloud data is acquired through LiDAR and depth cameras, both of which are capable of measurements.
Common formats for storing point cloud data include pts, asc, dat, stl, imw, xyz, txt, csv, etc. Point cloud data, much like an attribute table, records the X, Y, Z positions of corresponding points along with various attributes associated with each point.
### 4.5.3 Point Cloud
* **Depth Camera Point Cloud**
:::{Note}
This step is performed on the robot, and when entering commands, it's important to strictly differentiate between uppercase and lowercase letters. You can also use the **"Tab"** key to autocomplete keywords.
:::
(1) Start the robot and connect to the Ubuntu desktop remotely using NoMachine.
(2) Click-on 
to start the command-line terminal.
(3) Execute the below command and hit Enter key to disable the app auto-start service.
```bash
sudo systemctl stop start_app_node.service
```
(4) Open a new command-line terminal, and run the following command to open the point cloud view of the depth camera.
```bash
ros2 launch peripherals depth_camera.launch.py
```
(5) Open a new command-line terminal and execute the below command to initiate rviz.
```bash
rviz2
```
(6) Choose '**depth_cam_link**'.
(7) As shown in the image, click **Add -\> By topic -\> depth_cam -\> depth -\> points -\> PointCloud2**, and then click **OK**.
Once you've completed the previous settings, the point cloud will become visible.
* **Depth Camera Infrared Image**
:::{Note}
When entering commands, it's important to strictly observe capitalization, and you can use the "**Tab**" key to autocomplete keywords.
:::
(1) Start the robot, and access the ubuntu desktop using the software NoMachi.
(2) Click-on to open the command-line terminal.
(3) Enter the command below and hit Enter key to disable the app auto-start service.
```bash
sudo systemctl stop start_app_node.service
```
(4) Open a new command prompt, enter the command to open the depth camera point cloud view:
```bash
ros2 launch peripherals depth_camera.launch.py
```
(5) Open a new command-line terminal, then enter the command '**rviz2**' to activate the tool rviz.
(6) Choose "**depth_cam_link**".
(7) Click on "**Add-\>By topic-\>depth_cam-\>ir-\>image_raw-\>image**". Finally, click-on **'OK'** button.
The picture below will appear.
