3. AI Entry Level Lesson

3.1 OpenCV Introduction

3.1.1 OpenCV Introduction

Open Source Computer Vision Library (OpenCV) is based on optimised C/ C++ to allow you perform image processing, computer vision and machine learning.

It was created to compute effectively and mainly focus on real-time application development. Therefore, OpenCV is widely applied in image stitching, image denoising, face recognition, posture recognition, autonomous driving and other fields.

3.1.2 OpenCV Features

(1) Support C, C++, Python, JAVA and MATLAB

(2) Support Linux, iOS, Android, Windows and Windows CE

(3) OpenCV is written in C++ and deeply optimized to take full advantage of multi-core processors.

3.1.3 OpenCV Module

Open has a modular structure and contains several shared and static libraries. Some of modules in OpenCV is introduced as follow:

Name	Instruction
Core (Core functionality)	Core module contains the basic structure and operation of OpenCV library
Imgproc (Image Processing)	Image processing module contains linearimage filter, geometric image conversion and color space conversion functions.
Highgui（High-level GUI）	Contain function for graphical user interface design.
Video (Video Analysis)	Video analysis module contains motionestimation, background subtraction and target tracking algorithms.
Calib3d (Camera Calibration and 3D Reconstruction)	Contain algorithm calibrating single or several monocular cameras.
features2d (2D Features Framework)	Contains algorithm for detecting, describing and matching feature.
Objdetect (Object Detectio)	Contain algorithm for detecting specific target (such as face, car)
ML(Machine Learning)	Machine learning module contains lots ofmachine learning algorithms implementation.

3.2 USB Camera Test and Use

3.2.1 Device Connection

Connect USB camera to one of ports in following red box.

The connection effect is as follow:

3.2.2 Testing

Start Jetson Nano and connect it to Nomachine.

Note

For Remote Desktop Installation and Configuration, please refer to “ 1.1.7 Remote Desktop Installation and Configuration” .

Double click onto open command line terminal.
Shrink Nomachine window and drag “usb_camera_test.py” file under the same directory this document to Nomchine window.

Right click to select “Open Terminal” .

Enter command “cd Desktop/” and press Enter to enter desktop.

cd Desktop/

Enter command “python3 usb_camera_test.py” and press Enter to start testing USB camera.

python3 usb_camera_test.py

If the following window pops up and the camera returned image appear, camera can normally be used. Press “Esc” to close window.

3.2.3 Program Analysis

Following is the test code:

import cv2
if __name__ == "__main__":
    cap = cv2.VideoCapture(0)
    if cap.isOpened():
        window_handle = cv2.namedWindow("D435", cv2.WINDOW_AUTOSIZE)    
        # 逐帧显示
        while cv2.getWindowProperty("D435", 0) >= 0:
            ret_val, img = cap.read()
            cv2.imshow("D435", img)
            keyCode = cv2.waitKey(30) & 0xFF         
            if keyCode == 27:# ESC键退出
                break 
        cap.release()
        cv2.destroyAllWindows()
    else:
        print("打开摄像头失败")

Import opencv library:

import cv2

Select camera. Since only a single camera is connected, the parameter is set to 0.

cap = cv2.VideoCapture(0)

Set the window of returned image. The window name is D435 and video size is cv2.WINDOW_AUTOSIZE .

if cap.isOpened():
        window_handle = cv2.namedWindow("D435", cv2.WINDOW_AUTOSIZE)

Read and display the data of the returned image.

# 逐帧显示
        while cv2.getWindowProperty("D435", 0) >= 0:
            ret_val, img = cap.read()
            cv2.imshow("D435", img)

Press “Esc” to close window.

keyCode = cv2.waitKey(30) & 0xFF         
            if keyCode == 27:# ESC键退出
                break

If camera is not detected or other errors appear, “Camera Open Failure” will be printed.

cap.release()
        cv2.destroyAllWindows()
    else:
        print("打开摄像头失败")

3.3 CSI Camera Introduction and Installation

3.3.1 CSI Camera Introduction

The Camera Serial Interface (CSI) is is a interface specification of the Mobile Industry Processor Interface (MIPI) Alliance with higher bandwidth and lower power consumption compared to USB cameras, and Jetson series provide a sef of Camera SubSystem to improve efficiency, making it ideal for video image input on intelligent robots.

3.3.2 CSI Camera Installation

Preparation

Jetson Nano board
CSI Camera
Camera cable

Installation

Pull up the baffle over CSI interface.

Insert camera cable to CSI interface. Please note that the blue side should face outward. Then press down the baffle over CSI interface.

Pull up the baffle of the CSI camera module.

Complete the wiring as follow.

3.4 CSI Camera Test and Use

Note

Please complete camera assembly according to “3.3 CSI Camera Introduction and Installation”.

3.4.1 Operation Steps

Note

The entered command should be case sensitive, and “Tab” key can be used to auto-complete key words.

Start Jetson Nano and connect it to ubuntu desktop via NoMachine.
Import “Camera.py” file under the same directory with document to home directory.

Click on or press “Ctrl+Alt+T” to open command line terminal.
Enter command “python3 Camera.py” and press Enter to start game.

python3 Camera.py

If want to exit this game, please press “Ctrl+C” in terminal. Please try multiple times for failure operation.

3.4.2 Outcome

After this game starts, the camera returned image will appear on screen.

3.4.3 Program Analysis

Read CSI Camera

Posture is mainly adjusted by VideoCapture function. Take code cv2.VideoCapture(gstreamer_pipeline(flip_method=0), cv2.CAP_GSTREAMER) as example. The meaning of parameters is as follow:

The first parameter gstreamer_pipeline(flip_method=0) is the framework delivered to gstreamer_pipeline, which set the read property of video data.

The first parameter gstreamer_pipeline(flip_method=0) is the frame passed to gstreamer_pipeline, which sets the read property of the video data.
The second parameter cv2.CAP_GSTREAMER is to select Pipeline mode.

Read and Display Image

First read image by calling read, then display image by calling imshow. Take code cv2.imshow("img", frame) as example, the meaning of parameters is as follow:

The first parameter img is the title of display window.
The second parameter frame is the camera returned image.

When press q to close window, release method will be called first to release camera, and then destroyAllWindows method is called to open all windows.

3.5 MidiaPipe Introduction and Installation

3.5.1 Introduction to MediaPipe

MediaPipe is an open source application framework for multi-media machine learning models. This cross-platform Framework works on mobile devices, workstations and servers, and supports mobile GPU acceleration. It also supports TensorFlow and TF Lite inference engine, any TensorFlow and TF Lite models can be used on MediaPiple. Also, on mobile and embedded

platforms, MediaPipe supports GPU acceleration on the device itself.

3.5.2 MediaPipe Pros and Cons

Advantage

Support several platforms and languages, such as IOS 、Android、C++、 Python、JAVAScript 、Coral.
Fast speed. Model can basically run in real time.
High reusing rate of model and code.

Disadvantage

For mobile, MediaPipe requires 10M or more space.
Deelpy dependent on TensorFlow. If want to replace it with another machine learning framework, amount of codes need to change.
Static images are used, which can improve efficiency but also make it difficult to detect errors.

3.5.3 MediaPipe Workflow

Follow is MediaPipe workflow. The process of solid line needs to write code by yourself, but no need for the process of dotted line. MediaPipe has integrated Al-related models and games. Thus, user can use MediaPipe to quickly derive the framework needed to implement a feature

3.5.4 Install MediaPipe

Start Jetson Nano and connect it to NoMachine.
Double click on to open command line terminal.
Enter command “sudo apt update” to update APT download list.

sudo apt update

Enter command “sudo apt install python3-pip” and press Enter to install pip.

Note

If do not update pip, it may cause the installation to fail.

sudo apt install python3-pip

Enter command “python3 -m pip install –upgrade pip” and press Enter to update pip.

python3 -m pip install --upgrade pip

Zoom in NoMachine window. Then drag

“mediapipe-0.8.5_cuda102-cp36-cp36m-linux_aarch64.whl” under the same directory with this document to NoMachine window.

Right click to select “Open Terminal” to open command line terminal.

Enter command “pip3 install mediapipe-0.8.5_cuda102-cp36-cp36m-linux_aarch64.whl” and press Enter to install MediaPipe. The installation needs to take some time.

pip3 install mediapipe-0.8.5_cuda102-cp36-cp36m-linux_aarch64.whl

3.6 MediaPipe Face Recognition

3.6.1 Media Face Recognition Program

Start Jetson Nano and connect it to NoMachine.
Zoom in NoMachine window and drag file “mp_face_detect.py” under the same directory with this document to NoMachine window.

Double click on to open command line terminal.
Enter command “pip3 install dataclasses” and press Enter to install dependency package.

pip3 install dataclasses

Enter command “python3 mp_face_detect.py” and press Enter to execute program.

python3 mp_face_detect.py

Jetson Nano will automatically recognize and mark the face.

3.7 MediaPipe Gesture Recognition

3.7.1 Media Gesture Recognition Program

Start Jetson Nano and connect it to Nomachine.
Zoom in NoMachine window and drag the file “mp_face_detect.py” to NoMachine window.

Double click on to open command line terminal.
Enter command “pip3 install dataclasses” and press Enter to install dependency package.

pip3 install dataclasses

Enter command “python3 mp_gesture.py” and press Enter to execute program.

python3 mp_gesture.py

Do a specific gesture within camera’s field of view. When the gesture is recognized, the key points of hand are marked on returned image and the numbers of fingers is displayed on upper left corner.