# 3. Image Recognition Course

## 3.1 Image Transmission

This section will introduce you how connect to the WIFI generated by `ESP32-S3` camera, log in to the fixed URL to view the live image transmission.

:::{Note}
If you're failing to perform this functionality or have downloaded other firmwares for other functions, it is necessary to re-flash the firmware for image transmission.

Firmware download method can refer to [3.1.2 Flash Firmware (Optional)](#anchor_3_1_2) section in this document.
:::

### 3.1.1 Connect to ESP32S3 Camera

* **Connection Mode Introduction**

The `ESP32S3` camera module supports the following two network modes:

(1) AP Direct Connection Mode: the development board can create a hotspot that a mobile phone can connect to. (However, it cannot access the external internet.)

(2) STA (Station) Mode: The development board can actively connect to a specified hotspot/Wi-Fi network. (Internet access is available.)

Using AP direct connection mode is simpler and more convenient, so we recommend users start by learning how to configure and use this mode to get familiar with the basic functions. The STA mode can be explored later based on specific needs.

Additionally, no matter whether the user chooses AP mode or STA mode, the available features and functions remain the same.

* **AP Direct Connection Mode (Must read!)**

(1) Connect the module to your computer using a Type-C cable. (Check in Device Manager to ensure the ports are connected properly.)

<img src="../_static/media/chapter_3/section_1/image2.png" class="common_img" />

:::{Note}
If the device cannot be found in Port, it may be because the driver is not installed on your computer. You can find the installation package in the path [Appendix-\>ch34x Driver (Windows Environment)](Appendix.md) and install it manually.
:::

(2) Connect to the hotspot generated by the module: HW_ESP32S3CAM.

<img src="../_static/media/chapter_3/section_1/image3.png" class="common_img" />

<img src="../_static/media/chapter_3/section_1/image4.png" class="common_img" />

(3) Once connected, in the browser's address bar (both mobile and PC browsers can be used; we'll use the PC as an example), enter `"192.168.5.1"` and press Enter. On the opened page, click on <img src="../_static/media/chapter_3/section_1/image5.png" /> to turn on the live image transmission.

<img src="../_static/media/chapter_3/section_1/image6.png" class="common_img" />

* **STA Mode Connection (Optional)**

(1) Turn on the mobile hotspot and configure the SSID name to `"HiwonderESP"` and the password to `"hiwonder"`.

:::{Note}
The SSID name and password must match exactly as shown here, or the device will fail to connect.
:::

<img src="../_static/media/chapter_3/section_1/image7.jpeg" class="common_img" style="width:400px;"/>

<img src="../_static/media/chapter_3/section_1/image8.jpeg" class="common_img" style="width:400px;"/>

(2) Use a Type-C data cable to connect the camera to your computer. After a short moment, the camera will automatically connect to the mobile hotspot.

(3) After the network is successfully connected, open the Serial Debugging Tool from the path: [Appendix-\>Serial Debugging Tool](Appendix.md), then select the correct COM port and set the baud rate to `115200` to begin communication.

<img src="../_static/media/chapter_3/section_1/image9.png" class="common_img" />

(4) Select the correct COM port (COM3 is used as an example here; the port number may vary). Set the baud rate to `115200`.

:::{Note}
Do not select `COM1`, as it is reserved for system communication.
:::

<img src="../_static/media/chapter_3/section_1/image10.png" class="common_img" />

(5) Click the button <img src="../_static/media/chapter_3/section_1/image11.png" /> in the top-left corner of the interface to open `"UART Assistant"`. Then press the `"RST"` button on the `ESP32-S3` camera board. Scroll down to the last line in the serial output, and you will see the camera's IP address, such as `192.168.197.226`.

:::{Note}
The IP address may vary—please refer to the actual address displayed.
:::

<img src="../_static/media/chapter_3/section_1/image12.png" class="common_img" />

(6) At this point, disconnect the Type-C data cable and connect your computer to the same hotspot as the camera.

<img src="../_static/media/chapter_3/section_1/image13.png" class="common_img" />

(7) In your browser's address bar, enter the camera's IP address (e.g., `192.168.197.226`) to access the video stream interface.

<img src="../_static/media/chapter_3/section_1/image14.png" class="common_img" />

<p id="anchor_3_1_2"></p>

### 3.1.2 Flash Firmware (Optional)

(1) Locate and open the "**flash_download_tool_3.9.7**" stored in [Appendix-\>ESP32-S3 Firmware Flashing Tool-\>flash_download_tool_3.9.7.](Appendix.md).

<img src="../_static/media/chapter_3/section_1/image15.png" class="common_img" />

(2) Select Chip Type as `ESP32-S3`, and keep other settings as default, and then click `"OK"`.

<img src="../_static/media/chapter_3/section_1/image16.png" class="common_img" />

(3) After opening the tool, click **"..."** to select the bin file of the program you need to flash. Below is an example using the image transmission function firmware:

<img src="../_static/media/chapter_3/section_1/image17.jpeg" class="common_img" />

(4) Make sure to check the box on the left. Configure the remaining settings as shown in the image below, and select the `COM` port number corresponding to the port used by the vision module.

:::{Note}
If you set the `SPI MODE` to `DIO` as shown in the figure, the module may not function properly after flashing the firmware. Please try setting the `SPI MODE` to `DOUT` and flash again.
:::

<img src="../_static/media/chapter_3/section_1/image18.png" class="common_img" />

(5) Click `"ERASE"` to wipe the underlying firmware (Do not skip this step), then click `"Start"` to begin flashing the new firmware. Wait for the flashing to complete.

## 3.2 Face Recognition

### 3.2.1 Flash Face Recognition Firmware

[Source Code](../_static/source_code/FaceDetection.zip)

(1) Use a Type-C cable to connect the `ESP32S3` to the USB port of your computer.

(2) Locate and open the program file.

<img src="../_static/media/chapter_3/section_2/image2.png" class="common_img" />

(3) Choose the `"ESP32S3 Dev Module"` for the development board.

<img src="../_static/media/chapter_3/section_2/image3.png" class="common_img" />

(4) Click on the `"Tool"` menu and select the corresponding `ESP32S3` board configuration as shown in the image below.

<img src="../_static/media/chapter_3/section_2/image4.png" class="common_img" />

(5) Click on <img src="../_static/media/chapter_3/section_2/image5.png" /> to download the code to the `ESP32S3` module. Wait for the flashing to complete.

<img src="../_static/media/chapter_3/section_2/image6.png" class="common_img" />

### 3.2.2 Project Performance

(1) Please ensure that one end of the Type-C cable to the `ESP32S3` module and the another end is connected to the USB port on your computer.

(2) Open the serial debugging tool located in the [Appendix-\>Serial Debugging Tool](Appendix.md).

(3) Select the port. Following will take `"COM9"` port as example (the port number is not unique, if `COM1` appears, do not select it as it is the system communication port). Set the baud rate to `"115200"`.

<img src="../_static/media/chapter_3/section_2/image7.png" class="common_img" />

(4) Click on <img src="../_static/media/chapter_3/section_2/image8.png" /> in the top left corner of the page to open `"UART Assistant"`. Press the `"RST"` button on `ESP32` the board. Once the `ESP32` camera detects a face, it will print the coordinates of the face to the serial output.

For example:

If you receive data like `"center_x:103, center_y:114, width:185, length:247,"`, it means a face has been detected. Here, `center_x` and `center_y` represent the coordinates of the top-left corner of the face box, while `width` and `length` represent the width and height of the face box, with all units measured in pixels.

<img src="../_static/media/chapter_3/section_2/image9.png" class="common_img" />

## 3.3 Color Recognition

<p id="anchor_3_3_1"></p>

### 3.3.1 Flash Color Recognition Firmware

[Source Code](../_static/source_code/ColorDetection.zip)

(1) Use a Type-C cable to connect the `ESP32S3` to the USB port of your computer.

(2) Locate and open the program file.

<img src="../_static/media/chapter_3/section_3/image2.png" class="common_img" />

(3) Choose the **"ESP32S3 Dev Module"** for the development board.

<img src="../_static/media/chapter_3/section_3/image3.png" class="common_img" />

(4) Click on the **"Tool"** menu and select the corresponding `ESP32S3` board configuration as shown in the image below.

<img src="../_static/media/chapter_3/section_3/image4.png" class="common_img" />

(5) Click on the `"Tool"` menu and select the corresponding `ESP32S3` board configuration as shown in the image below.

<img src="../_static/media/chapter_3/section_3/image5.png" class="common_img" />

### 3.3.2 Project Performance

(1) Please ensure that one end of the Type-C cable to the `ESP32S3` module and the another end is connected to the USB port on your computer.

(2) Open the serial debugging tool located in the [Appendix-\>Serial Debugging Tool](Appendix.md).

(3) Select the port. Following will take `"COM11"` port as example (the port number is not unique, if `COM1` appears, do not select it as it is the system communication port). Set the baud rate to `"115200"`.

<img src="../_static/media/chapter_3/section_3/image6.png" class="common_img" />

(4) Click on <img src="../_static/media/chapter_3/section_3/image7.png" /> in the top left corner of the page to open `"UART Assistant"`. Press the `"RST"` button on `ESP32` the board. Once the `ESP32` camera detects the preset color, it will print the currently recognized `HSV` color index to the serial output.

<img src="../_static/media/chapter_3/section_3/image8.png" class="common_img" />

For instance,

If receiving `"color[0]"`, it indicates the `HSV` color value for red has been detected.

<img src="../_static/media/chapter_3/section_3/image9.png" class="common_img" />

### 3.3.3 Modify Recognition Color

Following will change the recognition color to yellow as example. The specific modification steps can be referenced as follows:

(1) Firstly, open the `"HSV.exe"` program file located in [Appendix-\> Color Recognition Routine & Tool->HSV->HSV.dist->HSV.exe](Appendix.md).

<img src="../_static/media/chapter_3/section_3/image10.png" class="common_img" />

(2) Click the img file to choose the image file that you want to import. (The imported image needs to be stored in the "img" folder.)

<img src="../_static/media/chapter_3/section_3/image11.png" class="common_img" />

(3) You can perform `HSV` threshold segmentation on the image by adjust the slider bar highlighted in the red box. Adjust it to a suitable `HSV` threshold range, referring to the color range table below for guidance.

<img src="../_static/media/chapter_3/section_3/image12.png" class="common_img" />

| **NO.** | **Actions** | **NO.** | **Actions** |
|:---:|:---:|:---:|:---:|
| 1 | Stand: stand() | 10 | Sit：sit() |
| 2 | Move a step forward: forward() | 11 | kick_ball_left/right() |
| 3 | Move backward: Step backwards: back() | 12 | Dance: moonwalk() |
| 4 | Turn left: turn_left() | 13 | Lie down: lie_down() |
| 5 | Turn right: turn_right() | 14 | Have a good stretch: temp() |
| 6 | Box: boxing() | 15 | Bow: bow() |
| 7 | Push up: push_up() | 16 | Kick red ball: kick_ball('red') |
| 8 | Handshake: shake_hands() | 17 | Trace red line: visual_patrol('red') |
| 9 | Nod: nod() |||

<img src="../_static/media/chapter_3/section_3/image13.png" class="common_img" />

(4) Next, save the `HSV` threshold values. Open the file located in [Color Recognition Routine and Tool -\> ColorDetection -\> color_detection.cpp](../_static/source_code/ColorDetection.zip). Modify the color data by replacing it with the saved `HSV` array. After that, follow [3.3.1 Flash Color Recognition Firmware](#anchor_3_3_1) to upload the modified program to the `ESPS3S2` module.

<img src="../_static/media/chapter_3/section_3/image14.png" class="common_img" />

:::{Note}
Please ensure that the format of the array elements is correct and that they are comma-separated.
:::

(5) After the program is flashed successfully, the `ESP32S3-Cam` will be able to recognize objects of other colors.