7. Multiple Controller UART Communication Course

7.1 Master-Slave Device Communication Principles

7.1.1 Introduction

This section aims to introduce you to detailed information about the master-slave relationship between the WonderMV and different devices (such as STM32, 51 microcontroller, Arduino, and Raspberry Pi) during communication. It helps you understand how WonderMV communicates as a subordinate device with other devices and how other devices control WonderMV as a master device.

Understand the position and function of WonderMV within the entire system. This will facilitate the effective integration of this module, realizing the desired image processing and recognition functions.

In this chapter, WonderMV is typically utilized as a subordinate device. It uses the UART serial port to transmit the information to other devices.

7.1.2 Master-Slave Relationship

In a master-slave control system, WonderMV acts as a subordinate device and other devices such as microcontrollers act as master devices.

WonderMV Functions

(1) Image processing and recognition

The intelligent algorithms need specialized vision processors to accelerate the processing. Microcontrollers like the 51 microcontroller or Arduino cannot easily achieve image processing and recognition. The WonderMV vision module is specifically designed to execute image processing and recognition algorithms. It has dedicated intelligent processing hardware units, which can accelerate the processing of results.

(2) Generation of recognition results

WonderMV is responsible for generating visual recognition results. This may include information about detected objects, shapes, colors, and more.

(3) Data encapsulation and communication

WonderMV encapsulates the recognition results into data packets according to a predefined communication protocol, enabling them to be transmitted to the master device.

Other Master Devices

(1) Control coordination

The master is responsible for coordinating the entire system, ensuring there are no conflicts in communication and operation between WonderMV and other devices to maintain a stable working state.

(2) Data reception

The master receives the recognition result sent by WonderMV via the UART serial port. It verifies the integrity and accuracy of the received data and parses the data packet for extracting useful information.

(3) Result processing The master device analyzes the relevant results and responds to the information. For example:

① If a 51 microcontroller development board receives recognition information of the red ball from WonderMV, the positional information of the ball will be printed.

② If a LeArm robotic arm receives face recognition information from WonderMV, the specific location of the face will be parsed to control the robotic arm to track the face.

③ If a Tonybot humanoid robot receives waste card recognition information from WonderMV, the recognized waste will be classified. The classification result will be broadcasted.

7.1.3 Hardware Connection

The wiring methods for communication between the WonderMV and other different master devices are as follows.

The wiring for the WonderMV and the Arduino development board.

The wiring for the WonderMV and the STM32 development board.

The wiring for the WonderMV and the ESP32 development board.

The wiring for the WonderMV and the Raspberry Pi.

7.1.4 Data Transmission Format

The default UART serial data transmission format of WonderMV is as follows:

Baud rate	Data bits	Parity	Stop bits
115200	8	None	1

7.1.5 Communication Protocol

Header	Function Code	Data Length	Data Information	CRC Checksum
0xAA 0x55	func	len	data	crc

Header: When the header is received, it indicates the presence of data transmission;

Function Code: It indicates the function of an information frame;

Data Length: It indicates the amount of data that follows;

Data Information: It indicates the transmitted data information;

CRC Checksum: Calculate all the data from the function code to the data information to obtain CRC checksum value, and retain only the lower 8 bits.

7.1.6 Note

(1) The host device and the WonderMV can have different power sources, but they must share a common ground when connected to ensure stable communication voltage.

(2) When you wire the devices, ensure that the TX and RX pins of the UART serial port are crossed, otherwise communication will not be possible.

7.2 Color Learning Recognition

7.2.1 K210

Game Introduction

In this section, program the K120 vision module to establish serial communication with the host device, enabling color acquisition and recognition. The coordinates of the recognized object’s bounding box and the corresponding result will be printed on the terminal. Additionally, the recognition result will be sent to the host device via the UART serial port.

Getting Ready

(1) Device Connection

Connect the K210 vision module to a computer with a Type-C data cable.

Power the other master devices with their respective power interfaces, and connect the UART interfaces with DuPont cables.

For example, connect the ESP32 to the PC with a USB data cable. Then, connect the K210 vision module to the ESP32’s serial port with a DuPont cable.

(2) Download and Run K210 Program

① Double-click to open the software.

② Click in the bottom left corner.

③ Select the corresponding serial port number. Check “Advanced Settings”, and select “Mode-3”.

④ Click “OK” and wait for the connection to be completed.

⑤ After the connection is successful, the CanMV IDE displays the icon in the bottom left corner.

⑥ If the connection takes more than 10 seconds, it indicates a connection failure. Click “Cancel”, and a pop-up window shown below appears. Click “OK” and re-check the connection.

Note

Tips for connection failure:

The selected serial port number is wrong.

Please unplug other serial ports connected to your PC. Try the above steps again to select the correct serial port number.

The cable used for the connection is not a data cable.

Use a Type-C data cable that supports data transmission. A Type-C data cable is included in the package before delivery.

Other K210 firmware is flashed.

Reflash the factory firmware, and then proceed with the connection.

There are two ways to run K210 programs: online and offline.

Online operation:

After connecting successfully, drag the program under the same folder as this lesson to the code editor area of CanMV IDE. Click the at the bottom left corner to run the program online.

Note

Programs run in this way will be lost when the connection is disconnected or the device is shut down. They will not be saved on the K210 vision module.

Offline operation:

① After connecting, drag the program under the same directory to the CanMV IDE code editing area. Click “Tools” on the toolbar. Select “Save open script to CanMV Cam (as main.py)”.

② Click “Yes”.

③ Once the writing is successful, a window shown below will appear. Click “OK”. This saves the MicroPython file into the K210 Vision Module.

By downloading in this way, after powering on the K210 Vision Module without connecting, it will run the MicroPython file, enabling offline operation.

(3) Download and Run Master Device Program

Please refer to the relevant tutorial to download and run the master device program.

Program Outcome

(1) K210 Vision Module

After downloading the program, place the object to be recognized, such as a red block, within the field of view of the vision module.

① Move the red block into the white box. Only the area within the box is used for color acquisition.

② Click the “K1” button on the module to aim the appearing green box at the object to be captured and recognized. Only the area within the central box is used for color acquisition.

③ Once the object is successfully recognized, the green box will disappear. The recognized red object will be highlighted with a white box and a cross.

If you want to change the color to be recognized, you can restart the K210 module to start a new capture. If you want to capture multiple colors (up to three colors are supported), please access the directory of this program and download or run “multi_color_recognition”. The capture method remains the same as described above.

(2) Mater Device

Upon the program is downloaded, the master device will print the detailed information that is received and parsed. Take the ESP32 as an example.

After downloading the program, the ESP32 will print the coordinates of the bounding box and the detection results in the terminal. Additionally, the message “crc ok” indicates that the checksum is correct.

Program Analysis

Source Code

Let’s focus on the program analysis for the K210 vision module on the transmitter end. For an analysis of the program on the receiver end, please refer to the 6. AI Vision Games -> 6.11 Object Recognition .

(1) Load the vision module to obtain camera images.

Load the image module for image buffering and barcode processing.

Load the time module for calculating video frame rates.

Load the LCD module library for displaying images on an LCD screen.

Load the hw_uart module from the hiwonder library for serial communication.

Load the binascii module for byte processing.

#载入相关模块(load relevant modules)
import sensor, image, time, lcd
from hiwonder import hw_uart
import time
import binascii

(2) Define the serial port transmission object.

#定义串口对象(define serial port object)
serial = hw_uart()

(3) Define a serial port transmission function. It encapsulates the recognized information based on the defined data format and transmits it via the serial port.

def send_data(x,y,w,h,msg):
    '''
    0xAA  0x55  功能号(function number)  数据长度(data length)  data  CRC
    '''
    START_1 = 0xAA
    START_2 = 0x55
    FUNC_num = FuncTag.COLOR_RECOGNITION #功能编号(function number)
    Length = 0  #数据长度(data length)
    crc = 0 #校验位(checksum bit)
    data = [] #数据组(dataset)

    #参数都为0(all parameters are set to 0)
    if x==0 and y==0 and w==0 and h ==0:
        pass
    else:
        #x(小端模式)(little-endian mode)
        low = x & 0xFF #低位(low byte)
        high = x >> 8& 0xFF #高位(high byte)
        data.append(low)
        data.append(high)

        #y(小端模式)(little-endian mode)
        low = y & 0xFF #低位(low byte)
        high = y >> 8& 0xFF #高位(high byte)
        data.append(low)
        data.append(high)
        
                #w(小端模式)(little-endian mode)
        low = w & 0xFF #低位(low byte)
        high = w >> 8& 0xFF #高位(high byte)
        data.append(low)
        data.append(high)

        #h(小端模式)(little-endian mode)
        low = h & 0xFF #低位(low byte)
        high = h >> 8& 0xFF #高位(high byte)
        data.append(low)
        data.append(high)

    #msg
    if msg != None:
        for i in range(len(msg)):
            msg_int = str_2_int(msg[i])
            data.append(msg_int)

    Length += len(data)
    send_buf = [FUNC_num,Length]
    for i in range(len(data)):
        send_buf.append(data[i])

    #进行CRC运算(perform a CRC calculation)
    crc = checksum_crc8(send_buf)

    send_buf.insert(0,START_1) #插入协议头1(insert protocol header 1)
    send_buf.insert(1,START_2) #插入协议头2(insert protocol header 1)
    send_buf.append(crc) #加入CRC校验码(incorporate the CRC checksum)

    print(send_buf) #打印数据(print data)
    serial.send_bytearray(send_buf) #发送数据(send data)

(4) Define the sent data storage variables. The send_x and send_y represent the x and y values of the top-left corner coordinates of the recognition bounding box. The send_msg represents the specific information extracted from the image.

#定义发送数据(define the transmitted data) 
send_x = 0
send_y = 0
send_w = 0
send_h = 0
send_msg = ""

(5) Initialize the LCD screen and the camera.

#初始化LCD(initialize LCD)
lcd.init()
#以下是初始化传感器(initialize sensors)
sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA)
sensor.skip_frames(time = 100)
sensor.set_auto_gain(False)
sensor.set_auto_whitebal(False)

(6) Define a clock counter for calculating frame rate.

#帧率时钟(frame rate clock)
clock = time.clock()

(7) Define parameters for the bounding box and display the initial bounding box.

#方框参数(box parameters)
r = [(320//2)-(50//2), (240//2)-(50//2), 50, 50]

#显示方框(display box)
for i in range(50):
    img = sensor.snapshot()
    img.draw_rectangle(r)
    lcd.display(img)

(8) Continuously obtain images and perform the average learning for the recognized color within the bounding box.

#开始学习(start learning)
print("Start learning ...")

threshold = [50, 50, 0, 0, 0, 0] # L、A、B 值(L, A, and B values)
for i in range(50):
    img = sensor.snapshot()
    hist = img.get_histogram(roi=r) # 获取图像的直方图(obtain the histogram of an image)
    lo = hist.get_percentile(0.01)  # 获取直方图在1%范围内的累积分布函数（根据需要进行调整）！(obtain the cumulative distribution function of the histogram within a 1% range (adjust as needed)!)
    hi = hist.get_percentile(0.99)  # 获取直方图在99%范围内的累积分布函数（根据需要进行调整）！(obtain the cumulative distribution function of the histogram within a 99% range (adjust as needed)!)

    # 将百分位值进行平均。(calculate the average of the percentile values)
    threshold[0] = (threshold[0] + lo.l_value()) // 2
    threshold[1] = (threshold[1] + hi.l_value()) // 2
    threshold[2] = (threshold[2] + lo.a_value()) // 2
    threshold[3] = (threshold[3] + hi.a_value()) // 2
    threshold[4] = (threshold[4] + lo.b_value()) // 2
    threshold[5] = (threshold[5] + hi.b_value()) // 2

    # 在图像中找到颜色块(locate the color block in the image)
    for blob in img.find_blobs([threshold], pixels_threshold=100, area_threshold=100, merge=True, margin=10):
        img.draw_rectangle(blob.rect()) # 在颜色块的位置绘制矩形(draw a rectangle at the location of the color block)
        img.draw_cross(blob.cx(), blob.cy()) # 在颜色块的中心位置绘制十字(draw a cross at the center of the color block)
        img.draw_rectangle(r, color=(0,255,0)) # 在 ROI 区域绘制矩形，标记学习的区域(draw a rectangle in the ROI region to mark the learned region)

    # 在 LCD 屏幕上显示图像(display image on the LCD screen)
    lcd.display(img)

(9) After the learning process is completed, enter the main loop.

#loop
while(True):

(10) Obtain the image from the sensor and perform color recognition.

    #从传感器捕获一张图像(capture an image from the sensor)
    img = sensor.snapshot()

(11) Locate the corresponding color threshold area within the recognized image. Once the corresponding color area is detected, enclose it with a bounding box on the image.

    #遍历图像中找到的颜色区块(iterate through the color blocks found in the image)
    for blob in img.find_blobs([threshold], pixels_threshold=100, area_threshold=100, merge=True, margin=10):
        #绘制矩形和十字标记(draw a rectangle and cross)
        img.draw_rectangle(blob.rect())
        img.draw_cross(blob.cx(), blob.cy())
        #将方框数据与消息赋值(assign values to box data and message)
        send_x = blob.x()
        send_y = blob.y()
        send_w = blob.w()
        send_h = blob.h()
        send_msg = 'u'

(12) Upon detecting information, use the send_data() function to transmit the information via the serial port.

    # 当消息标志为 'u' 时，发送数据（注意，即使x、y、w、h数据不变也会进入此条件）(When the message flag is set to 'u', transmit the data (note that this condition is triggered even if the x, y, w, and h data remain unchanged))
    if send_msg == 'u':
        #发送数据(send data)
        send_data(send_x,send_y,send_w,send_h,None)
        send_msg = ""  # 清空消息变量(clear message variables)
    lcd.display(img) #显示在LCD上(display on the LCD)
    print(clock.fps()) #打印帧率(print the frame rate)

(13) Display the image on the LCD screen and print the frame rate.

    lcd.display(img) #显示在LCD上(display on the LCD)
    print(clock.fps()) #打印帧率(print the frame 

7.2.2 Arduino UNO

For the instructions on downloading the program to K210 and analyzing the K210-related code, please refer to the “K210” located in the same directory as this tutorial. This lesson only analyzes the data transmitted by K210 and received by Arduino UNO.

Game Introduction

Send the learned color data to Arduino UNO via the K210 vision module.

Getting Ready

(1) Module Connection

① Connect the K210 vision module to a computer with a Type-C data cable.

② Connect the GND, RX, and TX pins of the K210’s UART interface to the corresponding GND, D6, and D7 pins on the Arduino UNO with a DuPont wire.

(2) Download and Run Program

① Locate and open “color_recognition” program file in the same folder as this lesson.

② In the “Select board” option, select the corresponding Arduino development board and port. Take “Arduino Uno” and “COM8” as an example to demonstrate. The COM port is not unique. You can check the COM port number in the computer’s device manager.

③ Click button to compile the program. After the compilation is completed, click to upload the program into the Arduino development board.

Program Outcome

After downloading the program, place the object to be recognized, such as a red block, within the field of view of the vision module.

① Move the red block into the white box. Only the area within the box is used for color acquisition.

② Click the “K1” button on the module to aim the appearing green box at the object to be captured and recognized. Only the area within the central box is used for color acquisition.

③ Once the object is successfully recognized, the green box will disappear. The recognized red object will be highlighted with a white box and a cross.

If you want to change the color to be recognized, you can restart the K210 module to start a new capture.

Open the Arduino IDE, the coordinates information of the red block is visible. Select the baud rate to 115200.

The touch screen of K210 vision module has a resolution of 320*240. The origin of its pixel coordinates located at the top left corner of the module, as illustrated in the diagram below:

The x and y represent the pixel coordinates of the red block along the X and Y axes, respectively. The h and w denote the height and width of the block in units of px.

Program Analysis

Source Code

(1) Initialize program

The program begins by initializing the communication through the serial port, with the baud rate set to 115200. This establishes a connection for communication with the K210 controller via the serial port.

void setup() {
  Serial.begin(115200);
  //开启串口接收(enable serial port receiving)
  wk.serial_begin();  
}

The function Wk.serial_begin() is used to initialize the serial communication on the K210, enabling the MCU to establish a connection with the K210 through serial communication.

(2) Loop to obtain the latest color recognition

void loop() {
  //更新消息(updata message)
  wk.update_data();
  //获取消息(obtain message)
  bool rt = wk.recive_box(&result , K210_COLOR_RECOGNITION);
  //若获取到新的消息(if a new message is obtained)

After entering the loop() function, use the wk.update_data() function to continuously obtain the latest color recognition results from the K210 vision module. Then, acquire and parse the data of the latest recognized frame’s structure, storing it in the result variable.

(3) Print result information

If a new result is obtained (rt is true), the serial port will print detailed coordinates and dimensions of the recognition frame.

void loop() {
  //更新消息(updata message)
  wk.update_data();
  //获取消息(obtain message)
  bool rt = wk.recive_box(&result , K210_COLOR_RECOGNITION);
  //若获取到新的消息(if a new message is obtained)
  if(rt)
  {
    Serial.print("x:");
    Serial.println(result.x);
    Serial.print("y:");
    Serial.println(result.y);
    Serial.print("w:");
    Serial.println(result.w);
    Serial.print("h:");
    Serial.println(result.h);
  }
}

7.2.3 ESP32

Game Introduction

In this section, program the K120 vision module to establish serial communication with the ESP32, enabling color learning recognition. The coordinates of the recognized object’s bounding box will be printed on the terminal.

Getting Ready

(1) Wiring Instruction

① Connect the K210 vision module to a computer with a Type-C data cable.

② Connect the ESP32 to the PC with a USB data cable. Then, connect the K210 vision module to the ESP32’s serial port with a 4PIN wire.

(2) Download and Run K210 Program

① Double-click to open the software.

② Click in the bottom left corner.

③ Select the corresponding serial port number. Check “Advanced Settings”, and select “Mode-3”.

④ Click “OK” and wait for the connection to be completed.

⑤ After the connection is successful, the CanMV IDE displays the icon in the bottom left corner.

⑥ If the connection takes more than 10 seconds, it indicates a connection failure. Click “Cancel”, and a pop-up window shown below appears. Click “OK” and re-check the connection.

Note

Tips for connection failure:

The selected serial port number is wrong.

Please unplug other serial ports connected to your PC. Try the above steps again to select the correct serial port number.

The cable used for the connection is not a data cable.

Use a Type-C data cable that supports data transmission. A Type-C data cable is included in the package before delivery.

Other K210 firmware is flashed.

Reflash the factory firmware, and then proceed with the connection.

There are two ways to run K210 programs: online and offline.

Online operation:

After connecting successfully, drag the program under the same folder as this lesson to the code editor area of CanMV IDE. Click the at the bottom left corner to run the program online.

Note

Programs run in this way will be lost when the connection is disconnected or the device is shut down. They will not be saved on the K210 vision module.

Offline operation:

① After connecting, drag the program under the same directory to the CanMV IDE code editing area. Click “Tools” on the toolbar. Select “Save open script to CanMV Cam (as main.py)”.

② Click “Yes”.

③ Once the writing is successful, a window shown below will appear. Click “OK”. This saves the MicroPython file into the K210 Vision Module.

By downloading in this way, after powering on the K210 Vision Module without connecting, it will run the MicroPython file, enabling offline operation.

(3) Download and Run ESP32 Program

① Connect the ESP32 controller to the computer with a USB data cable. Open the Python editor. Drag the programs “WonderK210.py” and “main.py” into the editor interface.

② Select the or click “Connect” in the menu bar to choose the corresponding COM port. Let’s use “COM15” as an example. The COM port is not unique. You can check the COM port number in the computer’s device manager.

Note

To avoid encountering any unexpected anomalies during the flashing process, please strictly follow the operation steps.

In you encounter difficulties in properly reading the COM port or downloading the program, please refer to the “ESP32 Board Firmware Recovery Method” to reflash the firmware.

③ After the connection is established, click or “Run - Download and Run” to download the program into the controller and execute it.

Program Outcome

Print the coordinates of the bounding box for the recognized object in the terminal.

Program Analysis

Source Code

Let’s focus on the ESP32 program analysis on the receiver end. For an analysis of the K210 program on the transmitted end, please refer to the 6. AI Vision Games -> 6.1 Color Acquisition and Recognition .

(1) Import Module

Import the time module, the WonderK210 communication class, and the message format classes ( Find_Box_Msg_st, Find_Box_st, and Find_Msg_st) for subsequent usage.

# 载入时间模块(load time module)
import time
# 载入K210通信类(load K210 communication class)（WonderK210），消息格式类(message format class)（Find_Box_Msg_st , Find_Box_st , Find_Msg_st）
from WonderK210 import WonderK210 , Find_Box_Msg_st , Find_Box_st , Find_Msg_st

(2) Establish Communication

Instantiate the K210 communication class.

# 实例化K210通信类(instantiate K210 communication class)
wk = WonderK210()

Enable the serial port.

# 开启串口(start serial port)
wk.serial_begin()

(3) Main Function

Enter a while loop to continuously update and receive messages. The function wk.find_color_recognition() searches for specific color in the data and retrieve the received messages.

while True:
  # 更新接收消息(update receiving message)
  wk.update_data()
  # 获取接收消息(obtain receiving message)
  ret = wk.find_color_recognition()

If a recognition result is found (ret is not None), the coordinates of the bounding box for the recognized object will be printed. The x and y represent the top-left coordinates of the box in the image, the w denotes the width of the box, and the h represents the height of the box.

  # 若有新消息获取到(if a new message is obtained)
  if ret is not None:
    # 打印消息内容(print message content)
    print(ret.x)
    print(ret.y)
    print(ret.w)
    print(ret.h)
  # 延时10ms(delay for 10ms)
  time.sleep_ms(10)

(4) Information Obtaining Function

  # 颜色学习识别 box(color learning recognition box)
  def find_color_recognition(self):
    if True == self.__read_succeed: # 若有新消息(if there is a new message)
      if k210_PACKET_FUNCTION.COLOR_RECOGNITION == self.__pk_result.function: # 是否为该功能消息(if it is the function message)
        print("find_color_recognition()")
        self.__read_succeed = False
        ret = Find_Box_st()
        ret.function = self.__pk_result.function
        ret.x = (self.__pk_result.data[0] & 0x00FF) | ((self.__pk_result.data[1] << 8 ) & 0xFF00)
        ret.y = (self.__pk_result.data[2] & 0x00FF) | ((self.__pk_result.data[3] << 8 ) & 0xFF00)
        ret.w = (self.__pk_result.data[4] & 0x00FF) | ((self.__pk_result.data[5] << 8 ) & 0xFF00)
        ret.h = (self.__pk_result.data[6] & 0x00FF) | ((self.__pk_result.data[7] << 8 ) & 0xFF00)
        # 清除存储变量(clear storage variables)
        self.__pk_result.data.clear()
        return ret
      else:
        return None
    else:
      return None

When a new message of the color recognition is received, this function will extract the position information of the recognized color box.

Firstly, it checks if the self.__read_succeed is True. If the message reading is successful, it then verifies if the self.__pk_result.function is the predefined COLOR_RECOGNITION function message. If it is a color recognition message, the following steps are performed:

① Print a message indicating that a color recognition message has been found.

② Set the self.__read_succeed to False, indicating that this message has been successfully processed.

③ Create an object named ret, which is an instance of the Find_Box_st class.

④ Extract the coordinate and dimension information from the color recognition result, and assign them to the corresponding attributes (x, y, w, and h) of the ret object.

⑤ Clear the variable self.__pk_result.data that stores message data.

⑥ Return the ret object containing the extracted recognition result information.

⑦ If a new message is not successfully read or the received message is not a color recognition message, return None.

7.2.4 STM32

For the instructions on downloading the program to K210 and analyzing the K210-related code, please refer to the “K210” located in the same directory as this tutorial. This lesson only analyzes the data transmitted by K210 and received by STM32.

Game Introduction

Send the learned color data to STM32 via the K210 vision module.

Getting Ready

(1) Module Connection

Connect the K210 vision module to a computer with a Type-C data cable.

Connect the GND, RX, and TX pins of the K210’s UART interface to the corresponding pins on the STM32 with a DuPont wire.

(2) Download and Run Program

① Locate and open “RosRobotControllerM4.uvprojx” program file in the same folder as this lesson.

② Click to perform compiling. After the compiling is completed, click to download the program to the STM32 board.

Program Outcome

Once the program runs, the STM32 will obtain the coordinates of the learned color from the K210 vision module. After running the program, open the located in Appendix->Serial Port Utility to view the printed information.

Program Analysis

Source Code

(1) Initial Settings

#include "k210_packet.h"
#include "global.h"

The “k210_packet.h” consists of data structure definitions, function declarations, and macros related to the K210. It is used to handle data packets, communication protocols, or device control related to the K210. The file “global.h” contains global constants, variables, and more.

typedef struct {
	uint16_t  x;
	uint16_t  y;
	uint16_t  w;
	uint16_t  h;
}BoxData;

Struct BoxData: a structure BoxData is defined. It contains four uint16_t type member variables: x, y, w, and h. These variables represent the position and size of a rectangular box.

//只有方框数据(only includes box data)
typedef struct {
	BoxData box;
}Find_Box_st;

Struct Find_Box_st: a structure Find_Box_st is defined. It contains a variable box of BoxData type. This structure represents a data type that includes only the information of a rectangular box, excluding other message data.

//有方框数据和消息数据(include box data and message data)
typedef struct {
	BoxData box;
	uint8_t msg[100];
}Find_Box_Msg_st;

Struct Find_Box_Msg_st: a structure Find_Box_Msg_st is defined. It contains a variable of type BoxData named box and an array of uint8_t type with a length of 100 named msg. This structure represents a data type that includes both the information of a rectangular box and a message data with a length of 100.

//只有消息数据(only includes message data)
typedef struct {
	uint8_t msg[100];
}Find_Msg_st;

Struct Find_Msg_st: a structure Find_Msg_st is defined. It contains a single array of uint8_t type with a length of 100 named msg. This structure represents a data type that includes only the message data with a length of 100 and does not include the information of a rectangular box.

(2) Message Data Processing

/**
* @brief 只有方框数据 的回调处理(callback processing for only including box data)
* @param frame 数据帧(data frame)
* @retval void
*/
static void k210_Box_cb(struct k210_PacketRawFrame *frame)
{
    Find_Box_st *rec = (Find_Box_st*)frame->data_and_checksum;
	printf("x:%d,y:%d,w:%d,h:%d\n",rec->box.x,rec->box.y,rec->box.w,rec->box.h);
}

k210_Box_cb: handles data frames that only contain box data. It extracts data of type Find_Box_st from the data frame. It prints the position and size information of the box based on the extracted data.

/**
* @brief 有方框数据和消息数据 的回调处理(callback processing for including box data and message data)
* @param frame 数据帧(data frame)
* @retval void
*/
static void k210_Box_Msg_cb(struct k210_PacketRawFrame *frame)
{
    Find_Box_Msg_st *rec = (Find_Box_Msg_st*)frame->data_and_checksum;
	printf("x:%d,y:%d,w:%d,h:%d\n",rec->box.x,rec->box.y,rec->box.w,rec->box.h);
	rec->msg[frame->data_length - 8] = '\0'; //将check位清除(clear check bit)
	printf("msg:%s\n",rec->msg);
}

k210_Box_Msg_cb: handles data frames that contain both box data and message data.

It extracts data of type Find_Box_Msg_st from the data frame. It prints the position and size information of the box based on the extracted data. It clears the checksum of the message data and then prints the message content.

/**
* @brief 只有消息数据 的回调处理(callback processing for only including message data)
* @param frame 数据帧(data frame)
* @retval void
*/
static void k210_Msg_cb(struct k210_PacketRawFrame *frame)
{
    Find_Msg_st *rec = (Find_Msg_st*)frame->data_and_checksum;
	rec->msg[frame->data_length] = '\0'; //将check位清除(clear check bit)
	printf("msg:%s\n",rec->msg);
}

k210_Msg_cb: handles data frames that only contain message data. It extracts data of type Find_Msg_st from the data frame. It clears the checksum of the message data and then prints the message content. These functions serve as callback functions. When receiving specific types of data frames, these functions are called to process the data within the frames.

(3) Callback Function

//注册回调函数(register callback function)
void k210_packet_handle_init(void)
{
    k210_packet_register_callback(&k210_packet_controller, K210_COLOR_RECOGNITION, k210_Box_cb); //颜色识别(color recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_BARCODES, k210_Box_Msg_cb); //条形码识别(barcode recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_QRCODES, k210_Box_Msg_cb);	 //二维码识别(QR code recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_APRILTAGS, k210_Box_Msg_cb);//机器码识别(AprilTag recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_FACE_YOLO, k210_Box_cb); 	 //人脸检测(face detection)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_FACEFEATURE, k210_Box_cb);  //人脸特征识别(face feature recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_FACEMASK, k210_Box_Msg_cb); //口罩识别(mask recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_OBJECT, k210_Box_Msg_cb);	 //物体识别(object recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_SELF_LEARNING, k210_Msg_cb);//自主学习分类(autonomous learning and classification)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_DIGITAL, k210_Msg_cb);      //手写数字识别(handwritten number recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_FACE_RECOGNITION, k210_Box_Msg_cb); //物体识别(object recognition)
    
    k210_packet_register_callback(&k210_packet_controller, K210_FIND_RED_FOLLOW, k210_Box_cb); //红色追踪(red tracking)
    k210_packet_register_callback(&k210_packet_controller, K210_FIND_SIGNPOST_FOLLOW, k210_Box_Msg_cb); //路标追踪(road sign tracking)
    k210_packet_register_callback(&k210_packet_controller, K210_FIND_DIGITAL_CARD, k210_Box_Msg_cb); //数字卡片识别(number card recognition)
    k210_packet_register_callback(&k210_packet_controller, K210_GARBAGE_SORTING, k210_Box_Msg_cb); //垃圾分类(waste classification)
    k210_packet_register_callback(&k210_packet_controller, K210_COLOR_SORTING, k210_Box_Msg_cb); //多种颜色识别(multiple colors recognition)
}

Multiple callback functions are registered through the function k210_packet_handle_init. Each callback function corresponds to a different processing program. These callback functions are associated with different functional modules related to the K210. Each module corresponds to a different recognition or data processing task.

Each call to k210_packet_register_callback associates a specific task’s processing function with the K210 controller. For example:

① K210_COLOR_RECOGNITION corresponds to the color recognition task and uses the function k210_Box_cb to process the data.

② K210_FIND_BARCODES corresponds to the barcode recognition task and uses the function k210_Box_Msg_cb to process the data.

Similar cases exist for other tasks. Take k210_packet_register_callback(&k210_packet_controller, K210_COLOR_RECOGNITION, k210_Box_cb) as an example:

① The k210_packet_register_callback function is used to associate the callback function k210_Box_cb with a specific task type K210_COLOR_RECOGNITION.

② &k210_packet_controller is a pointer to the K210 controller. It provides interfaces and methods for handling different tasks.

③ K210_COLOR_RECOGNITION is a specific function code. It represents the function that K210 needs to perform.

④ k210_Box_cb is a callback function. It will be called when data associated with the task type K210_COLOR_RECOGNITION is received.

Each callback function receives data frames from the K210 controller and uses specific processing functions based on the task type to handle the data.

7.2.5 Raspberry Pi

For the instructions on downloading the program to K210 and analyzing the K210-related code, please refer to the “K210” located in the same directory as this tutorial. This lesson only analyzes the data transmitted by K210 and received by Raspberry Pi.

Game Introduction

Send the learned color data to Raspberry Pi via the K210 vision module.

Getting Ready

(1) Module Connection

① Connect the K210 vision module to a computer with a Type-C data cable.

② Connect the GND, RX, and TX pins of the K210’s UART interface to the corresponding pins on the Raspberry Pi with a DuPont wire.

(2) Download and Run Program

① Move the program files “main.py” and “WonderK210.py” from the current directory to Raspberry Pi via a USB drive.

② Press “Ctrl+Alt+T” in the Raspberry Pi system to open the command line terminal.

③ In the command line terminal, enter the command to navigate to the directory where the program is located:

cd Desktop/

④ Enter the command to start the program:

python3 main.py

Program Outcome

Once the program is started, the Raspberry Pi will retrieve the coordinate information of learned color from the K210 vision module and print it in the terminal.

Program Analysis

Source Code

(1) Initial Settings

# 载入时间模块(load time module)
import time
# 载入K210通信类(load K210 communication class)（WonderK210），消息格式类(message format class)（Find_Box_Msg_st , Find_Box_st , Find_Msg_st），K210识别类型类（k210_PACKET_FUNCTION）
from WonderK210 import WonderK210 , Find_Box_Msg_st , Find_Box_st , Find_Msg_st , k210_PACKET_FUNCTION

Load the K210 communication class (WonderK210), message format class (Find_Box_Msg_st, Find_Box_st, Find_Msg_st), and K210 recognition type class (k210_PACKET_FUNCTION).

(2) Create Object

Instantiate the K210 communication class, and simultaneously activate the serial port.

# 实例化K210通信类(instantiate K210 communication class)
wk = WonderK210()

# 开启串口(start serial port)
wk.serial_begin()

(3) Main Function

Use the function wk.update_dat() to update data. Call the function wk.find_box() to obtain the received message.

When a new message is obtained, the contents related to the target color will be printed.

#loop
while True:
    # 更新接收消息(update receiving message)
    wk.update_data()
    # 获取接收消息(obtain receiving message)
    ret = wk.find_box(k210_PACKET_FUNCTION.COLOR_RECOGNITION)
    # 若有新消息获取到(if a new message is obtained)
    if ret is not None:
        # 打印消息内容(print message content)
        print("ret.x = {}".format(ret.x))
        print("ret.y = {}".format(ret.y))
        print("ret.w = {}".format(ret.w))
        print("ret.h = {}".format(ret.h))

7.3 Multiple Colors Recognition

7.3.1 K210

Game Introduction

In this program, the color thresholds have been predefined. It can directly recognize five colors: red, green, blue, yellow, and purple without the need of capturing. When these five colors are recognized, they will be highlighted in green boxes. The recognized data will be sent to the master device via the serial port.

Getting Ready

(1) Device Connection

Connect the K210 vision module to a computer with a Type-C data cable.

Power the other master devices with their respective power interfaces, and connect the UART interfaces with DuPont cables.

For example, connect the ESP32 to the PC with a USB data cable. Then, connect the K210 vision module to the ESP32’s serial port with a DuPont cable.

(2) Download and Run K210 Program

① Double-click to open the software.

② Click in the bottom left corner.

③ Select the corresponding serial port number. Check “Advanced Settings”, and select “Mode-3”.

⑤ Click “OK” and wait for the connection to be completed.

⑥ After the connection is successful, the CanMV IDE displays the icon in the bottom left corner.

⑦ If the connection takes more than 10 seconds, it indicates a connection failure. Click “Cancel”, and a pop-up window shown below appears. Click “OK” and re-check the connection.

Note

Tips for connection failure:

The selected serial port number is wrong.

Please unplug other serial ports connected to your PC. Try the above steps again to select the correct serial port number.

The cable used for the connection is not a data cable.

Use a Type-C data cable that supports data transmission. A Type-C data cable is included in the package before delivery.

Other K210 firmware is flashed.

Reflash the factory firmware, and then proceed with the connection.

There are two ways to run K210 programs: online and offline. Online operation:

After connecting successfully, drag the program under the same folder as this lesson to the code editor area of CanMV IDE. Click the at the bottom left corner to run the program online.

Note

Programs run in this way will be lost when the connection is disconnected or the device is shut down. They will not be saved on the K210 vision module.

Offline operation:

① After connecting, drag the program under the same directory to the CanMV IDE code editing area. Click “Tools” on the toolbar. Select “Save open script to CanMV Cam (as main.py)”.

② Click “Yes”.

③ Once the writing is successful, a window shown below will appear. Click “OK”. This saves the MicroPython file into the K210 Vision Module.

By downloading in this way, after powering on the K210 Vision Module without connecting, it will run the MicroPython file, enabling offline operation.

Program Outcome

(1) K210 Vision Module

After running the program, the K210 vision module will select the recognized objects in the color of red, green, blue, yellow, and purple with boxes, as shown below:

(2) Mater Device

Upon the program is downloaded, the master device will print the detailed information that is received and parsed. Take the ESP32 as an example.

After downloading the program, the ESP32 will print the coordinates of the bounding box and the detection results in the terminal. Additionally, the message “crc ok” indicates that the checksum is correct.

Program Analysis

Source Code

(1) Load the hw_uart module from the hiwonder library for serial communication.

Load the vision module to obtain camera images.

Load the image module for image buffering and barcode processing.

Load the time module for calculating video frame rates.

Load the LCD module library for displaying images on an LCD screen.

Load the binascii module for byte processing.

from hiwonder import hw_uart
import sensor
import image
import time
import lcd
import binascii

(2) Define the serial port transmission object.

#定义串口对象(define serial port object)
serial = hw_uart()

(3) Define a serial port transmission function. It encapsulates the recognized information based on the defined data format and transmits it via the serial port.

def send_data(x,y,w,h,msg):
    '''
    0xAA  0x55  功能号  数据长度  data  CRC
    '''
    START_1 = 0xAA
    START_2 = 0x55
    FUNC_num = FuncTag.COLOR_RECOGNITION #功能编号(function number)
    Length = 0  #数据长度(data length)
    crc = 0 #校验位(checksum bit)
    data = [] #数据组(dataset)

    #参数都为0(all parameters are set to 0)
    if x==0 and y==0 and w==0 and h ==0:
        pass
    else:
        #x(小端模式)(little-endian mode)
        low = x & 0xFF #低位(low byte)
        high = x >> 8& 0xFF #高位(high byte)
        data.append(low)
        data.append(high)

        #y(小端模式)(little-endian mode)
        low = y & 0xFF #低位(low byte)
        high = y >> 8& 0xFF #高位(high byte)
        data.append(low)
        data.append(high)

        #w(小端模式)(little-endian mode)
        low = w & 0xFF #低位(low byte)
        high = w >> 8& 0xFF #高位(high byte)
        data.append(low)
        data.append(high)

        #h(小端模式)(little-endian mode)
        low = h & 0xFF #低位(low byte)
        high = h >> 8& 0xFF #高位(high byte)
        data.append(low)
        data.append(high)
        
            #msg
    if msg != None:
        for i in range(len(msg)):
            msg_int = str_2_int(msg[i])
            data.append(msg_int)

    Length += len(data)
    send_buf = [FUNC_num,Length]
    for i in range(len(data)):
        send_buf.append(data[i])

    #进行CRC运算(perform a CRC calculation)
    crc = checksum_crc8(send_buf)

(4) Define the sent data storage variables. The send_x and send_y represent the x and y values of the top-left corner coordinates of the recognition bounding box. The send_msg represents the specific information extracted from the image.

#定义发送数据(define the transmitted data) 
send_x = 0
send_y = 0
send_w = 0
send_h = 0
send_msg = ""

(5) Initialize the LCD screen. Restart the camera module. Set its pixel mode to RGB565 and the image size to QVGA. Disable automatic gain and automatic white balance.

#初始化LCD(initialize LCD)
lcd.init()
#以下是初始化传感器(initialize sensors)
sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA)
sensor.skip_frames(time = 100)
sensor.set_auto_gain(False)
sensor.set_auto_whitebal(False)

(6) Define a clock object for calculating frame rate.

#帧率时钟(frame rate clock)
clock = time.clock()

(7) Set LAB thresholds for recognizing different colors. Store LAB thresholds for five colors. Define a color string for displaying the recognized color name.

# 储存6种颜色的LAB阈值(store the LAB threshold of the six colors)
color_thresholds = [
    (15,80,15,62,15,35), # Red
    (14, 61, -39, -6, 0, 14),  # Green
    (21,50,-7,8,-35,-11), # Blue
    (65, 78, -10, -5, 38, 50),     # Yellow
    (20, 50, 17, 37, -34, -14),      # purple
]

color_strings = ['Red', 'Green', 'Blue', 'Yellow', 'purple']

(8) Create a color recognition loop:

The program enters an infinite loop, capturing an image in each iteration.

Use the find_blobs method to iterate through all the defined color thresholds and identify color regions in the image.

For each recognized color, draw a rectangle and a cross, display the color name next to it, and send out the recognized data via the UART serial port.

Display the processed image on the LCD screen and print the current frame rate.

#loop
while True:
    #用于计算帧率的函数，这里表示开始计时(The function for calculating the frame rate, indicating the start of timing)
    #clock.tick()
    #从传感器捕获一张图像(capture an image from the sensor)
    img = sensor.snapshot()
    #遍历多种颜色(iterate through multiple colors)
    for color_idx, threshold in enumerate(color_thresholds):
        #遍历图像中找到的颜色区块(iterate through the color blocks found in the image)
        blobs = img.find_blobs([threshold], pixels_threshold=100, area_threshold=100, merge=True, margin=10)
        #是否找到(whether it is found)
        if blobs:
            area_max = 0
            blob_max = blobs[0]
            #寻找最大的颜色块(find the largest color block)
            for blob in blobs:
                area = blob.w()*blob.h()
                if(area_max < area):
                    blob_max = blob
                    area_max = area
            if area_max < 2000:
                continue
            #画方框(draw box)
            img.draw_rectangle(blob_max.rect())
            img.draw_cross(blob_max.cx(), blob_max.cy())
            img.draw_string(blob_max.cx() + 10, blob_max.cy() - 10, color_strings[color_idx], color=(255, 255, 255))
            #将方框数据与消息赋值(assign values to box data and message)
            send_x = blob_max.x()
            send_y = blob_max.y()
            send_w = blob_max.w()
            send_h = blob_max.h()
            #发送数据(send data)
            send_data(send_x,send_y,send_w,send_h,color_strings[color_idx])
    #显示在LCD上(display on the LCD)
    lcd.display(img)
    #打印帧率(print the frame rate)
    #print(clock.fps())

7.3.2 Arduino UNO

For the instructions on downloading the program to K210 and analyzing the K210-related code, please refer to the “K210” located in the same directory as this tutorial. This lesson only analyzes the data transmitted by K210 and received by Arduino UNO.

Game Introduction

Send the recognized color data to Arduino UNO via the K210 vision module.

Getting Ready

(1) Module Connection

Connect the K210 vision module to a computer with a Type-C data cable.

Connect the GND, RX, and TX pins of the K210’s UART interface to the corresponding GND, D6, and D7 pins on the Arduino UNO with a DuPont wire.

(2) Download and Run Program

① Locate and open the “5.3.2 find_multi_color.ino” program file in the same folder as this lesson.

② In the “Select board” option, select the corresponding Arduino development board and port. Take “Arduino Uno” and “COM8” as an example to demonstrate. The COM port is not unique. You can check the COM port number in the computer’s device manager.

③ Click button to compile the program. After the compilation is completed, click to upload the program into the Arduino development board.

Program Outcome

After running the program, the K210 vision module will select the recognized objects in the color of red, green, blue, yellow, and purple with boxes, as shown below:

Open the Arduino IDE serial monitor, the coordinates information of the recognized block is visible. Select the baud rate to 115200.

The touch screen of K210 vision module has a resolution of 320*240. The origin of its pixel coordinates located at the top left corner of the module, as illustrated in the diagram below:

The msg represents the color name of the recognized target. If green is recognized, print “Green”. The x and y represent the pixel coordinates of the target’s center along the X and Y axes, respectively. The h and w denote the height and width of the block in units of px.

Program Analysis

Source Code

(1) Initial Settings

The program begins by initializing the communication through the serial port, with the baud rate set to 115200. This establishes a connection for communication with the K210 controller via the serial port.

void setup() {
  Serial.begin(115200);
  //分配内存(allocate memory)
  wk = malloc(sizeof(WonderK210));
  //开启串口接收(enable serial port receiving)
  //这里不需要再调用初始化串口函数 Serial.begin(115200); 在 serial_begin() 函数里已经初始化了(Since the initialization of the serial port has already been performed within the serial_begin() function, there is no need to call Serial.begin(115200))
  wk->serial_begin();  
  //分配内存(allocate memory)
  result = malloc(sizeof(Find_Box_Msg_st));
}

The function Wk.serial_begin() is used to initialize the serial communication on the K210, enabling the MCU to establish a connection with the K210 through serial communication.

(2) Loop to obtain the latest color recognition

void loop() {
  //更新消息(updata message)
  wk.update_data();
  //获取消息(obtain message)
  bool rt = wk.recive_box(&result , K210_COLOR_RECOGNITION);

After entering the loop() function, use the wk.update_data() function to continuously obtain the latest color recognition results from the K210 vision module. Then, acquire and parse the data of the latest recognized frame’s structure, storing it in the result variable.

(3) Printing result information

If a new result is obtained (rt is true), the serial port will print detailed coordinates and dimensions of the recognition frame.

void loop() {
  //更新消息(updata message)
  wk.update_data();
  //获取消息(obtain message)
  bool rt = wk.recive_box(&result , K210_COLOR_RECOGNITION);
  //若获取到新的消息(if a new message is obtained)
  if(rt)
  {
    Serial.print("x:");
    Serial.println(result->x);
    Serial.print("y:");
    Serial.println(result->y);
    Serial.print("w:");
    Serial.println(result->w);
    Serial.print("h:");
    Serial.println(result->h);
    Serial.print("msg:");
    Serial.println(result->msg);
  }
}

7.3.3 ESP32

Game Introduction

In this section, program the K120 vision module to establish serial communication with the ESP32, enabling multiple colors recognition. The coordinates of the recognized color and recognition result will be printed on the terminal.

Getting Ready

(1) Wiring Instruction

Connect the K210 vision module to a computer with a Type-C data cable.

Connect the ESP32 to the PC with a USB data cable. Then, connect the K210 vision module to the ESP32’s serial port with a 4PIN wire.

(2) Download and Run Program

① Download and Run K210 Program

② Double-click to open the software.

③ Click in the bottom left corner.

④ Select the corresponding serial port number. Check “Advanced Settings”, and select “Mode-3”.

⑤ Click “OK” and wait for the connection to be completed.

⑥ After the connection is successful, the CanMV IDE displays the icon in the bottom left corner.

⑦ If the connection takes more than 10 seconds, it indicates a connection failure. Click “Cancel”, and a pop-up window shown below appears. Click “OK” and re-check the connection.

Note

Tips for connection failure:

The selected serial port number is wrong.

Please unplug other serial ports connected to your PC. Try the above steps again to select the correct serial port number.

The cable used for the connection is not a data cable.

Use a Type-C data cable that supports data transmission. A Type-C data cable is included in the package before delivery.

Other K210 firmware is flashed.

Reflash the factory firmware, and then proceed with the connection.

There are two ways to run K210 programs: online and offline.

Online operation:

After connecting successfully, drag the program under the same folder as this lesson to the code editor area of CanMV IDE. Click the at the bottom left corner to run the program online.

Note

Programs run in this way will be lost when the connection is disconnected or the device is shut down. They will not be saved on the K210 vision module.

Offline operation:

① After connecting, drag the program under the same directory to the CanMV IDE code editing area. Click “Tools” on the toolbar. Select “Save open script to CanMV Cam (as main.py)”.

② Click “Yes”.

③ Once the writing is successful, a window shown below will appear. Click “OK”. This saves the MicroPython file into the K210 Vision Module.

By downloading in this way, after powering on the K210 Vision Module without connecting, it will run the MicroPython file, enabling offline operation.

(3) Download and Run ESP32 Program

① Connect the ESP32 controller to the computer with a USB data cable. Open the Python editor. Drag the programs “WonderK210.py” and “main.py” into the editor interface.

② Select the or click “Connect” in the menu bar to choose the corresponding COM port. Let’s use “COM15” as an example. The COM port is not unique. You can check the COM port number in the computer’s device manager.

Note

To avoid encountering any unexpected anomalies during the flashing process, please strictly follow the operation steps.

In you encounter difficulties in properly reading the COM port or downloading the program, please refer to the “ESP32 Board Firmware Recovery Method” to reflash the firmware.

③ After the connection is established, click or “Run - Download and Run” to download the program into the controller and execute it.

Program Outcome

After downloading the program, the ESP32 will print the coordinates of the recognized color and the detection results in the terminal. Additionally, the message “crc ok” indicates that the checksum is correct.

Program Analysis

Source Code

Let’s focus on the ESP32 program analysis on the receiver end. For an analysis of the K210 program on the transmitted end, please refer to the 6. AI Vision Games -> 6.2 Multiple Colors Recognition .

(1) Import Module

Import the time module, the WonderK210 communication class, and the message format classes (Find_Box_Msg_st, Find_Box_st, and Find_Msg_st) for subsequent usage.

# 载入时间模块(load time module)
import time
# 载入K210通信类(load K210 communication class)（WonderK210），消息格式类(message format class)（Find_Box_Msg_st , Find_Box_st , Find_Msg_st）, 功能号(function number)（k210_PACKET_FUNCTION）
from WonderK210 import WonderK210 , Find_Box_Msg_st , Find_Box_st , Find_Msg_st , k210_PACKET_FUNCTION

(2) Establish Communication

Instantiate the K210 communication class.

# 实例化K210通信类(instantiate K210 communication class)
wk = WonderK210()

Enable the serial port.

# 开启串口(start serial port)
wk.serial_begin()

(3) Main Function

Enter a while loop to continuously update and receive messages. The function ret = wk.find_box_msg (k210_PACKET_FUNCTION.K210_COLOR_SORTING) searches for target in the data and retrieve the received messages.

#loop
while True:
  # 更新接收消息(update receiving message)
  wk.update_data()
  # 获取接收消息(obtain receiving message)
  ret = wk.find_box_msg(k210_PACKET_FUNCTION.K210_COLOR_SORTING)

If a recognition result is found (ret is not None), the recognition result will be printed. The x and y represent the top-left coordinates of the box in the image, the w denotes the width of the box, the h represents the height of the box, and the msg is the recognition result.

  # 若有新消息获取到(if a new message is obtained)
  if ret is not None:
    # 打印消息内容(print message content)
    print(ret.x)
    print(ret.y)
    print(ret.w)
    print(ret.h)
    print(ret.msg)
  # 延时10ms(delay for 10ms)
  time.sleep_ms(10)

(4) Information Obtaining Function

# 有方框数据和消息数据(there is data for the box and the message)
class Find_Box_Msg_st:
  def __init__(self):
    self.function = 0
    self.x = 0
    self.y = 0
    self.w = 0
    self.h = 0
    self.msg = [0x00] * 200

The class Find_Box_st is defined, which has the following attributes:

function: Represents a specific function or state.

x: Represents the x-coordinate of the box.

y: Represents the y-coordinate of the box.

w: Represents the width of the box.

h: Represents the height of the box.

msg: A list with a length of 200, used to store message data.

7.3.4 STM32

For the instructions on downloading the program to K210 and analyzing the K210-related code, please refer to the “K210” located in the same directory as this tutorial. This lesson only analyzes the data transmitted by K210 and received by STM32.

Game Introduction

Send the recognized color data to STM32 via the K210 vision module.

Getting Ready

(1) Module Connection

① Connect the K210 vision module to a computer with a Type-C data cable.

② Connect the GND, RX, and TX pins of the K210’s UART interface to the corresponding pins on the STM32 with a DuPont wire.

(2) Download and Run Program

① Locate and open “RosRobotControllerM4.uvprojx” program file in the same folder as this lesson.

② Click to perform compiling. After the compiling is completed, click to download the program to the STM32 board.

Program Outcome

Once the program runs, the STM32 will obtain the coordinates of the learned color from the K210 vision module. After running the program, open the located in Appendix->Serial Port Utility to view the printed information.

Program Analysis

Source Code

(1) Initial Settings

#include "k210_packet.h"
#include "global.h"

The “k210_packet.h” consists of data structure definitions, function declarations, and macros related to the K210. It is used to handle data packets, communication protocols, or device control related to the K210. The file “global.h” contains global constants, variables, and more.

typedef struct {
	uint16_t  x;
	uint16_t  y;
	uint16_t  w;
	uint16_t  h;
}BoxData;

Struct BoxData: a structure BoxData is defined. It contains four uint16_t type member variables: x, y, w, and h. These variables represent the position and size of a rectangular box.

//只有方框数据(only includes box data)
typedef struct {
	BoxData box;
}Find_Box_st;

Struct Find_Box_st: a structure Find_Box_st is defined. It contains a variable box of BoxData type. This structure represents a data type that includes only the information of a rectangular box, excluding other message data.

//有方框数据和消息数据(include box data and message data)
typedef struct {
	BoxData box;
	uint8_t msg[100];
}Find_Box_Msg_st;

Struct Find_Box_Msg_st: a structure Find_Box_Msg_st is defined. It contains a variable of type BoxData named box and an array of uint8_t type with a length of 100 named msg. This structure represents a data type that includes both the information of a rectangular box and a message data with a length of 100.

//只有消息数据(only includes message data)
typedef struct {
	uint8_t msg[100];
}Find_Msg_st;

Struct Find_Msg_st: a structure Find_Msg_st is defined. It contains a single array of uint8_t type with a length of 100 named msg. This structure represents a data type that includes only the message data with a length of 100 and does not include the information of a rectangular box.

(2) Message Data Processing

/**
* @brief 只有方框数据 的回调处理(callback processing for only including box data)
* @param frame 数据帧(data frame)
* @retval void
*/
static void k210_Box_cb(struct k210_PacketRawFrame *frame)
{
    Find_Box_st *rec = (Find_Box_st*)frame->data_and_checksum;
	printf("x:%d,y:%d,w:%d,h:%d\n",rec->box.x,rec->box.y,rec->box.w,rec->box.h);
}

k210_Box_cb: handles data frames that only contain box data. It extracts data of type Find_Box_st from the data frame. It prints the position and size information of the box based on the extracted data.

/**
* @brief 有方框数据和消息数据 的回调处理(callback processing for including box data and message data)
* @param frame 数据帧(data frame)
* @retval void
*/
static void k210_Box_Msg_cb(struct k210_PacketRawFrame *frame)
{
    Find_Box_Msg_st *rec = (Find_Box_Msg_st*)frame->data_and_checksum;
	printf("x:%d,y:%d,w:%d,h:%d\n",rec->box.x,rec->box.y,rec->box.w,rec->box.h);
	rec->msg[frame->data_length - 8] = '\0'; //将check位清除(clear check bit)
	printf("msg:%s\n",rec->msg);
}

It extracts data of type Find_Box_Msg_st from the data frame. It prints the position and size information of the box based on the extracted data. It clears the checksum of the message data and then prints the message content.

/**
* @brief 只有消息数据 的回调处理(callback processing for only including message data)
* @param frame 数据帧(data frame)
* @retval void
*/
static void k210_Msg_cb(struct k210_PacketRawFrame *frame)
{
    Find_Msg_st *rec = (Find_Msg_st*)frame->data_and_checksum;
	rec->msg[frame->data_length] = '\0'; //将check位清除(clear check bit)
	printf("msg:%s\n",rec->msg);
}

k210_Msg_cb: handles data frames that only contain message data. It extracts data of type Find_Msg_st from the data frame. It clears the checksum of the message data and then prints the message content. These functions serve as callback functions. When receiving specific types of data frames, these functions are called to process the data within the frames.

(3) Callback Function

//注册回调函数(register callback function)
void k210_packet_handle_init(void)
{
    k210_packet_register_callback(&k210_packet_controller, K210_COLOR_RECOGNITION, k210_Box_cb); //颜色识别(color recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_BARCODES, k210_Box_Msg_cb); //条形码识别(barcode recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_QRCODES, k210_Box_Msg_cb);	 //二维码识别(QR code recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_APRILTAGS, k210_Box_Msg_cb);//机器码识别(AprilTag recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_FACE_YOLO, k210_Box_cb); 	 //人脸检测(face detection)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_FACEFEATURE, k210_Box_cb);  //人脸特征识别(face feature recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_FACEMASK, k210_Box_Msg_cb); //口罩识别(mask recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_OBJECT, k210_Box_Msg_cb);	 //物体识别(object recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_SELF_LEARNING, k210_Msg_cb);//自主学习分类(autonomous learning and classification)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_DIGITAL, k210_Msg_cb);      //手写数字识别(handwritten number recognition)
	k210_packet_register_callback(&k210_packet_controller, K210_FIND_FACE_RECOGNITION, k210_Box_Msg_cb); //物体识别(object recognition)
    
    k210_packet_register_callback(&k210_packet_controller, K210_FIND_RED_FOLLOW, k210_Box_cb); //红色追踪(red tracking)
    k210_packet_register_callback(&k210_packet_controller, K210_FIND_SIGNPOST_FOLLOW, k210_Box_Msg_cb); //路标追踪(road sign tracking)
    k210_packet_register_callback(&k210_packet_controller, K210_FIND_DIGITAL_CARD, k210_Box_Msg_cb); //数字卡片识别(number card recognition)
    k210_packet_register_callback(&k210_packet_controller, K210_GARBAGE_SORTING, k210_Box_Msg_cb); //垃圾分类(waste classification)
    k210_packet_register_callback(&k210_packet_controller, K210_COLOR_SORTING, k210_Box_Msg_cb); //多种颜色识别(multiple colors recognition)
}