Vitis AI Tutorial – Part 4

Published On: November, 17, 2023 By: Eshtaartha Basu

This tutorial will walk you through the steps required to integrate Xilinx’s DPU-TRD Acceleration PL Kernel to your Acceleration Ready Vitis Platform. The tutorial will also show you how to build and run AI demos like Face Detection and ADAS Detection directly from the Xilinx AI Model Zoo on the OSDZU3-REF platform.

This tutorial can be used standalone or as Part 4 of a 4-part Acceleration tutorial series that will help you run Vitis-AI DPU-TRD based Face Detection demo, ADAS Detection demo (and other AI demos) on the OSDZU3-REF board. An overview of the flow is shown below:

Pre-requisites

This tutorial assumes the user has already gone through:

1. Tutorial 1 and has generated OSDZU3-REF_Accl_Platform.xsa file.
2. Tutorial 2 and has generated an Acceleration Ready Petalinux Software Components
3. Tutorial 3 and has generated an Acceleration Ready Vitis Platform named “osdzu3-ref-accl-vitis-platform.xpfm” under a Vitis workspace named “osdzu3-ref-accl-vitis-pkg”

Objectives

1. Integrate DPU-TRD PL Acceleration Kernel to our Acceleration Ready Vitis Platform
2. Compile and build ADAS, Face Detect demos
3. Run the demos on the OSDZU3-REF platform

Tutorial Requirements

This tutorial will require the following software setup.

• Docker – https://docs.docker.com/engine/install/ubuntu/
  • The steps in this tutorial have been tested with Docker Version 23.0.1. A more detailed version information is shown below.
    
• Download the “densebox_640_360.prototxt” file provided by Octavo Systems

Note: All the steps in this tutorial have been validated using Petalinux 2021.2 running on Ubuntu 20.04.LTS. However, the overall design flow should be similar for other Petalinux and host OS versions as well.

Integrate DPU-TRD PL Acceleration Kernel to our Acceleration Ready Vitis Platform

This section will describe the steps required to Integrate DPU-TRD PL Acceleration Kernel to our Acceleration Ready Vitis Platform.

Add Vitis AI 2.0 to Vitis IDE’s Library Repositories

1. Open Vitis 2021.2 IDE
2. Set the directory created as part of Tutorial 3, osdzu3-ref-accl-vitis-pkg, as your workspace directory. The workspace should already have osdzu3-ref-accl-vitis-platform created as part of Tutorial 3 under Explorer.
3. Under Window > Preferences > Library Repositories tab > Click Add button
4. Configure the settings of the Vitis AI 2.0 library as shown below. Ensure the Branch is 2.0 and not “master”.
5. Press Apply and Close
6. Go to Xilinx > Libraries… > Find the Vitis AI 2.0 library we just added > Press Download. The download may take a few minutes.
7. Once the download and installation finishes, the download button will say Installed as shown below. Click OK to close the window.

Download and install Vitis-AI specific sysroot

1. Navigate to <Vitis-AI 2.0 installation directory>/setup/mpsoc/VART
   Default location is: ~/.Xilinx/Vitis/2021.2/vitis-ai-2-0
2. Run the host cross compiler setup script using the command
   ./host_cross_compiler_setup.sh
   Default installation location is ~/petalinux_sdk_2021.2.
   If you need sysroot installed some other place, open the script and add the full path to below line:
   install_path=~/petalinux_sdk_2021.2
   

Create DPU-TRD application project

1. Go to File > New > Application Project
2. Click Next to skip the Welcome Page
3. Select platform osdzu3-ref-accl-vitis-platform, click Next
4. Name the project DPU-TRD, click Next
5. Under domain linux on psu_cortexa53, set the paths for sysroot (generated in previous section), EXT4 Rootfs file and Kernel Image (generated as part of Tutorial 2) appropriately as shown below. Click Next.
6. Under dsa > select DPU Kernel (choose the one that corresponds to Vitis AI 2.0 if you have multiple Vitis AI versions installed, similar to my case)
7. Press Finish to generate the application project.

Configure DPU-TRD project

This section describes how to configure DPU-TRD project for OSDZU3-REF platform.

Configure Build Target

1. Open system project file DPU-TRD_system.sprj
2. Change Active build configuration to “Hardware”

Configure DPU Settings

1. Under dpu_trd_kernels/src/prj/Vitis, open dpu_conf.vh
2. Update the following settings:
   1. Change `define B4096 to `define B2304
   2. Change `define CHANNEL_AUGMENTATION_ENABLE to `define CHANNEL_AUGMENTATION_DISABLE
   3. Change `define DWCV_ENABLE to `define DWCV_DISABLE
   4. Change `define POOL_AVG_ENABLE to `define POOL_AVG_DISABLE
   5. Change `define DSP48_USAGE_HIGH to `define DSP48_USAGE_LOW
   6. Save the file.

Configure system_hw_link for proper kernel instantiation

1. Double click on DPU-TRD_system_hw_link.prj
2. Remove softmax IP under Hardware Functions:
   1. Right click on sfm_xrt_top > Remove
   2. Change number of DPUCZDX8G compute units to 1

Set v++ configuration

1. Under Assitant View > DPU-TRD_system > DPU-TRD_system_hw_link > Hardware > Double click on dpu
2. In the line of V++ configuration settings, click on the three dots (…)
3. Update the configuration as follows:

   [clock]
   id=0:DPUCZDX8G_1.aclk
   id=1:DPUCZDX8G_1.ap_clk_2
   
   [connectivity]
   sp=DPUCZDX8G_1.M_AXI_GP0:HPC0
   sp=DPUCZDX8G_1.M_AXI_HP0:HP0
   sp=DPUCZDX8G_1.M_AXI_HP2:HP2

4. Click OK
5. Click Apply and Close

Build the DPU-TRD project and deploy it on OSDZU3-REF board

1. Select the DPU-TRD_system project > Right click > Build Project
2. Once the build finishes, you should be able to find the updated OSDZU3-REF Vivado project with integrated DPU and its dependencies under: DPU-TRD_system_hw_link/Hardware/dpu.build/link/vivado/vpl/prj/prj.xpr
3. Open the project using Vivado. The updated block design with DPU should look similar to the below figure.
4. Next, you should be able to see flashable “sd_card.img” under DPU-TRD_system/Hardware/package
   
5. Flash the sd_card.img file onto a µSD card using a tool like Balena Etcher
6. Insert the µSD card into OSDZU3-REF board
7. Set the boot mode to SD1 (i.e., 0101)
8. Connect a µUSB cable between the UART port of OSDZU3-REF board and your host computer
9. Open UART console on your host computer
10. Power on the OSDZU3-REF board. You should start seeing petalinux boot log on the UART console
11. Once you reach the OSDZU3 Disk Expander Script prompt, type “y” on the keyboard and press Enter to expand the ext4 Rootfs partition to use all of the remaining µSD card space.
    
12. Next, in the petalinux login prompt, use “root” as username and password.
    
13. Once you successfully login, check Rootfs size again using “df . -h” command to ensure disk expansion was successful. The output may be slightly different from what is shown below depending on your uSD card size.
14. Ensure VART configuration file correctly points to dpu.xclbin in the BOOT partition of the µSD card:
    nano /etc/vart.conf
    Make sure the contents are:
    firmware: /media/sd-mmcblk1p1/dpu.xclbin
15. Validate Vitis-AI runtime and DPU using the command:
    xdputil query
    The output should look similar to what is shown below except for the DPU fingerprint. Your DPU fingerprint will certainly be different since fingerprints are unique.

Compiling models from Vitis AI Model Zoo

The Vitis AI Model Zoo is a free repository that includes optimized, pre-trained, deep learning models to speed up the deployment of deep learning inference on Xilinx™ platforms. These models cover different applications, including but not limited to ADAS/AD, medical, video surveillance, robotics, data center, etc.

As examples, this section will show you how to manually compile the following two Café framework models from Vitis AI Model Zoo:

• Yolov3_cityscapes model for Advanced driver assistance systems (ADAS) detection from USB camera video feed
• Densebox_640_360 model for Live Face Detection from USB camera video feed

1. Let’s begin by cloning the “v2.0” branch of the Vitis-AI repository:

   git clone -b v2.0 https://github.com/Xilinx/Vitis-AI
   cd Vitis-AI
   export VITIS_AI_HOME=$PWD

2. The Vitis AI Model Zoo provides several versions of the Face Detect (Densebox_640_360) and ADAS (yolov3_cityscapes) models for different boards as listed in the model.yaml files of the respective models. The versions of the models we’re interested in are highlighted in the pictures below.

   cd $VITIS_AI_HOME/models/AI-Model-Zoo/model-list
   cat cf_densebox_wider_360_640_1.11G_2.0/model.yaml
   cat dk_yolov3_cityscapes_256_512_0.9_5.46G_2.0/model.yaml
   

   

   

3. Download the source archives for both models and extract them.
   1. Face Detect (Densebox_640_360)
      1. Download the model using the command:
         wget https://www.xilinx.com/bin/public/openDownload?filename=cf_densebox_wider_360_640_1.11G_2.0.zip -O cf_densebox_wider_360_640_1.11G_2.0.zip
      2. Unzip it using the command:
   2. ADAS (yolov3_cityscapes)
      1. Download the model using the command:
         wget https://www.xilinx.com/bin/public/openDownload?filename=dk_yolov3_cityscapes_256_512_0.9_5.46G_2.0.zip -O dk_yolov3_cityscapes_256_512_0.9_5.46G_2.0.zip
      2. Unzip it using the command:
         unzip dk_yolov3_cityscapes_256_512_0.9_5.46G_2.0.zip
4. Copy the arch.json architecture file for our DPU based hardware platform to our current directory. The arch.json contains encrypted content, that uniquely identifies the DPU configuration for our design.The arch.json file can be found in the Vitis workspace used to build DPU-TRD project under: <vitis_workspace>/ DPU-TRD_system_hw_link/Hardware/dpu.build/link/vivado/vpl/prj/prj.gen/sources_1/bd/design_1/ip/design_1_DPUCZDX8G_1_0
   cp {path_to_arch_json}/arch.json .
5. Pull and launch Vitis AI docker container using the following commands:

   sudo docker pull xilinx/vitis-ai:2.0.0.1103
   cd $VITIS_AI_HOME
   sudo sh -x docker_run.sh xilinx/vitis-ai:2.0.0.1103

6. Read all the license notification messages, when prompted press ENTER to go through the terms and press “y” on the keyboard to accept the terms. Next, you should see Vitis-AI Docker command line.
   
7. Navigate to AI Model Zoo directory and create a new directory for the compiled models

   cd models/AI-Model-Zoo
   mkdir compiled_output

8. Let’s create a generic script for compiling Caffe model named “compile_cafe_model.sh” with the following content:

   model_name=$1
   modelzoo_name=$2
   vai_c_caffe \
   --prototxt ./${modelzoo_name}/quantized/deploy.prototxt \
   --caffemodel ./${modelzoo_name}/quantized/deploy.caffemodel \
   --arch ./arch.json \
   --output_dir ./compiled_output/${model_name} \
   --net_name ${model_name}

9. Activate conda environment

   conda activate vitis-ai-caffe

10. Compile the Densebox_640_360 model to work with our newly created DPU based Hardware Platform using the command:
    source ./compile_cafe_model.sh densebox_640_360 cf_densebox_wider_360_640_1.11G_2.0
    
11. Similarly, compile the ADAS model to work with our newly created DPU based Hardware Platform using the command:
    source ./compile_cafe_model.sh yolov3_cityscapes dk_yolov3_cityscapes_256_512_0.9_5.46G_2.0/
    
12. Under “compiled_output” directory, you should be able to see “densebox_640_360.xmodel” and “yolov3_cityscapes.xmodel”
13. Exit docker with the following command:
    Exit
14. Copy “densebox_640_360.xmodel” and “yolov3_cityscapes.xmodel” files to the BOOT partition of the µSD card.

Building ADAS and Face Detect demo applications

1. Navigate to ADAS application directory under Vitis AI 2.0 repo from previous section
   cd $VITIS_AI_HOME/demo/VART/adas_detection
2. source <path_to_your_Vitis_AI_SDK>/environment-setup-cortexa72-cortexa53-xilinx-linux. In my case it is:
   source ~/petalinux_sdk_2021.2/environment-setup-cortexa72-cortexa53-xilinx-linux
3. Build the ADAS application:

   chmod +x build.sh
   ./build.sh

4. The newly built ADAS application is highlighted below
   
5. Similarly, build the face detect application by navigating to samples directory:

   cd $VITIS_AI_HOME/demo/Vitis-AI-Library/samples/facedetect
   chmod +x build.sh
   ./build.sh

6. The newly built face detect applications (there are more than one version) are shown below.
   
7. Copy “test_video_facedetect” and “adas_detection” applications to the BOOT partition of the µSD card.
8. Copy “densebox_640_360.prototxt” (downloaded in section 4) to the BOOT partition of the µSD card.

Running demos on the OSDZU3-REF board

1. Connect an external monitor to the OSDZU3-REF board directly via Display Port or via Display Port to HDMI monitor
2. Connect a webcam to OSDZU3-REF board directly or via a USB-C hub
3. Turn on the OSDZU3-REF board. Once you get to the petalinux login prompt, use “root” for username and password
4. Ensure a desktop appears on the external monitor
5. Check if the USB webcam (highlighted below) is detected by running the following commands:
   1. lsusb
      
   2. v4l2-ctl --list-devices
      
6. Set the DISPLAY environment variable:
   export DISPLAY=:0.0
7. Lower the resolution of DP monitor to 640 x 480
   xrandr --output DP-1 --mode 640x480
8. Navigate to home directory:
   cd ~
9. Copy the following files from BOOT partition of the µSD card to the home directory of petalinux:

   cp –r /mnt/sd-mmcblk1p1/test_video_facedetect ~
   cp –r /mnt/sd-mmcblk1p1/densebox_640_360.xmodel ~
   cp –r /mnt/sd-mmcblk1p1/densebox_640_360.prototxt ~
   cp –r /mnt/sd-mmcblk1p1/adas_detection ~
   cp –r /mnt/sd-mmcblk1p1/yolov3_cityscapes.xmodel ~

10. Create “models” directory under /usr/share/vitis-ai-library
    mkdir /usr/share/vitis_ai_library/models
11. Copy densebox_640_360.xmodel to /usr/share/vitis-ai-library/models
    cp ~/densebox_640_360.xmodel /usr/share/vitis-ai-library/models
12. Run the Face Detect demo application using the command:
    ./test_video_facedetect densebox_640_360.xmodel 0
    You can watch this demo in action here –https://www.youtube.com/watch?v=QxXFs8B_scU
13. Run the ADAS detection demo application using the command:
    ./adas_detection /dev/video0 yolov3_cityscapes.xmodel
Point the webcam towards live traffic on the street. You can also play dashcam traffic videos on your laptop and point the webcam towards it to observe the ADAS traffic detection in action as shown below.

Resource Utilization and Power Consumption Snapshot

The PL resource utilization by DPU-TRD PL Kernel on OSDZU3-REF is shown by the below figure.

The average input power consumption for the demos is as follows.

Further Reading

This has completed the Vitis AI Tutorial series.  Please visit any of the previous versions of this tutorial series for further information.

• Vitis AI Tutorial – Part 1
  The First Part of a 4-part Acceleration tutorial series that will help you run Vitis-AI DPU-TRD based Face Detection demo, ADAS Detection demo (and other AI demos) on the OSDZU3-REF board.
• Vitis AI Tutorial – Part 2
  The Second Part of a 4-part Acceleration tutorial series that will help you run Vitis-AI DPU-TRD based Face Detection demo, ADAS Detection demo (and other AI demos) on the OSDZU3-REF board. It focuses on generating Acceleration Ready Petalinux Software Components with Vitis-AI 2.0 and Xilinx Real Time (XRT) support
• Vitis AI Tutorial – Part 3
  The Third Part of a 4-part Acceleration tutorial series that will help you run Vitis-AI DPU-TRD based Face Detection demo, ADAS Detection demo (and other AI demos) on the OSDZU3-REF board. It focuses on the steps required to generate an Acceleration Ready Vitis Platform.

Revision History

Get notified of updates to this Application Note and all of our Technical Documentation.