cvw/docs/docker

Consistant Build of Toolchain for Wally

Dockerfile.* contains a multi-stage build for all the toolchains that are required for Wally's open-source features.

Hazards:

• If there is any change in ${CVW_HOME}/linux/buildroot-config-src folder with main.config, you have to copy it to the current folder to buildroot-config-src
• If there is any change in ${CVW_HOME}/linux/testvector-generation folder with main.config, you have to copy it to the current folder to testvector-generation

If you have any other questions, please read the troubleshooting first.

TODOs

• Pinning the tools version
  • As for the consistent tool build, should we use specific versions of tools in order to avoid bugs at the master branch?
  • And we will upgrade the versions of tool after a certain period of time to get the latest features and patches while verifying it won’t cause any problem.
• Mount the EDA ToolsQuestaSIM and Path
  • USE_QUESTA and QUESTA for path
• Enable X11 forwarding for docker
  • --network=host for docker run
  • xhost +localhost:${USER} for host
• Regression Script
• Configure the license for Questa
• Change the condition from empty string to 1
• Add linux testvector-generation
  • Estimate the useless building intermediate files

TL;DR

Steps:

1. Install either Docker Engine or Podman for container support
2. Run start-up script docs/docker/start.sh to start a stateless container to run the toolchains and EDA tool

Docker Engine or Podman

First and foremost, install either Docker Engine or Podman:

• Docker Engine (More Popular, default): https://docs.docker.com/engine/install/
• Podman: https://podman.io/docs/installation

Here are some common installation commands (not guarantee to be up to date)

# For Ubuntu
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
# Installation
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# Hello-World Example
docker run hello-world

Use of Start-up Script

Files at this folder can help you to build/fetch environment you need to run wally with the help of Docker.

Here are some common use cases, it will provides you an environment with RISC-V toolchains that required by this project:

# By default, we assume that you have cloned the cvw respository and running the script at relative path `docs/docker`

# For HMC students, /opt/riscv is available and nothing needs to be built
TOOLCHAINS_MOUNT=/opt/riscv QUESTA=/cad/mentor/questa_sim-2023.4 ./start.sh

# For those with all the toolchains installed, simply mount the toolchains
TOOLCHAINS_MOUNT=<path-to-toolchains> ./start.sh

# For those have nothing, fetching the builds are easiest thing
CVW_MOUNT=<path-to-cvw> ./start.sh
# if you want to use Podman instead of Docker Engine
USE_PODMAN=1 ./start.sh

# For other cases, checkout start-up script for building option

For further usage, please consult the following configuration.

Regression on Master Branch

The regression script is used by image wallysoc/regression_wally to run regressions on the master branch of a specific repository.

If you are using docker, then replace the following podman with docker.

# create volume for permanent storage
podman volume create cvw_temp

# run regression on the OpenHW/cvw
podman run \
    -e CLEAN_CVW=1 -e BUILD_RISCOF=1 -e RUN_QUESTA=1 \
    -v cvw_temp:/home/cad/cvw \
    -v /cad/mentor/questa_sim-2023.4:/cad/mentor/questa_sim-xxxx.x_x \
    --privileged --network=host \
    --rm wallysoc/regression_wally

# run regression on the Karl-Han/cvw
podman run \
    -e CLEAN_CVW=1 -e BUILD_RISCOF=1 -e RUN_QUESTA=1 \
    -e CVW_GIT=https://github.com/Karl-Han/cvw \
    -v cvw_temp:/home/cad/cvw \
    -v /cad/mentor/questa_sim-2023.4:/cad/mentor/questa_sim-xxxx.x_x \
    --privileged --network=host \
    --rm wallysoc/regression_wally

# get into the container command line to debug or reading files
podman run -it \
    -e RUN_QUESTA=1 \
    -v cvw_temp:/home/cad/cvw \
    -v /cad/mentor/questa_sim-2023.4:/cad/mentor/questa_sim-xxxx.x_x \
    --privileged --network=host \
    wallysoc/regression_wally /bin/bash

Conventions

• In the container
  • default user is cad
  • RISCV is defined as /opt/riscv
  • QUESTA is defined as /cad/mentor/questa_sim-xxxx.x_x
    • bin location is in $QUESTA/questasim/bin
  • cvw folder should be mounted on /home/${USERNAME}/cvw
    • as for cad, it is /home/cad/cvw
• In the current shell environment: checkout the constants in the following script section

New Dockerfile Design

There are two parts of the new docker-related design:

• build proper image(s)
• scripts for different purposes

Problem Statement

The following 3 problems are to be resolved:

• remove storage of useless files with Dockerfile
  • git build: clean up
  • packages info
    • apt: apt-get clean
    • pip: --no-cache-dir
• reuse storage
  • read-only $RISCV volume across different users with docker image
  • local-built RISCV with start-up script
• use commercial EDA tools: optional environment variable configuration with start-up script

Dockerfiles

There are two Dockerfiles:

• Dockerfile.ubuntu: basic ubuntu system setup and produce ubuntu_wally image in docker
  • corresponds to wallysoc/ubuntu_wally
• Dockerfile.builds: all the required toolchains are built
  • corresponds to wallysoc/toolchains_wally

Because we are going to use the whole environment of ubuntu to get both executables and python packages, we are not going to use multi-stage builds, which runs only executables.

Scripts

There are four scripts:

• start.sh (most often used): start running the container
  • if you don't care about toolchains and running regression automatically, this script is only thing you need to know
• get_images.sh: get docker image wallysoc/ubuntu_wally or wallysoc/toolchains_wally
• get_buildroot_testvector.py: copy buildroot and testvector configuration required by building
• run_regression.sh: run regressions with Verilator (and QuestaSIM) on specific CVW
  • this script is not intended to be run directly, but inside the container
  • However, it is a good resource to look into to know what is happening
• test.sh: a test script to facilitate the above three

All the following options in the corresponding scripts should be set by either:

• define it right before the command: USE_PODMAN=1 ./start.sh with USE_PODMAN on start-up script
  • the variable is only effective for the current command, not the following environment
• declare it globally and use it any time afterwards:

# declare it globally in the environment
export DOCKER_EXEC=$(which docker)
export UBUNTU_BUILD=1

# run the script with all the above variables
./get_images.sh

Start-up Script: start.sh

There are two settings:

• build/fetch both ubuntu_wally and toolchains in docker image and use it
• build/fetch only ubuntu_wally and use local toolchains folder

Options:

• USE_PODMAN:
  • by default, docker is used
  • set USE_PODMAN=1 otherwise
• UBUNTU_BUILD:
  • fetch by default
  • set UBUNTU_BUILD=1 if you want to build with Dockerfile.ubuntu
• TOOLCHAINS_BUILD:
  • fetch by default
  • set TOOLCHAINS_BUILD=1 if you want to build with Dockerfile.build

Image Building Script: get_images.sh

Options (if you want to build the images):

• DOCKER_EXEC:
  • docker by default
  • if you want to use podman, then set it to $(which podman)
• UBUNTU_BUILD:
  • fetch by default
  • set it to 1 if you want to build it instead of fetching it
• TOOLCHAINS_BUILD:
  • fetch by default
  • set it to 1 if you want to build it instead of fetching it

Regression Script: run_regression.sh

There are two parts for regression:

• Verilator: must be able to run as it is open-sourced
• Questa: OPTIONAL as it is commercial EDA Tool

There are three main knobs:

1. CLEAN_CVW: remove the /home/${USERNAME}/cvw inside the container (it can be a volume) and clone the ${CVW_GIT}.
2. BUILD_RISCOF: build RISCOF in the /home/${USERNAME}/cvw, sometimes you don't want to rebuild if there is no change in the test suite.
3. RUN_QUESTA: enable the QuestaSIM in regression

Options:

• CVW_GIT: git clone address
• CLEAN_CVW: clone CVW_GIT if enabled with -e CLEAN_CVW=1
• BUILD_RISCOF: rebuild RISCOF if enabled with -e BUILD_RISCOF=1
• RUN_QUESTA: run vsim to check if enabled with -e RUN_QUESTA=1
  • QUESTA: home folder for mounted QuestaSIM /cad/mentor/questa_sim-xxxx.x_x if enabled
  • for example, if your vsim is in /cad/mentor/questa_sim-2023.4/questasim/bin/vsim then your local QuestaSIM folder is /cad/mentor/questa_sim-2023.4, so you have to add -v /cad/mentor/questa_sim-2023.4:/cad/mentor/questa_sim-xxxx.x_x -e RUN_QUESTA=1

Commercial EDA Tools

This is kind of tricky, because Docker network is a different network from the host. Therefore, it should be carefully handled in order to use the host's license server while minimizing the access of docker container.

There are at least two ways to solve this problem:

• use --network=host in docker-run to use the same network
• (NOT WORKING) use bridge in docker-run while forwarding docker network's traffic to localhost without setting X11UseLocalhost no
  • this idea is from https://stackoverflow.com/a/64284364/10765798

Old Dockerfile Analysis

Refer to 91919150a9/docs/Dockerfile

There are stages in the old Dockerfile:

• debian-based package installtion
  • apt package
  • python3 package
• user and its group configuration
• clone and build toolchain with prefix=$RISCV
  • riscv-gnu-toolchain: https://github.com/riscv-collab/riscv-gnu-toolchain
  • elf2hex: https://github.com/sifive/elf2hex
  • qemu: https://github.com/qemu/qemu
  • spike: https://github.com/riscv-software-src/riscv-isa-sim
  • sail: https://github.com/riscv/sail-riscv/commits/master/
  • buildroot: https://github.com/buildroot/buildroot
  • verilator: https://github.com/verilator/verilator

Tool Versions till 20240331

• riscv-gnu-toolchain: 2024.03.01
• elf2hex: f28a3103c06131ed3895052b1341daf4ca0b1c9c
• qemu: not tested
• spike: 3427b459f88d2334368a1abbdf5a3000957f08e8
• sail: f601c866153c79a7ae8404f939dc2d66aa2e41f9
• buildroot: 2021.05
• verilator: v5.022

Troubleshooting

Permission Denied for .git

Description: permission problem in /home/$USERNAME/cvw.

$ podman run -v cvw_temp:/home/cad/cvw -e CLEAN_CVW=1 -e BUILD_RISCOF=1 -e RUN_QUESTA=1 -v /cad/mentor/questa_sim-2023.4:/cad/mentor/questa_sim-xxxx.x_x --rm wallysoc/regression_wally                          
No CVW_GIT is provided                                                                                 
rm: cannot remove '/home/cad/cvw': Device or resource busy                                             
Cloning into '/home/cad/cvw'...                                                                        
/home/cad/cvw/.git: Permission denied                                                                  
chmod: cannot access '/home/cad/cvw/setup.sh': No such file or directory                               
chmod: cannot access '/home/cad/cvw/site-setup.sh': No such file or directory                          
make: *** No rule to make target 'install'.  Stop.                                                     
make: *** No rule to make target 'verify'.  Stop.                                                      
make: *** No rule to make target 'coverage'.  Stop.                                                    
make: *** No rule to make target 'benchmarks'.  Stop.                                                  
/home/cad/run_regression.sh: line 64: cd: /home/cad/cvw/sim: No such file or directory                 
/home/cad/run_regression.sh: line 65: /home/cad/cvw/sim/regression_verilator.out: No such file or direc
tory       

It may be caused by podman and I am not sure why it happens.

Solution: get into the container with interaction as root and change the permission of the folder/volume /home/$USERNAME/cvw

podman run -it --user root -v cvw_temp:/home/cad/cvw -e RUN_QUESTA= -v /cad/mentor/questa_sim-2023.4:/cad/mentor/questa_sim-xxxx.x_x --net=host --rm --privileged wallysoc/regression_wally /bin/bash

chown -R $USERNAME:$USERNAME /home/$USERNAME

References

• Dockerfile Docs: https://docs.docker.com/reference/dockerfile/
• Multi-stage builds: https://docs.docker.com/build/building/multi-stage/
• Best Practices: https://docs.docker.com/develop/develop-images/guidelines/
• Chinese Reference: https://yeasy.gitbook.io/docker_practice/
• Clean Cache
  • apt cache: https://gist.github.com/marvell/7c812736565928e602c4
  • pip cache: https://stackoverflow.com/questions/50333650/install-python-package-in-docker-file
• Docker Network: https://docs.docker.com/network/