Tools | Cyrille Froehlich

Gitea ARM crossbuild (with GoModules)

Sat, 20 Mar 2021 10:45:24 +0000

Upgrading to Gitea 1.13.4 didn’t go as planned and it failed to compile the go-sqlite3 module with my previous setup. While crossbuilding, the CGO parameter is automatically disabled (i.e. compiling is done with CGO_ENABLED=0) but the go-sqlite3 module won’t build anymore if this option is not set (probably require some optimisation in C and a pure Go implementation won’t fit anymore). As a consequence of enabling this option, setting the environment variable CC to the C cross-compiler explicitly is now mandatory to ensure modules don’t target the wrong platform.

A recent Go compiler will also have the Go Modules support enabled, thus we can now simplify the build process with a more usual checkout/make pattern.

Dockerfile sample:

FROM golang:latest

RUN dpkg --add-architecture armhf

RUN apt-get update
RUN apt-get install -y --no-install-recommends curl git build-essential crossbuild-essential-armhf

SHELL [ "/bin/bash", "--login", "-c" ]

ARG NVM_VERSION="0.37.2"
RUN\
  ( export PROFILE=$HOME/.profile;\
    /usr/bin/curl -o- "https://raw.githubusercontent.com/creationix/nvm/v${NVM_VERSION}/install.sh" | /bin/bash )

ARG NODE_VERSION="14.15.3"
RUN\
  nvm install "${NODE_VERSION}" &&\
  nvm use "v${NODE_VERSION}" &&\
  nvm alias default "v${NODE_VERSION}"

ARG YARN_VERSION="1.22.10"
RUN\
  ( export PROFILE=$HOME/.profile;\
    /usr/bin/curl -o- -L "https:/yarnpkg.com/install.sh" | /bin/bash -s -- --version ${YARN_VERSION})

RUN\
  ( echo ;\
    echo 'source $HOME/.nvm/nvm.sh';\
    echo 'export PATH=/go/bin:/usr/local/go/bin:${PATH}'; ) >> $HOME/.profile

WORKDIR /go

ENV GOARM=7
ENV GOARCH=arm
ENV GOOS=linux
ENV CGO_ENABLED=1
ENV CC=arm-linux-gnueabihf-gcc

ARG GITEA_VERSION="v1.13.4"
RUN git clone --shallow-submodules --recurse-submodules --branch "${GITEA_VERSION}" --depth 1 https://github.com/go-gitea/gitea.git gitea

ENV TAGS="bindata sqlite sqlite_unlock_notify"
RUN cd gitea && make build

Which we can run with (replace the host_output_dir with some folder on the host):

docker build -t gitea-arm-builder .
docker run --rm -v [host_output_dir]:/output gitea-arm-builder cp /go/src/code.gitea.io/gitea/gitea /output/gitea-1.13.4-arm-linux

GPU Support With Docker Compose

Fri, 01 Jan 2021 03:52:55 +0000

I use Docker quite a lot as it allows me to install, test, remove libraries or components without any risk to break my main system. VM is another option but even if a linux install is now just a few clicks in virt-manager GUI, it takes time and I don’t need the full isolation provided for this use-case. One thing that is quite difficult to setup in docker is the host GPU sharing. Same issue with the VM, unless having 2 cards installed in the host, giving access to the guest system either have an heavy performance impact, either require unlinking the card from the host to bind it to the guest system (and the host is now headless in the meantime, so hope the switch works without fail or you’ll be blind-fixing the issue in front of a dark screen).

It is usually not a problem as most of the containers are running services, and my emacs can easily be displayed on the host Xserver even when running inside a container. A Jupyter notebook with Tensorflow and GPU support or an instance of Leela to review a Go game is another story…

nvidia-docker, nvidia-docker2 or nvidia-container provided somewhat working solutions but replacing the default docker runtime broke some of my other containers, some were ok with nvidia-docker while others only with nvidia-container, and docker-compose was not usable with them at all. Of course, switching between the different approach required to uninstall/install the correct set of packages with matching versions. I had to rely to wrapper scripts to run the correct docker commands, which is basically the job of docker-compose… Somewhat working, but quite a messy solution.

GPU support was discussed for a long time and it seems we’re near fixing all of this docker-compose 1.28.0-rc1 is announcing “Support for Nvidia GPUs via device requests”.

I’ll update this with a more detailled setup when 1.28 is released, as things may change. The first part is the setup described here, moved to a container. The rest is a few lines to add to the compose yaml

My docker daemon config only move docker files out of /var and define a local registry:

> cat /etc/docker/daemon.json
{
  "data-root": "/home/docker",
  "registry-mirrors": [ "https://192.168.1.178:5000" ]
}

On the host the buster-backports nvidia-driver and nvidia-container-runtime are installed. In the Dockerfile, install nvidia-cuda-dev from buster-backports (thus we match the host driver version).

Tensorflow require cuDNN, so install the matching libcudnn8 too. I have a copy of this in a local debian repository: adding a source.list file to the image then apt-get install is faster than pushing it from the docker context (these libraries are huge).

In the docker-compose yaml, add:

jupyter:
  ...
  deploy:
    resources:
      reservations:
        devices:
        - 'driver': 'nvidia'
          'count': 1
          'capabilities': ['gpu', 'utility']
  devices:
    - /dev/nvidia-uvm:/dev/nvidia-uvm
    - /dev/nvidia-uvm-tools:/dev/nvidia-uvm-tools
  ...

The deploy parameters will act as the –gpus option. The nvidia-uvm module is not always loaded, running modprobe beforehand (or force loading the module at boottime) may be required. Cuda will fail to initialize if these devices are not present. Once the devices are available on the host, nvidia-container doesn’t map them into the container, so we manually add them in the compose yaml.

Save last kernel used as grub's default

Fri, 25 Dec 2020 03:01:15 +0000

The Nvidia driver currently available in buster-backport (450.80) seems to have issues with the 5.9 kernel, resulting in an unavailable uvm module.

[ 8883.526038] nvidia_uvm: module uses symbols from proprietary module nvidia, inheriting taint.
[ 8883.526306] nvidia_uvm: Unknown symbol radix_tree_preloads (err -2)
[ 8883.526339] nvidia_uvm: Unknown symbol set_cpus_allowed_ptr (err -2)
[ 8883.526380] nvidia_uvm: Unknown symbol mmu_notifier_unregister (err -2)
[ 8883.526485] nvidia_uvm: Unknown symbol __mmu_notifier_register (err -2)

The next release (455.+) was said to fix this but has yet to reach the backports repository. In the meantime, sticking to the 5.8 kernel is the easiest workaround. But I keep forgetting to change the selection at boot time: power on the computer, untangle the mouse cord, look at the screen and it’s already on the login. Speedy NVMe are great but better not blink . Thus, I first thought of forcing grub to use the second entry as default and found these lines:

> grep DEFAULT /etc/grub.d/*
...
if [ "x${GRUB_DEFAULT}" = "x" ] ; then GRUB_DEFAULT=0 ; fi
if [ "x${GRUB_DEFAULT}" = "xsaved" ] ; then GRUB_DEFAULT='${saved_entry}' ; fi
...

So instead of hardcoding a value, it turns out we can also load a previously saved entry, just by setting the two following variables in /etc/default/grub and running update-grub.

GRUB_DEFAULT=saved
GRUB_SAVEDEFAULT=true

The first line enable the default selection based on the saved value, the second set the default value on the current selection (thus a manual change on the grub menu is all that’s required to update the saved default). Looking in the /boot/grub/grub.cfg, we can see the additional command savedefault at the beginning of each menuentry section.

Probably not a recent feature, but it is neat to find out your odd use-case is already handled nicely in the configuration files.

Tensorflow with GPU acceleration on Debian Buster

Wed, 18 Dec 2019 04:23:17 +0000

A bit early for christmas presents, but I got myself a new machine (AMD Ryzen 3900X / 32G / RTX2060 Super, by the way Ryzen CPUs are real monsters). Now, let’s try to make use of this.

Nvidia drivers and tools

Assuming the Debian backports are configured, we can choose to use a more up to date driver version.

> cat /etc/apt/sources.list.d/backports.list
deb http://deb.debian.org/debian buster-backports main contrib non-free

Install the nvidia-driver (currently 450.80) and the nvidia-cuda-toolkit (currently 11.1)

cuDNN packages

Tensorflow require the cuDNN components but they are not available in the usual repository so we have to download the packages from Nvidia. cuDNN is not available for the Debian distribution but Ubuntu ones are compatible enough. The packages we need are :

libcudnn8_8.0.5.39-1+cuda11.1_amd64.deb
libcudnn8-dev_8.0.5.39-1+cuda11.1_amd64.deb

We’ll install them manually with a dpkg -i as configuring the Nvidia repository will probably fetch conflicting packages too.

Jupyter notebook

Most of the documentation on the internet related to machine learning is using the python API,

> python3 -mvenv jupyter-env
> cd jupyter-env
> . ./bin/activate
> pip install -U pip setuptools wheel
> pip install jupyterlab tensorflow-gpu

Then check if the GPU is available to Tensorflow :

> python3
>>> import tensorflow as tf
2020-12-26 11:19:12.763644: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
>>> tf.config.list_physical_devices()
2020-12-26 11:19:50.377149: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-12-26 11:19:50.382895: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2020-12-26 11:19:50.471910: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-26 11:19:50.472295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: GeForce RTX 2060 SUPER computeCapability: 7.5
coreClock: 1.695GHz coreCount: 34 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-12-26 11:19:50.472310: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2020-12-26 11:19:50.485168: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2020-12-26 11:19:50.485199: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2020-12-26 11:19:50.491058: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2020-12-26 11:19:50.494631: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2020-12-26 11:19:50.494706: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory
2020-12-26 11:19:50.498531: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2020-12-26 11:19:50.498616: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2020-12-26 11:19:50.498625: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]
>>>

And, we failed ! cuDNN was loaded, GPU is detected but Tensorflow wants an older version of libcusolver: nvidia-cuda-toolkit installed the version 11 but we need the version 10 (no idea why as other libraries use cuda11, which is the correct version for tensorflow 2.4).

~~Not a big issue as a libcusolver10 package is available in the backports. Let’s install that then give it another try:~~

Update: Now, libcusolver11 is no longer available in the buster-backports and tensorflow-gpu keep requesting the v10, it will probably be fixed in next tensorflow or nvidia drivers release but in the meantime, it seems the workaround is to use the cuda11 library through a symlink:

cd /usr/lib/x86_64-linux-gnu
ln -sf libcusolver.so.11 libcusolver.so.10
ldconfig


...
2020-12-26 11:21:28.182053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
>>>

Now launch a notebook with jupyter lab, and run some tutorial code

...
2020-12-26 11:27:51.811618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6736 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060 SUPER, pci bus id: 0000:0a:00.0, compute capability: 7.5)
...

Yep, seems good now.

Gitea ARM crossbuild

Sun, 03 Nov 2019 12:45:24 +0000

Instead of building the Gitea binary on a Raspberry PI, we can speed up things a bit by crossbuilding from some other computer with more punch. Recent version of Gitea will need nodejs/npm to be available at build time, so we’ll install this too.

Dockerfile sample:

FROM golang:latest

RUN dpkg --add-architecture armhf

RUN apt-get update
RUN apt-get install -y --no-install-recommends curl git build-essential crossbuild-essential-armhf

SHELL [ "/bin/bash", "--login", "-c" ]

ARG NVM_VERSION="0.37.2"
RUN\
  ( export PROFILE=$HOME/.profile;\
    /usr/bin/curl -o- "https://raw.githubusercontent.com/creationix/nvm/v${NVM_VERSION}/install.sh" | /bin/bash )

ARG NODE_VERSION="14.15.3"
RUN\
  nvm install "${NODE_VERSION}" &&\
  nvm use "v${NODE_VERSION}" &&\
  nvm alias default "v${NODE_VERSION}"

ARG YARN_VERSION="1.22.10"
RUN\
  ( export PROFILE=$HOME/.profile;\
    /usr/bin/curl -o- -L "https:/yarnpkg.com/install.sh" | /bin/bash -s -- --version ${YARN_VERSION})

RUN\
  ( echo ;\
    echo 'source $HOME/.nvm/nvm.sh';\
    echo 'export PATH=/go/bin:/usr/local/go/bin:${PATH}'; ) >> $HOME/.profile

WORKDIR /go

ENV GOARM=7
ENV GOARCH=arm
ENV GOOS=linux

ARG GITEA_VERSION="v1.12.6"

RUN go get -d -u code.gitea.io/gitea

RUN cd /go/src/code.gitea.io/gitea && git checkout ${GITEA_VERSION}

RUN cd /go/src/code.gitea.io/gitea && TAGS="bindata sqlite sqlite_unlock_notify" make build

Which we can run with (replace the host_output_dir with some folder on the host):

docker build -t gitea-arm-builder .
docker run --rm -v [host_output_dir]:/output gitea-arm-builder cp /go/src/code.gitea.io/gitea/gitea /output/gitea-1.12.5-arm-linux

My real setup is a little bit more involved as it uses Docker-Compose and multi-staged Dockerfiles to share the nodejs install and optimize the build.

Gitea ARM7 build (Raspberry PI 2)

Sun, 03 Nov 2019 12:45:24 +0000

I’m hosting my projects code on a Gitea instance running on a Raspberry PI. Thing is, the last Raspbian update broke the gitea binary: probably some change in the kernel or libc. The official gitea build for ARM kept crashing with a segmentation fault before even entering the main. Of course, latest gitea builds were also broken. Only building Gitea from source eventually fixed the issue.

I had to bump the swapfile size a bit as the build process burst to 1.2Go of memory (RPI2 have 1Go, and default swap was set to 100Mo… Murphy strikes again) :

$ dphys-swapfile swapoff
# Edit the size in /etc/dphys-swapfile
# Then recreate the swap
$ dphys-swapfile setup
$ dphys-swapfile swapon

Grab an up-to-date golang compiler:

$ docker pull golang
$ docker run -it golang bash

Now, inside the container, retrieve the source and build:

% export GOOS=linux GOARCH=arm GOARM=7
% go get -d -u code.gitea.io/gitea
% cd $GOPATH/src/code.gitea.io/gitea
% git checkout v1.12.5
% TAGS="bindata sqlite sqlite_unlock_notify" make build

Retrieve the file:

$ docker cp [your container name]:/go/src/code.gitea.io/gitea/gitea gitea-1.12.5-linux-arm-7
# My service file references the gitea symlimk, just point to the new file
$ ln -sf gitea-1.12.5-linux-arm-7 gitea
$ systemctl restart gitea