Docker Images For Arm Cluster

Docker Images For Arm Cluster

Now there's Kubernetes cluster on Raspberry Pis we needed to provide some docker images of our services. While working on Raspberry Pi as a Desktop Computer it is easy do build Docker image for Arm (arm64/aarch64 as my Raspberry Pi has Raspberry Pi OS 64bit version).

Such an image is then easily built, tagged and pushed to the cluster's repository by (as per Microk8s documentation):

docker build . -t <master-node-ip-or-dn>:32000/myimage:1.0.1
docker tag <image-sha> <master-node-ip-or-dn>:32000/myimage:1.0.1
docker push <master-node-ip-or-dn>:32000/myimage

But, what if the Raspberry Pi is not the development platform but an x86 system like MacBook Pro or a linux desktop?

Creating Docker Images on x86 Platforms (OSX)

Luckily Docker has come up with Multi Arch Support in a form of docker buildx command.

But, as with everything, there are gotchas...

First, making build image is not as straight-forward as above documentation said. Main issue is that Microk8s's local repository is, by default, insecure. So, pushing image from docker buildx need to allow for insecure repositories. And that's where all the problems are.

And it's all but easy to set up. If nothing else but because of some discrepancies in documentation and apparently a bug in the command itself. So, here are the steps I've done in order to work around it:

Setting Insecure Repository Access For Docker Buildx

This issue here is pointing to use of --config parameter (while creating build image!) and documentation here is pointing to example config file. But, the latest version of Docker for Mac (2.3.4.0 (46980) Edge - at the time of writing this blog post) expects a directory passed in --config:

$ docker buildx create --config ~/.docker/buildx/config.toml  --name mybuilder
WARNING: unable to read config file: open ~/.docker/buildx/config.toml/config.json: not a directory

So, to make create image happy, I've just dropped to directory where config image will reside:

docker buildx create --config ~/.docker/buildx/ --name mybuilder

And then amended ~/.docker/buildx/instances/mybuilder. With above create command instance definition file looks like this:

{"Name":"x","Driver":"docker-container","Nodes":[{"Name":"x0","Endpoint":"unix:///var/run/docker.sock","Platforms":null,"Flags":null,"ConfigFile":"~/.docker/buildx/","DriverOpts":{}}]}

See "ConfigFile":"~/.docker/buildx/". If I am to try to use that builder it is going to complain that config file ~/.docker/buildx/ does not exist. Well, it doesn't. So, I've changed it to:

"ConfigFile":"~/.docker/buildx/config.toml"

and added ~/.docker/buildx/config.toml as:

debug = true

[registry."<master-node-ip-or-dn>:32000"]
  http = true
  insecure = true

Note: in above I've replaced /Users/username with '~' in all instances.

Creating and Pushing Images

After the above - docker buildx command to build image was happy and was able to push to insecure Microk8s repo on my cluster with:

docker buildx build --platform linux/arm64,linux/arm/v7 -t <master-node-ip-or-dn>:32000/test:1.0.2 --push .

Such image can be inspected with:

docker manifest inspect --insecure -v <master-node-ip-or-dn>:32000/test:1.0.2

and the result will come up with something like following:

[
    {
        "Ref": "<master-node-ip-or-dn>:32000/test:1.0.2@sha256:c5535b8e086343a221dc4c48d136693303a7d378c4ace4847824471032c6fefd",
        "Descriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:c5535b8e086343a221dc4c48d136693303a7d378c4ace4847824471032c6fefd",
            "size": 6582,
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        },
        "SchemaV2Manifest": {
            ...
            "layers": [...]
        } 
    },
    {
        "Ref": "cluster-master.thenet:32000/test:1.0.2@sha256:01c3b06353e762a7af1234b8f9c83a63fd71f4a60166563f6b1eb9d0494994bc",
        "Descriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:01c3b06353e762a7af1234b8f9c83a63fd71f4a60166563f6b1eb9d0494994bc",
            "size": 6582,
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v7"
            }
        },
        "SchemaV2Manifest": {
            ...     
            "layers": [...]
        }
   }
]

To use such an image you can use version name, the 'latest' or just directly point to it with full sha. For instance:

<master-node-ip-or-dn>:32000/test:1.0.2@sha256:c5535b8e086343a221dc4c48d136693303a7d378c4ace4847824471032c6fefd

to select arm64 (aarch64) architecture.

Running Arm Image on x86 Platform

Fortunately it doesn't stop here. Docker had provided option to run such Arm architecture images easy locally, too:

docker run -p8082:8082 <master-node-ip-or-dn>:32000/test:1.0.2@sha256:c5535b8e086343a221dc4c48d136693303a7d378c4ace4847824471032c6fefd

Conclusion

This nicely rounds up whole Raspberry Pi cluster and development for it.

Make a code (Python, Java or whatever), pick generic arm (or arm64/aarch64) image for your Dockerfile, create Docker image and push it to the cluster's local (insecure!) repository quickly. Of course, not before you tested it locally (if appropriate).