QEMU issue with curl and SSL certificates on Debian

[dubo-dubon-duponey]

FROM    debian:buster-slim
RUN    apt-get update && apt-get install -y --no-install-recommends curl ca-certificates
RUN    curl https://www.google.com

docker buildx build --platform="linux/arm64,linux/amd64,linux/386" -f Dockerfile . <- works as expected

docker buildx build --platform="linux/arm/v7" -f Dockerfile . <- fails with curl: (60) SSL certificate problem: unable to get local issuer certificate

Same problem with linux/arm/v6.

This problem is not visible on a native arm machine - so, I'm guessing it's a QEMU issue.

[justincormack]

I believe this is this issue https://bugs.launchpad.net/qemu/+bug/1805913 which is kind of complicated to fix as it is a qemu/kernel/glibc combined issue that qemu on its own cannot fix...

[dubo-dubon-duponey]

"Same things happens with update-ca-certificates. It calls c_rehash through openssl, which ends up doing nothing. As a result, curl with https and probably anything else that uses SSL fails to work."

🤦🏽‍♂

@justincormack do you know if musl would present the same issue?

[justincormack]

It seems to be mainly a problem on ext4 filesystems, so if it is an option, changing the filesystem that your system uses might be an option.

[justincormack]

It should not apply not Musl, and it is fine with older glibc. (pre-Buster and Ubuntu 18.04 and earlier are fine for example).

[dubo-dubon-duponey]

Thanks.

[justincormack]

It seems that if we compile qemu-static as a 32 bit binary this fixes the issue as it uses 32 bit syscalls. I opened linuxkit/linuxkit#3438

[Silex]

Related issues:

https://sourceware.org/bugzilla/show_bug.cgi?id=23960
https://bugzilla.kernel.org/show_bug.cgi?id=205957
https://bugs.launchpad.net/qemu/+bug/1861161

[Silex]

@justincormack: interesting and more logical workaround. For the record another workaround is to remove dir_index from ext4 filesystems but for me it does not work.

So I just have to cross-compile qemu-static as a 32 bit binary? That does not sound too hard, I'll try.

[Silex]

So I did apt install qemu-user-static:i386 but it still bugs 😢

The process still gets stuck in a loop involving getdents64.

root@earth:~# file /usr/bin/qemu-arm-static
/usr/bin/qemu-arm-static: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=ff1224d87ca5dece8d0b0f5735cfee7fae97ee58, stripped

root@earth:~# ps afx | grep qemu
ports/org.qemu.guest_agent.0
10584 ?        Ssl    0:00      |   |   \_ /usr/bin/qemu-arm-static /usr/bin/make install
28768 ?        Sl     0:01      |   |       \_ /usr/bin/qemu-arm-static /usr/bin/make -C src VCSWITNESS=$(srcdir)/../.git/logs/HEAD all
16718 ?        Sl     0:00      |   |           \_ /usr/bin/qemu-arm-static /usr/bin/make -C ../lisp compile-first EMACS=../src/bootstrap-emacs
16726 ?        Rl    48:24      |   |               \_ /usr/bin/qemu-arm-static ../src/bootstrap-emacs -batch --no-site-file --no-site-lisp --eval (setq load-prefer-newer t) -f batch-byte-compile emacs-lisp/macroexp.el
10696 ?        Ssl    0:00      |       \_ /usr/bin/qemu-aarch64-static /usr/bin/make install
10972 ?        Sl     0:02      |           \_ /usr/bin/qemu-aarch64-static /usr/bin/make -C src VCSWITNESS=$(srcdir)/../.git/logs/HEAD all
20397 ?        Sl     0:00      |               \_ /usr/bin/qemu-aarch64-static /usr/bin/make -C ../lisp compile-first EMACS=../src/bootstrap-emacs
20405 ?        Rl    24:09      |                   \_ /usr/bin/qemu-aarch64-static ../src/bootstrap-emacs -batch --no-site-file --no-site-lisp --eval (setq load-prefer-newer t) -f batch-byte-compile emacs-lisp/macroexp.el

root@earth:~# strace -p 16726
clock_gettime(CLOCK_REALTIME, {tv_sec=1584794027, tv_nsec=921230669}) = 0
getdents64(5, /* 0 entries */, 2048)    = 0
_llseek(5, 0, [0], SEEK_SET)            = 0
getdents64(5, /* 5 entries */, 2048)    = 144
tgkill(29984, 29987, SIGRT_2)           = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_REALTIME, {tv_sec=1584794027, tv_nsec=921642405}) = 0
getdents64(5, /* 0 entries */, 2048)    = 0
_llseek(5, 0, [0], SEEK_SET)            = 0
getdents64(5, /* 5 entries */, 2048)    = 144
tgkill(29984, 29987, SIGRT_2)           = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_REALTIME, {tv_sec=1584794027, tv_nsec=922333065}) = 0
getdents64(5, /* 0 entries */, 2048)    = 0
_llseek(5, 0, [0], SEEK_SET)            = 0
getdents64(5, /* 5 entries */, 2048)    = 144
tgkill(29984, 29987, SIGRT_2)           = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_REALTIME, ^C{tv_sec=1584794027, tv_nsec=923201432}) = 0
strace: Process 16726 detached

Looks like I need to dig deeper and actually debug QEMU 😟

[tonistiigi]

Looks like this has been fixed in debian bullseye. Based on https://bugzilla.kernel.org/show_bug.cgi?id=205957 the kernel patches for it are not accepted and newer and properly configured glibc is needed to work around it.