Linux kernel v6.4.16 crash on virtualization.framework

[ilikejam]

Description

Seeing kernel crash errors on an M1 Mac with one of our images. Was working prevously, but not sure is the OS or docker has been upgraded since.

Reproduce

$ mkdir test && touch test/go.py
# cap SYS_ADMIN is required as the container creates read-write overlay mounts at runtime
$ docker run --rm -ti --cap-add=SYS_ADMIN -v "$(PWD)/test":/repo:ro public.ecr.aws/bbc-ats/bbc-ats-lint
Running linters
ERRO[0000] error waiting for container:

Expected behavior

Container should run OK, not crash the Linux VM

docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.24.0 (122432)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /Users/starkd06/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.22.0-desktop.2
    Path:     /Users/starkd06/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/starkd06/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/starkd06/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.8
    Path:     /Users/starkd06/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/starkd06/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/starkd06/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.7
    Path:     /Users/starkd06/.docker/cli-plugins/docker-scout

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 2
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 11
 Total Memory: 7.667GiB
 Name: docker-desktop
 ID: 99583941-1097-41a4-a9f8-0a59eb661c6d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

F123AF07-8555-41C8-8B6A-13C59C0AA3FB/20231009144422

Additional Info

Full error is:

Fatal error reported: Linux kernel v6.4.16 crash on virtualization.framework

[ 214.238908] Unable to handle kernel NULL pointer dereference at virtual address 000000000000046c
[ 214.239039] Mem abort info:
[ 214.239058] ESR = 0x0000000096000006
[ 214.239097] EC = 0x25: DABT (current EL), IL = 32 bits
[ 214.239154] SET = 0, FnV = 0
[ 214.239209] EA = 0, S1PTW = 0
[ 214.239262] FSC = 0x06: level 2 translation fault
[ 214.239305] Data abort info:
[ 214.239345] ISV = 0, ISS = 0x00000006
[ 214.239389] CM = 0, WnR = 0
[ 214.239425] user pgtable: 4k pages, 48-bit VAs, pgdp=000000010a6ec000
[ 214.239551] [000000000000046c] pgd=0800000140ff4003, p4d=0800000140ff4003, pud=0800000154c05003, pmd=0000000000000000
[ 214.239638] Internal error: Oops: 0000000096000006 [#1] PREEMPT SMP
[ 214.239738] Modules linked in: xfrm_user xfrm_algo nfsd auth_rpcgss nfs lockd grace sunrpc fakeowner(O) shiftfs(O) grpcfuse(O) vmw_vsock_virtio_transport vmw_vsock_virtio_transport_common vsock
[ 214.239949] CPU: 6 PID: 2272 Comm: black Tainted: G O 6.4.16-linuxkit #1
[ 214.240036] pstate: 41401005 (nZcv daif +PAN -UAO -TCO +DIT +SSBS BTYPE=--)
[ 214.240154] pc : errseq_sample+0x14/0x30
[ 214.240215] lr : do_dentry_open+0x60/0x38c
[ 214.240247] sp : ffff80000be5b8a0
[ 214.240309] x29: ffff80000be5b8a0 x28: 0000000000000000 x27: 0000000000000000
[ 214.240399] x26: 0000000000000004 x25: ffffbc1e1a36f3f8 x24: 0000000000000004
[ 214.240476] x23: 0000000000000000 x22: ffff31421968e510 x21: ffff314221803b00
[ 214.240561] x20: ffff314221803b00 x19: 000000000000046c x18: 0000000000000000
[ 214.240634] x17: 0000000000000000 x16: ffffbc1e1908d658 x15: 0000000000000000
[ 214.240698] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 214.240829] x11: 0000000000000000 x10: 0000000000000000 x9 : ffffbc1e1909cf08
[ 214.240939] x8 : ffff31421968e600 x7 : 0000000000000000 x6 : 0000000000000064
[ 214.241035] x5 : 0000000000000000 x4 : 00000000ffffffff x3 : ffff314219648b00
[ 214.241126] x2 : 0000000100000000 x1 : 0000000000000000 x0 : 000000000000046c
[ 214.241188] Call trace:
[ 214.241222] errseq_sample+0x14/0x30
[ 214.241261] do_dentry_open+0x60/0x38c
[ 214.241317] open_with_fake_path+0x58/0x84
[ 214.241363] fakeowner_open_realfile+0x60/0x8c [fakeowner]
[ 214.241421] fakeowner_open+0x38/0x74 [fakeowner]
[ 214.241482] do_dentry_open+0x2f8/0x38c
[ 214.241507] open_with_fake_path+0x58/0x84
[ 214.241535] ovl_open_realfile+0xbc/0xe8
[ 214.241567] ovl_open+0x74/0xbc
[ 214.241613] do_dentry_open+0x2f8/0x38c
[ 214.241685] vfs_open+0x3c/0x4c
[ 214.241717] path_openat+0x9d8/0xb30
[ 214.241758] do_filp_open+0x74/0xe4
[ 214.241806] do_sys_openat2+0xa8/0x10c
[ 214.241875] do_sys_open+0x5c/0x8c
[ 214.241911] __arm64_sys_openat+0x30/0x40
[ 214.241956] invoke_syscall.constprop.0+0x88/0xd8
[ 214.242002] do_el0_svc+0xc4/0x128
[ 214.242037] el0_svc+0x9c/0xcc
[ 214.242084] el0t_64_sync_handler+0xac/0x13c
[ 214.242150] el0t_64_sync+0x190/0x194
[ 214.242191] Code: a9be7bfd 910003fd f9000bf3 aa0003f3 (b9400260)
[ 214.242249] ---[ end trace 0000000000000000 ]---

[ilikejam]

Switching to osxfs file sharing eliminates the problem.

[fredericdalleau]

Hi, thanks for reporting, we are investigating.

[jechter]

FWIW, I am experiencing the exact same issue (came here by googling for the stack trace): Docker for mac crashes in virtualization framework when doing overlay mounts in a linux container. This is on a M2 MacBook Pro running macOS 14.0.

[ilikejam]

For current version (4.27.0) google hits, the error is now:

ERRO[0004] error waiting for container: unexpected EOF

Trace looks like:

Fatal error reported: Linux kernel v6.6.12 crash on virtualization.framework

[ 3.739983] Internal error: Oops: 0000000096000006 [#1] SMP
[ 3.740033] Modules linked in: selfowner(O) shiftfs(O) rosetta(O) grpcfuse(O) fakeowner(O)
[ 3.740161] CPU: 2 PID: 1056 Comm: black Tainted: G O 6.6.12-linuxkit #1
[ 3.740242] pstate: 81400005 (Nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 3.740310] pc : backing_file_open+0x6c/0xc0
[ 3.740360] lr : backing_file_open+0x54/0xc0
[ 3.740433] sp : ffff0000cb5279c0
[ 3.740478] x29: ffff0000cb5279c0 x28: ffff0000cb527c60 x27: 0000000000000000
[ 3.740546] x26: 0000000000000000 x25: ffff0000c0678000 x24: ffff800081ffc648
[ 3.740623] x23: ffff80007a2691d8 x22: ffff0000f8a7f098 x21: ffff0000f8a7f098
[ 3.740696] x20: ffff0000cb5279f8 x19: ffff0000f8a7e700 x18: 0000000000000000
[ 3.740778] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[ 3.740865] x14: 0000000000000000 x13: ffff0000c17e2020 x12: ffff0000cb527ca4
[ 3.740958] x11: 000000050556c1c0 x10: 0000000000000002 x9 : ffff800080301f8c
[ 3.741027] x8 : ffff0000f8a7e800 x7 : 0000000000000000 x6 : 000000000000003f
[ 3.741082] x5 : 0000000000000040 x4 : ffff0000cb5278f0 x3 : 0000000000000000
[ 3.741160] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000f8a7e700
[ 3.741235] Call trace:
[ 3.741268] backing_file_open+0x6c/0xc0
[ 3.741315] fakeowner_open+0x6c/0x190 [fakeowner]
[ 3.741367] do_dentry_open+0x17c/0x528
[ 3.741406] backing_file_open+0x74/0xc0
[ 3.741436] ovl_open_realfile+0xdc/0xf0
[ 3.741481] ovl_open+0xc8/0x110
[ 3.741523] do_dentry_open+0x17c/0x528
[ 3.741567] vfs_open+0x34/0x40
[ 3.741604] path_openat+0xb04/0xdd8
[ 3.741641] do_filp_open+0xa4/0x160
[ 3.741681] do_sys_openat2+0xcc/0x108
[ 3.741716] __arm64_sys_openat+0x6c/0xc0
[ 3.741748] invoke_syscall.constprop.0+0x58/0xf8
[ 3.741811] do_el0_svc+0x48/0xd8
[ 3.741852] el0_svc+0x44/0x130
[ 3.741898] el0t_64_sync_handler+0x120/0x130
[ 3.741940] el0t_64_sync+0x190/0x198
[ 3.741993] Code: a9400e82 a9000c22 f9400681 d2800002 (f9401821)
[ 3.742072] ---[ end trace 0000000000000000 ]---

[nadamai]

Having the same issue on my MacBook Pro 13" 2019 (Intel Core i5) with MacOS Sonoma 14.2.1, here is my report:

Fatal error reported: Linux kernel v6.5.11 crash on virtualization.framework

[13371.149560] invalid opcode: 0(2002) - 2024-02-11T20:10:41.158474497Z: kernel BUG at fs/super.c:501!000 [#1] PREEMPT
SMP NOPTI
[13371.152602] C(2003) - 2024-02-11T20:10:41.163124497Z: invalid opcode: 0000 [#1] PREEMPT SMP NOPTIPU: 6 PID: 302 Comm: lifecycle-serve Tainted: G        W  O       6.5.11-linuxkit #1

[13371.152972] R(2004) - 2024-02-11T20:10:41.166166497Z: CPU: 6 PID: 302 Comm: lifecycle-serve Tainted: G        W  O       6.5.11-linuxkit #1IP: 0010:generic_shutdown_super+0x142/0x150
[13371.153092] C
ode: cc cc e8 d1 5a f7 ff 48 8b bb e0 00 00 00 eb db 48 8b 43 28(2005) - 2024-02-11T20:10:41.166536497Z: RIP: 0010:generic_shutdown_super+0x142/0x150 48 8d b3 98 03 
00 00 48 c7 c7 38 37 39 ab 48 8b 10 e8 8e 18 de ff <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90
[13371.153437] RSP: 0018:ffffaba480737c00 EFLAGS: 00010246
[13371.153550] RAX: 0000000000000037 RBX: ffff903d574e9800 RCX: 0000000000000000
[13371.153629] RDX: 0000000000000000 RSI: ffff903e7fd9c400 RDI: ffff903e7fd9c400
[13371.153781] RBP: ffffffffc03bb6e0 R08: 0000000000000000 R09: ffffaba480737ab8
[13371.154295] R10: 0000000000000003 R11: ffffffffabac9c68 R12: ffff903d4d87526c
[13371.159833] R13: ffff903d40d22101 R14: ffff903d4006d5b8 R15: ffff903d4d875268
[13371.160253] FS:  0000000000000000(0000) GS:ffff903e7fd80000(0000) knlGS:0000000000000000
[13371.160642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[13371.160840] CR2: 00007f83c8b27b30 CR3: 0000000100c5a006 CR4: 00000000003706a0
[13371.160982] Call Trace:
[13371.161174]  <TASK>
[13371.161256]  ? die+0x36/0x90
[13371.161403]  ? do_trap+0xda/0x100
[13371.161580]  ? generic_shutdown_super+0x142/0x150
[13371.161708]  ? do_error_trap+0x65/0x80
[13371.161751]  ? generic_shutdown_super+0x142/0x150
[13371.161803]  ? exc_invalid_op+0x50/0x70
[13371.161847]  ? generic_shutdown_super+0x142/0x150
[13371.161905]  ? asm_exc_invalid_op+0x1a/0x20
[13371.162110]  ? generic_shutdown_super+0x142/0x150
[13371.162404]  kill_anon_super+0x12/0x30
[13371.162462]  deactivate_locked_super+0x30/0xa0
[13371.162516]  cleanup_mnt+0xbd/0x150
[13371.162559]  task_work_run+0x5a/0x90
[13371.162597]  do_exit+0x35a/0xad0
[13371.162632]  ? futex_unqueue+0x3c/0x60
[13371.162729]  do_group_exit+0x31/0x80
[13371.162860]  get_signal+0x98c/0x9c0
[13371.162913]  arch_do_signal_or_restart+0x3e/0x270
[13371.162993]  exit_to_user_mode_prepare+0xb1/0x110
[13371.163032]  syscall_exit_to_user_mode+0x21/0x50
[13371.163082]  do_syscall_64+0x6b/0x90
[13371.163158]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[13371.163206] RIP: 0033:0x472c83
[13371.163246] Code: Unable to access opcode bytes at 0x472c59.
[13371.163312] RSP: 002b:00007fd63ff05948 EFLAGS: 00000286 ORIG_RAX: 00000000000000ca
[13371.163368] RAX: fffffffffffffe00 RBX: 0000000000000000 RCX: 0000000000472c83
[13371.163416] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000c000073948
[13371.163485] RBP: 00007fd63ff05990 R08: 0000000000000000 R09: 0000000000000000
[13371.163930] R10: 0000000000000000 R11: 0000000000000286 R12: 00000000004477c0
[13371.163951] R13: 0000000000000010 R14: 000000c000007d40 R15: 000000c000f44580
[13371.164081]  </TASK>
[13371.164142] Modules linked in: xfrm_user xfrm_algo nfsd auth_rpcgss nfs lockd grace sunrpc fakeowner(O) shiftfs(O) grpcfuse(O) vmw_vsock_virtio_transport vmw_vsock_virtio_transport_common vsock [last unloaded: selfowner(O)]
[13371.165500] ---[ end trace 0000000000000000 ]---

[bbc-davidstark]

This appears to be fixed in v4.30.0 (although looking at the changelogs, it seems it was fixed in 4.28?)