Kernel Debugging

Debugging a VM kernel

avhsupport avatar
Written by avhsupport
Updated over a week ago

Using GDB/LLDB

AVH implements a GDB server stub to support VM debugging. This allows any debugger that understands this protocol to be used as well, including lldb.

After creating a VM, the Connect tab provides an lldb one-liner that can be used to establish a debugging session with the VM.

While lldb is specified in the UI, users are free to use gdb as well.

$ gdb
(gdb) target remote 10.11.1.1:4000
Remote debugging using 10.11.1.1:4000
0xffff80001117094c in ?? ()
(gdb) info reg
x0 0xe0 224
x1 0x11c430 1164336
x2 0x0 0
x3 0x4000000000000000 4611686018427387904
x4 0x0 0
x5 0xffff8000dde56000 -140733765558272
x6 0x80 128
...
(gdb) x/10i $pc
=> 0xffff80001117094c: hint #0x1d
0xffff800011170950: ret
0xffff800011170954: hint #0x19
0xffff800011170958: stp x29, x30, [sp,#-16]!
0xffff80001117095c: mov x29, sp
0xffff800011170960: bl 0xffff800011170940
0xffff800011170964: ldp x29, x30, [sp],#16
0xffff800011170968: hint #0x1d
0xffff80001117096c: ret
0xffff800011170970: hint #0x19

All the usual debugger features work, including single-step, breakpoints, watchpoints, memory inspection, register inspection, register or memory editing, etc. Consult a guide specific to your debugger.

Monitor Commands

The so-called monitor commands are raw packets interpreted directly by the debug server and not by the client (or marshaled into any known command format). In other words, the monitor is convenient way to expand a debugger's functionality without changing the client code. gdb exposes this functionality by prefixing your command with monitor. Sending e.g. monitor help requests that AVH respond with its list of available commands.

(gdb) monitor help
Available monitor commands:
help List all commands.
info Show VM information.
sr Access CPU system registers.
pt Dump current pagetable.
v2p Translate a virtual address to physical.
p2v Translate a physical address to virtual.
vcont-dflt Apply default vCont action if specific actions are present.
phys Switch to physical access mode (optionally takes base address).
virt Switch to virtual access mode.
bt Show iOS-style backtrace.
thread Show current thread.

To list the contents of the system registers one can use monitor sr.
โ€‹

(gdb) monitor sr
System registers of CPU 0:
sctlr_el1 = 0x0000000030d4d91d
cpacr_el1 = 0x0000000000300000
zcr_el1 = 0x0000000000000000
ttbr0_el1 = 0x00000001061c5000
ttbr1_el1 = 0x0188000041859000
tcr_el1 = 0x00000032b5503510
...

Or to list the current configuration of the page tables of the current virtual CPU (vCPU).

(gdb) monitor pt
Pagetable for CPU 0:
0000004000000000-0000004000000fff -> 010a022000 local user pXN uXN memory Owb-rwa Iwb-rwa
0000004000001000-0000004000001fff -> 010a0d9000 local user pXN uXN memory Owb-rwa Iwb-rwa
0000004000002000-0000004000002fff -> 010a023000 local user pXN uXN memory Owb-rwa Iwb-rwa
0000004000008000-0000004000008fff -> 0108871000 local user pXN uXN memory Owb-rwa Iwb-rwa
0000004000009000-0000004000009fff -> 0108741000 local user pXN uXN memory Owb-rwa Iwb-rwa
000000400000a000-000000400000afff -> 010a1db000 local user pXN uXN memory Owb-rwa Iwb-rwa
...

lldb can be used similarly to send monitor commands using a slightly more verbose command structure.

$ lldb --one-line "gdb-remote 10.11.1.1:4000"
(lldb) gdb-remote 10.11.1.1:4000
Process 1 stopped
* thread #1, stop reason = signal SIGINT
frame #0: 0xffffffe01117094c
Target 0: (No executable module.) stopped.
(lldb) process plugin packet monitor help
Available monitor commands:
help List all commands.
info Show VM information.
sr Access CPU system registers.
...

Note that debugger threads are discrete VM CPUs, and so switching between threads allows users to interact with specific cores.

Did this answer your question?