Debugging the kernel using Ftrace

[xeno74]

For example debugging Mac-on-Linux/KVM with Ftrace.

Ftrace shows the events that lead up to a crash. It gives a better chance of finding exactly what caused it and can help the developer in creating the correct solution.

1. First you need the kernel 3.14.4 with ftrace support. Download: vmlinux-3.14.4-KVM-PR-FTRACE-AmigaOneX1000.tar.bz2
2. After the start of the new kernel 3.14.4, the directory /sys/kernel/debug will be created. To mount
   this directory, you can add to your /etc/fstab file:
   

   Code: Select all

   debugfs       /sys/kernel/debug          debugfs defaults        0       0
   

   Or you can mount it at run time with:

   Code: Select all

   mount -t debugfs nodev /sys/kernel/debug
   

3. Change to the directory /sys/kernel/debug/tracing

   Code: Select all

   cd /sys/kernel/debug/tracing/
   

4. Activate the function graph tracer:

   Code: Select all

   echo function_graph > current_tracer
   

5. Start Mac OS X with startmol -X
6. Show the kernel KVM events:

   Code: Select all

   cat trace | head -20
   

For example ftrace of Mac-on-Linux/KVM:

ftrace_mac-on-linux.txt.gz

(4.63 KiB) Downloaded 378 times

[xeno74]

Problem Mac OS X with MOL on KVM: Sometimes I get a shell in the single user mode. But the cpu load rise up to 100% and I can't enter any commands.

A detailed ftrace from Mac-on-Linux/KVM:

1. Code: Select all

   modprobe kvm-pr

2. Code: Select all

   mount -t debugfs nodev /sys/kernel/debug

3. Code: Select all

   echo 1 > /sys/kernel/debug/tracing/events/kvm/enable

4. Code: Select all

   echo 1 > /sys/kernel/debug/tracing/events/kvm_pr/enable

5. Code: Select all

   cat /sys/kernel/debug/tracing/trace_pipe > /dev/shm/trace &

6. Code: Select all

   startmol -X

   (Mac OS X Tiger guest in the single user mode from the DVD ISO)

7. Code: Select all

   cat /dev/shm/trace

8. Code: Select all

   startmol -X

   (Mac OS X Jaguar guest in the single user mode from the disk image)

9. Code: Select all

   cat /dev/shm/trace

Here are the traces:

trace_mol_mac_os_x_tiger.txt.tar.gz
trace_mol_mac_os_x_jaguar.txt.tar.gz

Screenshots:

Note: This is a Linux thread. Mac-on-Linux is only an example for tracing the Linux kernel. Please do not ask a question about Mac OS X.

[xeno74]

I have compiled the kernel 3.14.4 with KVM DEBUG_MMU.

Code: Select all

diff --git a/arch/powerpc/kvm/book3s_32_mmu.c b/arch/powerpc/kvm/book3s_32_mmu.c
index 93503bb..939b14f 100644
--- a/arch/powerpc/kvm/book3s_32_mmu.c
+++ b/arch/powerpc/kvm/book3s_32_mmu.c
@@ -27,8 +27,8 @@
#include <asm/kvm_ppc.h>
#include <asm/kvm_book3s.h>

-/* #define DEBUG_MMU */
-/* #define DEBUG_MMU_PTE */
+#define DEBUG_MMU
+#define DEBUG_MMU_PTE
/* #define DEBUG_MMU_PTE_IP 0xfff14c40 */

#ifdef DEBUG_MMU 

Now I get KVM messages over dmesg:

Code: Select all

watch -n 0 "dmesg" > dmesg_mol_mac_os_x_tiger.txt

Code: Select all

watch -n 0 "dmesg" > dmesg_mol_mac_os_x_jaguar.txt

dmesg_mol_mac_os_x_tiger.txt.tar.gz

(73.33 KiB) Downloaded 389 times

dmesg_mol_mac_os_x_jaguar.txt.tar.gz

(101.48 KiB) Downloaded 345 times

Note: This is a Linux thread. Mac-on-Linux is only an example for debugging the Linux kernel. Please do not ask a question about Mac OS X.