Re: KGTP (Linux Kernel debugger and tracer) 20110424 release
From: Hui Zhu
Date: Sat May 07 2011 - 06:41:27 EST
Thanks for your help, Steven and Randy.
I update doc according to your mail in
https://code.google.com/p/kgtp/source/detail?r=306
They will be included in next release.
Best,
Hui
On Mon, Apr 25, 2011 at 22:42, Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:
> On Sun, 2011-04-24 at 22:28 +0800, Hui Zhu wrote:
>
>
> There is several grammar mistakes, I'll correct those here that caused
> me to have trouble understanding what you were saying. There's more
> mistakes that I'm not correcting because I still understand what you
> meant. I'll let Randy correct those, as he can come up with better
> sentences than I can.
>
> Do not take this as an insult, I have an utmost respect for those that
> communicate on LKML in a language that is not their own mother tongue.
> And I also understand that the Asian languages are even more different
> than English that it makes it 10 times more difficult to get it right
> (as suppose to those that speak German).
>> --- /dev/null
>> +++ b/Documentation/trace/gtp.txt
>> @@ -0,0 +1,873 @@
>> + Linux Kernel GDB tracepoint module (KGTP)
>> + =========================================
>> + By Hui Zhu <teawater@xxxxxxxxx>
>> + https://code.google.com/p/kgtp/wiki/HOWTO
>> + 2011-04-24
>> +
>> +Table of contents
>> +-----------------
>> +
>> +What is KGTP
>> +Report issue about KGTP
>> +Get info about GDB tracepoint
>> +Get KGTP through http
>> +Get KGTP through svn
>> +Config KGTP
>> +Compile KGTP
>> +Install KGTP
>> +Uninstall KGTP
>> +Howto use
>> + Exec it
>> + Make GDB connect to gtp
>> + Add module symbols to GDB
>> + Get GDB tracepoint commands
>> + Get registers info from Kernel
>> + Get the value of variable from Kernel
>> + How to use use tracepoint condition
>> + How to use trace state variables
>> + Simple trace state variables
>> + Special trace state variables $current_task,
>> + $current_thread_info, $cpu_id, $dump_stack, $printk_level,
>> + $printk_format, $printk_tmp and $clock
>> + Show all the traced data of current frame
>> + Get backtrace info from Kernel
>> + Howto let tracepoint output value directly
>> + Output stack dump directly
>> + Switch collect to output the value directly
>> + Use printf command in actions
>> + Get status of KGTP from Kernel
>> + Set the trace buffer into a circular buffer
>> + Do not stop tracepoint when the GDB disconnect
>> + Howto show the variable that value of it is optimized
>> + Linux kernel "Compile with almost no optimization" patch
>> + Update your GCC
>> + Offline debug
>> +
>> +
>> +
>> +
>> +What is KGTP
>> +------------
>> +
>> +KGTP is a realtime and lightweight Linux Kernel GDB debugger and tracer that
>> +use Kprobe.
>
> "uses Kprobes."
>
>> +
>> +It make Linux Kernel supply a GDB remote debug interface. Then GDB in current
>
> It supplies the Linux Kernel with a GDB remote debug interface. ...
>
>> +machine or remote machine(see "Make GDB connect to gtp") can debug Linux
>> +through GDB tracepoint without stop the Linux Kernel.
>
> without stopping the Linux Kernel.
>
>> +And even if the board doesn't have GDB on it and doesn't have interface for
>> +remote debug. It can debug the Linux Kernel use offline debug (See "Offline
>
> using offline debug.
>
>> +debug").
>> +It support X86-32, X86-64, MIPS and ARM.
>> +http://www.tudou.com/programs/view/_p6VTgxmCRA/ This is a video(in Chinese)
>> +to show how to use it.
>> +
>> +
>> +
>> +
>> +Report issue about KGTP
>
> Report an issues about KGTP
>
>> +-----------------------
>> +You can post it in https://code.google.com/p/kgtp/issues/list or write Email
>
> or write an email to ...
>
>> +to teawater@xxxxxxxxxx
>> +
>> +
>> +
>> +
>> +Get info about GDB tracepoint
>> +-----------------------------
>> +Please goto http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html
>> +
>> +
>> +
>> +
>> +Get KGTP through http
>> +---------------------
>> +Please goto http://code.google.com/p/kgtp/downloads/list OR UPDATE to down
>
> to download the package (?)
>
>> +the package.
>> +
>> +
>> +
>> +
>> +Get KGTP through svn
>> +--------------------
>> +Some people have trouble with access to KGTP website. You can access kgtp
>> +through svn:
>> +
>> +------------------------------------------------------------
>> +svn checkout http://kgtp.googlecode.com/svn/ kgtp-read-only
>> +------------------------------------------------------------
>> +
>> +kgtp-read-only/tags/ There is the each release of KGTP.
>
> This has each release of KGTP (?)
>
>> +kgtp-read-only/trunk/ There is the main trunk of KGTP.
>
> This is the main code repository of KGTP (?)
>
>> +
>> +
>> +
>> +
>> +Config KGTP
>> +-----------
>> +
>> +Before compile KGTP, you can choice which Kernel you want build with and
>
> Before compiling KGTP, you can choose which kernel you want to build
> with and
>
>> +which compiler you want through change the Makefile. For example:
>
> which compiler you want by changing the Makefile in your KGTP
> repository. (?)
>
>
>> +
>> +-------------------------------------------
>> +KERNELDIR := /lib/modules/`uname -r`/build
>> +CROSS_COMPILE :=
>> +-------------------------------------------
>> +
>> +KERNELDIR is set the directory which Kernel you want build for. Now it set to
>
> KERELDIR is set to the directory which holds the kernel you want to
> build for. By default, it is set to the kernel that you are running.
>
>> +the current kernel that you use.
>
>> +CROSS_COMPILE is set the compiler that you want to build the KGTP. Empty mean
>> +use current compiler.
>
> CROSS_COMPILE is set to the compiler you want to build KGTP. Leave empty
> to compile with your default compiler.
>
>> +ARCH is the architecture.
>> +
>> +------------------------------------------
>> +KERNELDIR := /home/teawater/kernel/bamd64
>> +CROSS_COMPILE :=x86_64-glibc_std-
>> +ARCH := x86_64
>> +------------------------------------------
>> +
>> +KERNELDIR is set to /home/teawater/kernel/bamd64. Compiler will
>> +use x86_64-glibc_std-gcc.
>> +
>> +
>> +
>> +
>> +Compile KGTP
>> +------------
>> +
>> +For normal use:
>> +
>> +---------
>> +cd kgtp/
>> +make
>> +---------
>> +
>> +If you want KGTP out the debug message,please use following (It is available
>> +after 20100825):
>
>
> I'm not sure what you mean by the above. Do you mean:
>
> If you want KGTP to output debug messages, then us the following (....)
>
>> +
>> +---------
>> +cd kgtp/
>> +make D=1
>> +---------
>> +
>> +
>> +
>> +
>> +Install KGTP
>> +------------
>> +
>> +------------------
>> +cd kgtp/
>> +sudo make install
>> +------------------
>> +
>> +
>> +
>> +
>> +Uninstall KGTP
>> +--------------
>> +
>> +--------------------
>> +cd kgtp/
>> +sudo make uninstall
>> +--------------------
>> +
>> +
>> +
>> +
>> +Howto use
>> +---------
>> +
>> +Exec it
>> +-------
>> +
>> +If you had installed the KGTP in your system, you can:
>> +
>> +------------------
>> +sudo modprobe gtp
>> +------------------
>> +
>> +Or you can use the kgtp module in the directory.
>> +
>> +-------------------
>> +cd kgtp/
>> +sudo insmod gtp.ko
>> +-------------------
>> +
>> +
>> +
>> +Make GDB connect to gtp
>> +-----------------------
>> +
>> +In current machine:
>
> On current machine.
>
> Now it may be better to use "server" instead of current machine. I guess
> you mean the man box the user is using.
>
>> +
>> +---------------------------------
>> +sudo gdb ./vmlinux
>> +(gdb) target remote /sys/kernel/debug/gtp
>> +Remote debugging using /sys/kernel/debug/gtp
>> +0x0000000000000000 in ?? ()
>> +---------------------------------
>> +
>> +In remote machine:
>
> On remote machine:
>
> "target" may also be better, but I'm fine with "remote".
>
>> +
>> +---------------------------------------------
>> +#Open the KGTP interface in current machine.
>
> Now do you mean current == remote? I'm confused here. It is best to stay
> consistent. Always use "remote" and always use "server".
>
>> +sudo su
>> +nc -l 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp
>> +(nc -l -p 1234 </sys/kernel/debug/gtp >/sys/kernel/debug/gtp for old version
>> +netcat.)
>> +#Let gdb connect to the port 1234
>> +gdb ./vmlinux
>> +(gdb) target remote xxx.xxx.xxx.xxx:1234
>
> You run gdb on the remote machine too?
>
>> +---------------------------------------------
>> +
>> +
>> +
>> +Add module symbols to GDB
>> +-------------------------
>> +
>> +Sometime, you need add the Linux Kernel module(lkm) symbols to GDB to debug it.
>> +Following a example howto add ext3.ko's symbols to GDB:
>
> The following example howto adds ext3.ko symbols to GDB:
>
>> +
>> +--------------------------------------------------------------------------------
>> +cat /proc/modules | grep ext
>> +ext3 116512 3 - Live 0xf9083000
>> +#Get the address 0xf9083000.
>> +modinfo ext3
>> +filename: /lib/modules/2.6.36-rc2+/kernel/fs/ext3/ext3.ko
>> +#Get the directory name of ext3.
>> +(gdb) add-symbol-file /lib/modules/2.6.36-rc2+/kernel/fs/ext3/ext3.ko
>> 0xf9083000
>> +#Then, GDB get the symbols of ext3.
>
> Now GDB has the symbols for ext3 (?)
>
>> +--------------------------------------------------------------------------------
>> +
>> +You will see that it just add the text section info to GDB, sometime you need
> just adds the ..., sometimes you need to add the info section of another
> section. For example:
>
>> +add another section info of other section, for example:
>> +
>> +--------------------------------------------------------------------------------
>> +cat /sys/module/ext3/sections/.bss
>> +0xf908170c
>> +#Get the bss address 0xf908170c.
>> +(gdb) add-symbol-file /lib/modules/2.6.36-rc2+/kernel/fs/ext3/ext3.ko
>> 0xf9083000 -s .bss 0xf908170c
>> +#Then, GDB get the symbols of ext3.
>
> Now GDB has the .bss symbols for ext3 (?)
>
>> +--------------------------------------------------------------------------------
>> +
>> +
>> +
>> +Get GDB tracepoint commands
>> +---------------------------
>> +
>> +Please goto http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html
>> +
>> +
>> +Get registers info from Kernel
>> +------------------------------
>> +
>> +Following part is an example that record the value of all registers when Linux
>> +Kernel call function "vfs_readdir".
>
> The following is an example that records the value of all registers when
> "vfs_readdir" is called.
>
>> +
>> +--------------------------------------------------------------------------------
>> +(gdb) trace vfs_readdir
>
> Is this on the server or remote? Or do you mean that you can do this
> from either one or the other.
>
> Again, I'm a bit confused ;)
>
>
>> +Tracepoint 1 at 0xc01a1ac0: file
>> +/home/teawater/kernel/linux-2.6/fs/readdir.c, line 23.
>> +(gdb) actions
>> +Enter actions for tracepoint 1, one per line.
>> +End with a line saying just "end".
>> +>collect $reg
>> +>end
>> +(gdb) tstart
>> +(gdb) shell ls
>
> I take it that this is run on the remote (target) box. I don't think
> this would work on the server, would it?
>
>> +(gdb) tstop
>> +(gdb) tfind
>> +Found trace frame 0, tracepoint 1
>> +#0 0xc01a1ac1 in vfs_readdir (file=0xc5528d00, filler=0xc01a1900 <filldir64>,
>> + buf=0xc0d09f90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
>> +23 /home/teawater/kernel/linux-2.6/fs/readdir.c: No such file or
>> directory.
>> + in /home/teawater/kernel/linux-2.6/fs/readdir.c
>> +(gdb) info reg
>> +eax 0xc5528d00 -984445696
>> +ecx 0xc0d09f90 -1060069488
>> +edx 0xc01a1900 -1072031488
>> +ebx 0xfffffff7 -9
>> +esp 0xc0d09f8c 0xc0d09f8c
>> +ebp 0x0 0x0
>> +esi 0x8061480 134616192
>> +edi 0xc5528d00 -984445696
>> +eip 0xc01a1ac1 0xc01a1ac1 <vfs_readdir+1>
>> +eflags 0x286 [ PF SF IF ]
>> +cs 0x60 96
>> +ss 0x8061480 134616192
>> +ds 0x7b 123
>> +es 0x7b 123
>> +fs 0x0 0
>> +gs 0x0 0
>> +(gdb) tfind
>> +Found trace frame 1, tracepoint 1
>> +0xc01a1ac1 23 in /home/teawater/kernel/linux-2.6/fs/readdir.c
>> +(gdb) info reg
>> +eax 0xc5528d00 -984445696
>> +ecx 0xc0d09f90 -1060069488
>> +edx 0xc01a1900 -1072031488
>> +ebx 0xfffffff7 -9
>> +esp 0xc0d09f8c 0xc0d09f8c
>> +ebp 0x0 0x0
>> +esi 0x8061480 134616192
>> +edi 0xc5528d00 -984445696
>> +eip 0xc01a1ac1 0xc01a1ac1 <vfs_readdir+1>
>> +eflags 0x286 [ PF SF IF ]
>> +cs 0x60 96
>> +ss 0x8061480 134616192
>> +ds 0x7b 123
>> +es 0x7b 123
>> +fs 0x0 0
>> +gs 0x0 0
>> +--------------------------------------------------------------------------------
>> +
>> +
>> +
>> +Get the value of variable from Kernel
>> +-------------------------------------
>> +
>> +Following part is an example that record the value of "jiffies_64" when Linux
>> +Kernel call function "vfs_readdir".
>
> The following is an example that records the value of "jiffies_64" when
> the function "vfs_readdir" is called:
>
>> +
>> +--------------------------------------------------------------------------------
>> +(gdb) trace vfs_readdir
>> +Tracepoint 1 at 0xc01ed740: file
>> /home/teawater/kernel/linux-2.6/fs/readdir.c, line 24.
>> +(gdb) actions
>> +Enter actions for tracepoint 1, one per line.
>> +End with a line saying just "end".
>> +>collect jiffies_64
>> +>collect file->f_path.dentry->d_iname
>> +>end
>> +(gdb) tstart
>> +(gdb) shell ls
>> +arch drivers include kernel mm Module.symvers
>> security System.map virt
>> +block firmware init lib modules.builtin net
>> sound t vmlinux
>> +crypto fs ipc Makefile modules.order scripts
>> source usr vmlinux.o
>> +(gdb) tstop
>> +(gdb) tfind
>> +Found trace frame 0, tracepoint 1
>> +#0 0xc01ed741 in vfs_readdir (file=0xf4063000, filler=0xc01ed580
>> <filldir64>, buf=0xd6dfdf90)
>> + at /home/teawater/kernel/linux-2.6/fs/readdir.c:24
>> +24 {
>> +(gdb) p jiffies_64
>> +$1 = 4297248706
>> +(gdb) p file->f_path.dentry->d_iname
>> +$1 = "b26", '\000' <repeats 28 times>
>> +--------------------------------------------------------------------------------
>> +
>> +
>> +
>> +How to use use tracepoint condition
>> +-----------------------------------
>> +
>> +http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoint-Conditions.html
>> +Like the breakpoint, we can set condition to tracepoint. But it speed is more
>> +fast than breakpoint's condition because KGTP do all the condition check.
>
> Like breakpoints, conditions can be set to tracepoints. The speed of
> tracepoints is faster than breakpoints because KGTP can do all the
> condition checks.
>
>> +For example:
>> +
>> +------------------------------
>> +(gdb) trace handle_irq
>> +(gdb) condition 1 (irq == 47)
>> +------------------------------
>> +
>> +This action of tracepoint 1 will work only when irq number is 47.
>> +
>> +
>> +
>> +How to use trace state variables
>> +--------------------------------
>> +
>> +http://sourceware.org/gdb/current/onlinedocs/gdb/Trace-State-Variables.html
>> +Tracepoint have special variables. You can trace it or use it in tracepoint
>> +condition.
>
> Tracepoints have special variables. The variables can be traced
> directly, or used in tracepoint conditions.
>
>> +Note that only GDB 7.2.1 and later version support use trace state variables
>
> Note that only GDB versions 7.2.1 and later support the sue of trace
> state variables directly,..
>
>> +directly, the old version GDB just can show the value of trace state variables
>
> , the old version of GDB can show the value of trace ...
>
>> +through command "info tvariables".
>> +
>> +
>> +Simple trace state variables
>> +----------------------------
>> +
>> +Define a trace state variable $c.
>> +
>> +-------------------
>> +(gdb) tvariable $c
>> +-------------------
>> +
>> +Trace state variable $c created, with initial value 0.
>
> ... variable $c is created with initial value 0.
>
>> +Use it in the action. Follow action will use $c to count how much irq happen
>> +in Kernel.
>
> The following action uses $c to count how many irqs happened in the
> Kernel.
>
>> +
>> +-----------------------------------------------------------------------
>> +(gdb) trace handle_irq
>> +(gdb) actions
>> +Enter actions for tracepoint 3, one per line.
>> +End with a line saying just "end".
>> +>collect $c #Save current value of $c to the trace frame buffer.
>> +>teval $c=$c+1 #Increase the $c.
>> +>end
>> +-----------------------------------------------------------------------
>> +
>> +You can get the current value of $c when the trace is running or stop.
>
> ... of $c while the trace is running or stopped.
>
>> +
>> +----------------------------------
>> +(gdb) tstart
>> +(gdb) info tvariables
>> +$c 0 31554
>> +(gdb) p $c
>> +$5 = 33652
>> +(gdb) tstop
>> +(gdb) p $c
>> +$9 = 105559
>> +----------------------------------
>> +
>> +When use the tfind parse the trace frame buffer, if the value of trace state
>> +variable is collect. You can use it.
>
> I do not understand the above at all.
>
>> +
>> +------------------------------
>> +(gdb) tstop
>> +(gdb) tfind
>> +(gdb) info tvariables
>> +$c 0 0
>> +(gdb) p $c
>> +$6 = 0
>> +(gdb) tfind 100
>> +(gdb) p $c
>> +$7 = 100
>> +------------------------------
>
> Nor do I understand the above example.
>
>
>> +
>> +
>> +Special trace state variables $current_task, $current_thread_info, $cpu_id,
>> +$dump_stack, $printk_level, $printk_format, $printk_tmp and $clock
>> +--------------------------------------------------------------------------
>> +
>> +KGTP have special trace state variables $current_task, $current_thread_info,
>
> KGTP has special ...
>
>> +$cpu_id and $clock can very easy to access to some special value. You can see
>
> ... $clock that can be used to access their special values. (?)
>
>> +them when GDB connect to the KGTP. You can use them in tracepoint condition
>
> connects conditions
>
>> +or actions.
>> +Access $current_task in tracepoint condition and action will get the return
>> +of get_current().
>
> The use of $current_task in tracepoint conditions will return the action
> of get_current().
>
>> +Access $current_thread_info in tracepoint condition and action will get the
>> +return of current_thread_info().
>
> Same
>
>> +Access $cpu_id in tracepoint condition and action will get the return of
>> +smp_processor_id().
>
> Same
>
>> +Access $clock in tracepoint condition and action will get the return of
>> +local_clock() that return the timestamp in nanoseconds.
>
> Same
>
>> +
>> +And KGTP have other special trace state variables $dump_stack, $printk_level,
>
> KGTP has other special trace state variables ...
>
>> +$printk_format and $printk_tmp, all of them are for output value directly that
>
> $print_tmp. Their output is explained in
> "Howto ..."
>
>> +will introduce in "Howto let tracepoint output value directly".
>> +
>> +Following example will count process 16663 call how many sys_read in $c
>> +and collect the struct thread_info of current task:
>
> I'm confused by what $c is to give it a proper translation.
>
>> +
>> +--------------------------------------------------------------------------------
>> +(gdb) trace vfs_read if ((*(struct task_struct *)$current_task)->pid == 16663)
>> +(gdb) tvariable $c
>> +(gdb) actions
>> +Enter actions for tracepoint 4, one per line.
>> +End with a line saying just "end".
>> +>teval $c=$c+1
>> +>collect (*(struct thread_info *)$current_thread_info)
>> +>end
>> +(gdb) tstart
>> +(gdb) info tvariables
>> +Name Initial Current
>> +$c 0 184
>> +$current_task 0 <unknown>
>> +$current_thread_info 0 <unknown>
>> +$cpu_id 0 <unknown>
>> +(gdb) tstop
>> +(gdb) tfind
>> +(gdb) p *(struct thread_info *)$current_thread_info
>> +$10 = {task = 0xf0ac6580, exec_domain = 0xc07b1400, flags = 0, status
>> = 0, cpu = 1, preempt_count = 2, addr_limit = {
>> + seg = 4294967295}, restart_block = {fn = 0xc0159fb0
>> <do_no_restart_syscall>, {{arg0 = 138300720, arg1 = 11,
>> + arg2 = 1, arg3 = 78}, futex = {uaddr = 0x83e4d30, val = 11,
>> flags = 1, bitset = 78, time = 977063750,
>> + uaddr2 = 0x0}, nanosleep = {index = 138300720, rmtp = 0xb,
>> expires = 335007449089}, poll = {
>> + ufds = 0x83e4d30, nfds = 11, has_timeout = 1, tv_sec = 78,
>> tv_nsec = 977063750}}},
>> + sysenter_return = 0xb77ce424, previous_esp = 0, supervisor_stack =
>> 0xef340044 "", uaccess_err = 0}
>> +--------------------------------------------------------------------------------
>
>
> This is all I have time for this now, but this work looks very
> promising. When I get some more time, I will have to start playing with
> it.
>
> Thanks!
>
> -- Steve
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/