Re: [PROBLEM] hard-lock with kmemtrace, relayfs, and splice

From: Eduard - Gabriel Munteanu
Date: Sat Oct 25 2008 - 10:04:36 EST


On Sat, Oct 25, 2008 at 08:56:07AM +0800, Peter Teoh wrote:
> ok. just ignore it. i think based on "git log" of
> "git://repo.or.cz/linux-2.6/kmemtrace.git":
>
> commit 0441e5ff6ab71cf7a3e9ec3116f14d0fd7d20d51
> Author: Eduard - Gabriel Munteanu <eduard.munteanu@xxxxxxxxxxx>
> Date: Thu Jul 10 20:20:05 2008 +0300
>
> kmemtrace: SLOB hooks.
>
> the repo seemed way out of date. But a clone of Pekka's slab-2.6's
> topic/kmemtrace branch have no problem. The steps are as follows:
>
> 1. git clone git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6.git
> 2. git checkout -m origin/topic/kmemtrace
> 3. git checkout -b kmemtrace

Hi,

Yes, that repo may be outdated, please use Pekka's for the latest
working stuff.

> git log gives:
>
> commit 51b19be3535c8fbcce6b6f838d89b9a6a4cc5b92
> Author: Tom Zanussi <zanussi@xxxxxxxxxxx>
> Date: Fri Oct 10 23:58:51 2008 -0500
>
> relayfs: fix infinite loop with splice()
>
> But the userspace tool I still get it from: git://repo.or.cz/kmemtrace-user.git
>
> Correct?

Yes, I maintain this one and it's up-to-date.

> Now I have some problems:
>
> a. I would like to extract out all the commit as diff - may I know
> how to do that? ("git log" only gives the descriptive part).

You can use "git-log -p" to get diffs along with commit history.

> b. boot-time memory profiling....how can it be done (or extracted
> out)? (kmemtrace-user does not have that?)

If you enable kmemtrace at boot-time using the "kmemtrace.enabled=yes"
parameter, kmemtrace will start logging just after kmem_cache_init()
finishes. There are plans for making tracing possible even earlier,
however it involves quite a big rework of how relay allocates its
buffers.

> c. please provide some pointers to documentation: how do I
> interpret the following:
>
> Allocation #83468 (CPU0) already exists!
> by __kmalloc_track_caller+25
> last touched by __kmalloc_track_caller
> Allocation #83740 (CPU0) already exists!
> by __kmalloc_track_caller+25
> last touched by __kmalloc_track_caller

The userspace app writes out those messages when it looks like a memory
region has been allocated twice. In reality, it's most likely a free
hasn't been traced (because a misc free function isn't traced).

> For my purpose, I would like to trace how and where is memory
> allocated - just one one single userspace program which access the
> mmap("/dev/mem") to read the memory. How can it be done? "Tracing
> memory" in my context will mean list all the kernel-function +
> memory-allocated-within-the-function + its-PFN-or-PTE-information (to
> pinpoint the exact physical page). My understanding is that these
> can be done, via checking the value of "current", as each userspace
> program started will have a unique "current" context value, and
> therefore within any kernel function, just check this value, to focus
> tracing on just one particular userspace program. Correct?

I have made provisions for exporting the whole stack trace, however it
is not yet implemented.

> d. The following:
>
> /sys/kernel/debug/kmemtrace>ls -al
> ???? 0
> drwxr-xr-x 2 root root 0 2008-10-25 .
> drwxr-xr-x 9 root root 0 2008-10-25 ..
> -r-------- 1 root root 0 2008-10-25 abi_version
> -r-------- 1 root root 2946936192 2008-10-25 cpu0
> -r-------- 1 root root 1701263952 2008-10-25 cpu1
> -rw------- 1 root root 0 2008-10-25 enabled
> -r-------- 1 root root 0 2008-10-25 total_overruns
>
> So the information is stored in memory, right? Is it possible to
> reset it? I don't want these information to clog up the memory.

Relay uses fixed size buffers, so there is no point in clearing them.

> e. The balance between cpu0 and cpu1 does not seemed equal, any
> implications on the scheduler/ IO Scheduler? or memory allocation
> scheduler (if there exists one :-))....sorry about that....my
> knowledge is limited.

Much of the booting sequence is done on a single CPU, then the kernel
switches to SMP. It is natural to see one CPU log more allocations.

> --
> Regards,
> Peter Teoh

Thanks for testing.


Cheers,
Eduard

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/