> 0.000556 read(5, "\v\245\10\0\0\3\35\340\0"..., 8192) = 8192 <3176.731624>
> 3176.732664 lseek(5, 89088000, SEEK_SET) = 89088000 <0.000124>
>
> The last call took almost an hour to return. The preceeding hang
> (about an hour earlier) recovered and continured within 10 minutes.
>
> According to ps, the solid process was blocked in WCHAN 1261d9:
>
> FLAGS UID PID PPID PRI NI VSZ RSS WCHAN STAT TT TIME COMMA
> 100130 0 140 861 0 0 12240 12096 1261d9 D ? 8:27 /var/soli
>
> According to System.map, that's somewhere in the middle of
> try_to_free_buffer, if I'm reading ps right:
>
> 00125fb8 T generic_readpage
> 00126038 t grow_buffers
> > 00126120 T try_to_free_buffer
> 0012635c T show_buffers
> 00126434 T buffer_init
I think I was able to get the line number of fs/buffer.c for that address.
It seems to be line 348
344 static inline void remove_from_free_list(struct buffer_head * bh)
345 {
346 int isize = BUFSIZE_INDEX(bh->b_size);
347 if (!(bh->b_prev_free) || !(bh->b_next_free))
348 panic("VFS: Free block list corrupted");
349 if(bh->b_dev != B_FREE)
350 panic("Free list corrupted");
This inline function is "called" from line 367
364 static inline void remove_from_queues(struct buffer_head * bh)
365 {
366 if(bh->b_dev == B_FREE) {
367 remove_from_free_list(bh); /* Free list entries should not be
368 in the hash queue */
and this is expanded on line 1423 of buffer.c in try_to_free_buffer:
1420 if (p == *bhp) /* Was this the last in the list? */
1421 *bhp = NULL;
1422 }
1423 remove_from_queues(p);
1424 put_unused_buffer_head(p);
1425 } while (tmp != bh);
I will present below the way I have reached that conclusion because either:
1. It is wrong, and I would like to know it.
OR
2. It is a good way of tracing kernel addresses and may be put into some
documentation.
I. Find the address in the System.map and note the function name
II. Locate the source file defining this function
cd /usr/src/linux
grep try_to_free_buffer *.c */*.c */*/*.c */*/*/*.c */*/*/*/*.c */*/*/*/*/*.c
After some analysis you find it in fs/buffer.c
III. Modify Makefile - add "-g" option to CC definition:
LD =$(CROSS_COMPILE)ld
CC =$(CROSS_COMPILE)gcc -D__KERNEL__ -g -I$(HPATH)
CPP =$(CC) -E
IV. configure and compile the kernel. Use the same configuration and kernel
version as running when you obtained the address. You may check if the
addresses in the new System.map are consistent with the ones used before.
V. Run
objdump -d -l --prefix-addresses fs/buffer.o > buffer.dump
VI. Substract the start address of the function from System.map from your
address (0x1261d9 - 0x126120 = 0xb9) and look it up in buffer.dump
/scratch/kernel/linux/fs/buffer.c:347
00001e23 <try_to_free_buffer+a7> movl 0x3c(%ecx),%edx
00001e26 <try_to_free_buffer+aa> testl %edx,%edx
00001e28 <try_to_free_buffer+ac> je 00001e31 <try_to_free_buffer+b5>
00001e2a <try_to_free_buffer+ae> movl 0x18(%ecx),%eax
00001e2d <try_to_free_buffer+b1> testl %eax,%eax
00001e2f <try_to_free_buffer+b3> jne 00001e3c <try_to_free_buffer+c0>
/scratch/kernel/linux/fs/buffer.c:348
00001e31 <try_to_free_buffer+b5> pushl $0x31
00001e36 <try_to_free_buffer+ba> call 00001e37 <try_to_free_buffer+bb>
00001e3b <try_to_free_buffer+bf> nop
/scratch/kernel/linux/fs/buffer.c:351
00001e3c <try_to_free_buffer+c0> cmpl $0x0,0x24(,%ebx,4)
00001e44 <try_to_free_buffer+c8> jne 00001e50 <try_to_free_buffer+d4>
VII. The line number 348 is found. It may be not always correct because gcc
may perform optimization.
VIII. If the line you have found is inside some macro or inline function try
to look for line numbers before and after the point you have found to locate
the place where these inline functions were "called".
I am sure, that method is known, but probably not by all of the list
subscribers. Btw, I have used 2.0.31 compilled with CONFIG_M586=y. My
System.map is consistent (all functions shifted -0x134 bytes) with the
fragment reported by you.
Cheers,
-- Tomasz Motylewski