Re: [LTP] [PATCH 1/8] Scaling msgmni to the amount of lowmem

From: Nadia Derbey
Date: Tue Feb 19 2008 - 12:17:42 EST

Next message: Kok, Auke: "Re: [2.6.25-rc2] e100: Trying to free already-free IRQ 11 duringsuspend ..."
Previous message: Pierre Ossman: "random wedges with 2.6.25-rc*"
In reply to: Subrata Modak: "Re: [LTP] [PATCH 1/8] Scaling msgmni to the amount of lowmem"
Next in thread: Matt Helsley: "Re: [LTP] [PATCH 1/8] Scaling msgmni to the amount of lowmem"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Subrata Modak wrote:

Nadia Derbey wrote:

Andrew Morton wrote:

On Mon, 11 Feb 2008 15:16:47 +0100 Nadia.Derbey@xxxxxxxx wrote:

[PATCH 01/08]

This patch computes msg_ctlmni to make it scale with the amount of lowmem.
msg_ctlmni is now set to make the message queues occupy 1/32 of the available
lowmem.

Some cleaning has also been done for the MSGPOOL constant: the msgctl man page
says it's not used, but it also defines it as a size in bytes (the code
expresses it in Kbytes).

Something's wrong here. Running LTP's msgctl08 (specifically:
ltp-full-20070228) cripples the machine. It's a 4-way 4GB x86_64.

http://userweb.kernel.org/~akpm/config-x.txt
http://userweb.kernel.org/~akpm/dmesg-x.txt

Normally msgctl08 will complete in a second or two. With this patch I
don't know how long it will take to complete, and the machine is horridly
bogged down. It does recover if you manage to kill msgctl08. Feels like
a terrible memory shortage, but there's plenty of memory free and it isn't
swapping.

Before the patchset, msgctl08 used to be run with the old msgmni value: 16. Now it is run with a much higher msgmni value (1746 in my case), since it scales to the memory size.
When I call "msgctl08 100000 16" it completes fast.

Doing the follwing on the ref kernel:
echo 1746 > /proc/sys/kernel/msgmni
msgctl08 100000 1746

makes th test block too :-(

Will check to see where the problem comes from.

Well, actually, the test does not block, it only takes much much more time to be executed:

doing this:
date; ./msgctl08 100000 XXX; date

gives us the following results:
XXX 16 32 64 128 256 512 1024 1746
time(secs) 2 4 8 16 32 64 132 241

XXX is the # of msg queues to be created = # of processes to be forked as readers = # of processes to be created as writers
time is approximative since it is obtained by a "date" before and after.

XXX used to be 16 before the patchset ---> 1st column
--> 16 processes forked as reader
--> + 16 processes forked as writers
--> + 16 msg queues
XXX = 1746 (on my victim) after the patchset ---> last column
--> 1746 reader processes forked
--> + 1746 writers forked
--> + 1746 msg queues created

The same tests on the ref kernel give approximatly the same results.

So if we don't want this longer time to appear as a regression, the LTP should be changed:
1) either by setting the result of get_max_msgqueues() as the MSGMNI constant (16) (that would be the best solution in my mind)
2) or by warning the tester that it may take a long time to finish.

There would be 3 tests impacted:

kernel/syscalls/ipc/msgctl/msgctl08.c
kernel/syscalls/ipc/msgctl/msgctl09.c
kernel/syscalls/ipc/msgget/msgget03.c

We will change the test case if need that be. Nadia, kindly send us the
patch set which will do the necessary changes.

Regards--
Subrata

Subrata,

You'll find the patch in attachment.
FYI I didn't change msgget03.c since we need to get the actual max value in order to generate an error.

Regards,
Nadia

Since msgmni now scales to the memory size, it may reach big values.
To avoid forking 2*msgmni processes and create msgmni msg queues, do not take
msgmni from procfs anymore.
Just define it as 16 (which is the MSGMNI constant value in linux/msg.h)

Also fixed the Makefiles in ipc/lib and ipc/msgctl: there was no dependency
on the lib/ipc*.h header files.

Signed-off-by: Nadia Derbey <Nadia.Derbey@xxxxxxxx>

---
testcases/kernel/syscalls/ipc/lib/Makefile | 3 +++
testcases/kernel/syscalls/ipc/lib/ipcmsg.h | 2 ++
testcases/kernel/syscalls/ipc/msgctl/Makefile | 3 +++
testcases/kernel/syscalls/ipc/msgctl/msgctl08.c | 23 ++---------------------
testcases/kernel/syscalls/ipc/msgctl/msgctl09.c | 24 ++----------------------
5 files changed, 12 insertions(+), 43 deletions(-)

Index: ltp-full-20080131/testcases/kernel/syscalls/ipc/msgctl/msgctl08.c
===================================================================
--- ltp-full-20080131.orig/testcases/kernel/syscalls/ipc/msgctl/msgctl08.c 2006-02-11 05:46:36.000000000 +0100
+++ ltp-full-20080131/testcases/kernel/syscalls/ipc/msgctl/msgctl08.c 2008-02-19 18:45:27.000000000 +0100
@@ -50,6 +50,7 @@
#include <sys/msg.h>
#include "test.h"
#include "usctest.h"
+#include "ipcmsg.h"

void setup();
void cleanup();
@@ -479,26 +480,6 @@ static int get_used_msgqueues()
return used_queues;
}

-/** Get the max number of message queues allowed on system */
-static int get_max_msgqueues()
-{
- FILE *f;
- char buff[BUFSIZE];
-
- /* Get the max number of message queues allowed on system */
- f = fopen("/proc/sys/kernel/msgmni", "r");
- if (!f){
- tst_resm(TBROK,"Could not open /proc/sys/kernel/msgmni");
- tst_exit();
- }
- if (!fgets(buff, BUFSIZE, f)) {
- tst_resm(TBROK,"Could not read /proc/sys/kernel/msgmni");
- tst_exit();
- }
- fclose(f);
- return atoi(buff);
-}
-
/***************************************************************
* setup() - performs all ONE TIME setup for this test.
*****************************************************************/
@@ -520,7 +501,7 @@ setup()
*/
TEST_PAUSE;

- MSGMNI = get_max_msgqueues() - get_used_msgqueues();
+ MSGMNI = MAX_MSGQUEUES - get_used_msgqueues();
if (MSGMNI <= 0){
tst_resm(TBROK,"Max number of message queues already used, cannot create more.");
cleanup();
Index: ltp-full-20080131/testcases/kernel/syscalls/ipc/msgctl/msgctl09.c
===================================================================
--- ltp-full-20080131.orig/testcases/kernel/syscalls/ipc/msgctl/msgctl09.c 2006-02-11 05:46:36.000000000 +0100
+++ ltp-full-20080131/testcases/kernel/syscalls/ipc/msgctl/msgctl09.c 2008-02-19 18:46:44.000000000 +0100
@@ -49,6 +49,7 @@
#include <unistd.h>
#include "test.h"
#include "usctest.h"
+#include "ipcmsg.h"

#define MAXNREPS 1000
#ifndef CONFIG_COLDFIRE
@@ -649,26 +650,6 @@ static int get_used_msgqueues()
return used_queues;
}

-/** Get the max number of message queues allowed on system */
-static int get_max_msgqueues()
-{
- FILE *f;
- char buff[BUFSIZE];
-
- /* Get the max number of message queues allowed on system */
- f = fopen("/proc/sys/kernel/msgmni", "r");
- if (!f){
- tst_resm(TBROK,"Could not open /proc/sys/kernel/msgmni");
- tst_exit();
- }
- if (!fgets(buff, BUFSIZE, f)) {
- tst_resm(TBROK,"Could not read /proc/sys/kernel/msgmni");
- tst_exit();
- }
- fclose(f);
- return atoi(buff);
-}
-
/***************************************************************
* setup() - performs all ONE TIME setup for this test.
*****************************************************************/
@@ -689,12 +670,11 @@ setup()
*/
TEST_PAUSE;

- MSGMNI = get_max_msgqueues() - get_used_msgqueues();
+ MSGMNI = MAX_MSGQUEUES - get_used_msgqueues();
if (MSGMNI <= 0){
tst_resm(TBROK,"Max number of message queues already used, cannot create more.");
cleanup();
}
-
}

Index: ltp-full-20080131/testcases/kernel/syscalls/ipc/lib/ipcmsg.h
===================================================================
--- ltp-full-20080131.orig/testcases/kernel/syscalls/ipc/lib/ipcmsg.h 2005-12-22 21:18:23.000000000 +0100
+++ ltp-full-20080131/testcases/kernel/syscalls/ipc/lib/ipcmsg.h 2008-02-19 18:51:38.000000000 +0100
@@ -41,6 +41,8 @@ void setup(void);
#define MSGSIZE 1024 /* a resonable size for a message */
#define MSGTYPE 1 /* a type ID for a message */

+#define MAX_MSGQUEUES 16 /* MSGMNI as defined in linux/msg.h */
+
typedef struct mbuf { /* a generic message structure */
long mtype;
char mtext[MSGSIZE + 1]; /* add 1 here so the message can be 1024 */
Index: ltp-full-20080131/testcases/kernel/syscalls/ipc/lib/Makefile
===================================================================
--- ltp-full-20080131.orig/testcases/kernel/syscalls/ipc/lib/Makefile 2006-08-21 08:58:39.000000000 +0200
+++ ltp-full-20080131/testcases/kernel/syscalls/ipc/lib/Makefile 2008-02-19 18:50:19.000000000 +0100
@@ -19,6 +19,7 @@
SRCS = libipc.c
OBJS = $(SRCS:.c=.o)
LIBIPC = ../libipc.a
+LIBIPC_HEADERS = ipcmsg.h ipcsem.h

CFLAGS += -I../../../../../include -Wall

@@ -27,6 +28,8 @@ all: $(LIBIPC)
$(LIBIPC): $(OBJS)
$(AR) -rc $@ $(OBJS)

+$(OBJS): $(LIBIPC_HEADERS)
+
install:

clean:
Index: ltp-full-20080131/testcases/kernel/syscalls/ipc/msgctl/Makefile
===================================================================
--- ltp-full-20080131.orig/testcases/kernel/syscalls/ipc/msgctl/Makefile 2006-08-23 07:46:27.000000000 +0200
+++ ltp-full-20080131/testcases/kernel/syscalls/ipc/msgctl/Makefile 2008-02-19 19:02:26.000000000 +0100
@@ -18,12 +18,15 @@

CFLAGS += -I../lib -I../../../../../include -Wall
LDLIBS += -L../../../../../lib -lltp -L.. -lipc
+LIBIPC_HEADERS = ../lib/ipcmsg.h

SRCS = $(wildcard *.c)
TARGETS = $(patsubst %.c,%,$(SRCS))

all: $(TARGETS)

+$(TARGETS): $(LIBIPC_HEADERS)
+
install:
@set -e; for i in $(TARGETS); do ln -f $$i ../../../../bin/$$i ; done

Next message: Kok, Auke: "Re: [2.6.25-rc2] e100: Trying to free already-free IRQ 11 duringsuspend ..."
Previous message: Pierre Ossman: "random wedges with 2.6.25-rc*"
In reply to: Subrata Modak: "Re: [LTP] [PATCH 1/8] Scaling msgmni to the amount of lowmem"
Next in thread: Matt Helsley: "Re: [LTP] [PATCH 1/8] Scaling msgmni to the amount of lowmem"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]