net: cleanup_net is slow

[Andrey Konovalov]

Hi!

We're investigating some approaches to improve isolation of syzkaller
programs. One of the ideas is run each program in it's own user/net
namespace. However, while I was experimenting with this, I stumbled
upon a problem.

It seems that cleanup_net() might take a very long time to execute.

I've attached the reproducer and kernel .config that I used. Run as
"./a.out 1". The reproducer just forks and does unshare(CLONE_NEWNET)
in a loop. Note, that I have a lot of network-related configs enabled,
which causes a few interfaces to be set up by default.

What I see with this reproducer is that at first a huge number
(~200-300) net namespaces are created without any contention. But then
(probably when one of these namespaces gets destroyed) the program
hangs for a considerable amount of time (~100 seconds in my vm).
Nothing locks up inside the kernel and the CPU is mostly idle.

Adding debug printfs showed that the part that takes almost all of
that time is the lines between synchronize_rcu() and
mutex_unlock(&net_mutex) in cleanup_net. Running perf showed that the
cause of this might be a lot of calls to synchronize_net that happen
while executing those lines.

Is there any change that can be done to speed up the
creation/destruction of a huge number of net namespaces?

Running the reproducer with unshare(CLONE_NEWUSER) doesn't seem to
cause any delays.

Thanks!

Attachment: .config
Description: Binary data

#include <sys/ioctl.h>
#include <sys/mman.h>
#include <sys/mount.h>
#include <sys/prctl.h>
#include <sys/resource.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/wait.h>

#include <linux/capability.h>
#include <linux/if.h>
#include <linux/if_tun.h>
#include <linux/kvm.h>
#include <linux/sched.h>
#include <net/if_arp.h>

#include <assert.h>
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <grp.h>
#include <pthread.h>
#include <setjmp.h>
#include <signal.h>
#include <stdarg.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

void execute_one() {
	fprintf(stderr, "unshare start\n");
	unshare(CLONE_NEWNET);
	fprintf(stderr, "unshare end\n");
	exit(0);
}

void loop() {
	for (;;) {
		int pid = fork();
		if (pid == 0) {
			execute_one();
		}

		int status = 0;
		while (waitpid(-1, &status, __WALL) != pid) {
		}
	}
}

int main(int argc, char **argv) {
	if (argc != 2) {
		printf("Usage: %s <num>\n", argv[0]);
		exit(1);
	}

	int i;
	for (i = 0; i < atoi(argv[1]); i++) {
		int pid = fork();
		if (pid == 0) {
			loop();
		}
	}

	sleep(1000000);

	return 0;
}