Re: [PATCH v33 1/7] sched/psi: Preserve sleep-persistent PSI flags on off-CPU sleeping dequeues

[Tejun Heo]

Hello, John.

This patch also fixes a trigger without proxy execution, through
switching_from_fair(). A fair task sleeps in iowait and is delay-dequeued.
sched_setscheduler() then moves it to SCHED_FIFO, and switching_from_fair()
dequeues it with DEQUEUE_SLEEP | DEQUEUE_DELAYED. TSK_ONCPU is clear, so
since c2ae8b0df2d1 ("sched/core: Fix psi_dequeue() for Proxy Execution")
psi_dequeue() clears all of its psi flags, TSK_IOWAIT included. The wakeup's
psi_enqueue() trips the check:

  psi: inconsistent task state! task=1318:python3 cpu=3 psi_flags=0 clear=1 set=4

Tested on mainline 4dd16e31135e with CONFIG_SCHED_PROXY_EXEC disabled, each
run a fresh boot. Without the patch, the reproducer below hit the warning on
10 of 10 boots within about a second. With only 1/7 applied, 0 of 10
sixty-second boots hit it, while switching_from_fair() dequeued a delayed
iowait task about 3,500 to 16,000 times per boot.

Tested-by: Tejun Heo <tj@xxxxxxxxxx>

The reproducer needs root and a block device for O_DIRECT reads, and takes
[secs [cpu [blockdev]]].

  #!/bin/bash
  # psi-delayed.sh [secs [cpu [blockdev]]]
  SECS=${1:-60} CPU=${2:-3} DEV=${3:-/dev/sda}
  PY=$(mktemp --suffix=.py)
  trap 'kill -9 $(jobs -p) 2>/dev/null; rm -f "$PY"' EXIT
  cat > "$PY" <<'EOF'
  import mmap, os, random, sys, time
  if sys.argv[1] == "io":
  	# spin ~3ms so the entity runs past its share, then a 4K O_DIRECT read
  	fd = os.open(sys.argv[2], os.O_RDONLY | os.O_DIRECT)
  	nblk = os.lseek(fd, 0, os.SEEK_END) // 4096
  	buf = mmap.mmap(-1, 4096)
  	while True:
  		t = time.perf_counter() + 0.003
  		while time.perf_counter() < t:
  			pass
  		os.preadv(fd, [buf], random.randrange(nblk) * 4096)
  else:
  	# flip the I/O task between SCHED_FIFO and SCHED_OTHER
  	pid, end = int(sys.argv[2]), time.time() + float(sys.argv[3])
  	while time.time() < end:
  		os.sched_setscheduler(pid, os.SCHED_FIFO, os.sched_param(1))
  		os.sched_setscheduler(pid, os.SCHED_OTHER, os.sched_param(0))
  EOF
  for i in 1 2; do taskset -c "$CPU" bash -c 'while :; do :; done' & done
  taskset -c "$CPU" python3 -I "$PY" io "$DEV" &
  IOPID=$!
  sleep 1
  taskset -c $(( (CPU + 2) % $(nproc) )) python3 -I "$PY" switch "$IOPID" "$SECS"
  echo "io task pid $IOPID"
  dmesg | grep 'psi: inconsistent' || echo "not reproduced"

Thanks.

--
tejun