Re: [RFC PATCH v2 1/1] drm/xe: keep VM-bound WC BOs resident during reclaim

[Neil Zhong]

Hi Matt,

> Can you give me instructions on how to recreate this on our end and your
> machine, memory details? I have a bunch of various reproducers which I
> have been using for shrinker work and the more the better.

Yes. The test system and workload are as follows.

  - Panther Lake iGPU using shared system memory
  - 8 GiB installed memory; /proc/meminfo reports 7,723,392 KiB
  - 4 KiB base pages
  - four zram swap devices, 3,858,416 KiB in total
  - Linux 6.18.15, x86_64, PREEMPT_DYNAMIC
  - X11 fullscreen output at 3840x2160 and 60 Hz
  - a 3840x2160, 60 fps, HEVC HDR video played continuously in a loop

The player uses hardware decoding. Each decoded frame is imported through
DMA-BUF and processed by libplacebo/OpenGL for HDR tone mapping before
presentation. It calls glFlush() for every rendered frame. During steady
playback, the player accounts for about 1.85 GiB of logical Xe BO
allocation, including about 672 MiB reported as shared.

The player is not currently public. However, an equivalent pipeline should
reproduce the condition if it keeps the decoded surfaces, HDR intermediate
render targets and presentation surfaces alive, rather than creating a
small synthetic BO set. 

No additional memory-pressure tool was used for the first reproduction.
I start the video, let its working set reach steady state, and then capture
ten minutes while playback continues. On the 8 GiB system, the priority-only
run had MemAvailable between 2.89 and 3.15 GiB, while 97.8% of kswapd wakeups
were for order-10 allocations. Thus this reproduces without forcing an
order-0 shortage.

I collected the trace with:

  sudo ./capture_xe_memory_churn.sh \
          -t 600 \
          -s 0.2 \
          -o ./xe-memory-churn

`-t 600` records ten minutes. `-t 0` can instead be used to record until
Ctrl-C. The 0.2 second option is only the /proc and TTM-pool sampling
interval; ftrace events are recorded continuously.

The player also logs one line after each frame submission in this form:

  gl_sw_submit_frame timing: ... Flush=123.456 ms

The log prefix contains the wall-clock timestamp. An equivalent reproducer
can record the time immediately after glFlush() returns and the measured
duration. The trace script inserts a wall-clock epoch marker into a
mono_raw ftrace stream so that the two timelines can be aligned.

The figures in my previous email came from the 6.18.15 kernel with [9]-[13]
backported, the fragmentation check applied to direct reclaim as well, and
the high-plus-min watermark experiment described there. The default-device
watermark capture used vm.min_free_kbytes=131072. The follow-up used 50000;
the workload and trace procedure were otherwise unchanged.

> Also a quick write up how you extracted these numbers from reproducer so
> I can recreate on my end.

The capture script creates temporary entry and return kprobes for:

  ttm_tt_backup()
  ttm_tt_restore()
  ttm_pool_alloc()
  ttm_pool_free()
  ttm_pool_shrink()

It also traces the Xe shrinker, TTM restore and cache-attribute functions,
kswapd and direct-reclaim events, compaction, and allocation
fragmentation.

At ttm_tt_backup() entry, the probe records the ttm_tt pointer and
num_pages. At return, it records the positive return value, which is the
number of pages actually backed up. I sum those successful return values,
not the requested page count, when reporting backup volume. A later
ttm_tt_restore() is matched to the most recent successful backup using the
same ttm_tt pointer. That provides per-object backup-to-restore latency and
repeated-cycle counts.

The byte-volume calculation is:

  backup bytes = sum(successful backup return pages) * PAGE_SIZE
  restore bytes = sum(num_pages for matched restores) * PAGE_SIZE

The reported backup-plus-restore volume is the sum of those two values.
It is cumulative migration/copy traffic, not resident memory and not net
memory freed. A shmem backup remains resident system memory unless those
shmem pages are subsequently swapped out.

> This customer, in particular, utilizes priority bands to express this
> heuristic (e.g., the compositor is the highest priority, any
> non-privileged UI-related content is normal priority, and everything
> else is low priority).

I think priority bands are useful for relative reclaim ordering, but they
do not by themselves express the guarantee needed here.

Consider a system with one large GPU workload. If nearly all reclaimable
BOs belong to that client and are placed in the high-priority band, the low
bands will contain few or no candidates. When enough memory is requested,
the shrinker must eventually enter the high band. In that case, high
priority delays reclaim but does not prevent it. The priority-only test
showed exactly this limitation: over ten minutes there were 6,743
successful backups and 6,693 restores, with 157 Flush calls over 16.667 ms
and a maximum Flush of 494.434 ms.

I am not suggesting that every BO of a high-priority client should be
unreclaimable. Whether reclaim is acceptable depends on the workload, and
the kernel cannot infer that semantic from WC, VM-bound state, client count
or BO size alone. For example, the driver-visible behavior of these two
workloads can look very similar:

  1. Foreground 4K60 HDR playback. Its active decoded surfaces, HDR render
     targets and presentation surfaces have a 16.667 ms deadline. Backing
     them up and restoring them causes an immediate and clearly visible
     product failure. These BOs should avoid eviction and shrinking while
     they are part of the active visual pipeline.

  2. Background image recognition or classification. Its BOs have no
     presentation deadline. Reclaiming them under system memory pressure
     is reasonable, even if the job later has to reconstruct its working
     set.

Priority bands cannot distinguish those cases if both clients assign their
current working set a high relative priority. This is why I think the
business semantic has to come from userspace. Priority bands can remain the
general ordering mechanism, while a separate, explicit and bounded
NO_SHRINK or latency-critical mark protects only the BOs in an active visual
pipeline. The mark should be removed as soon as a surface leaves that
working set.

> To be clear - this would be an addition to fixes discussed above, right?

Yes. I see explicit workload-semantic protection as an addition to the
core MM fragmentation/shrinker fixes and the Xe/TTM priority bands, not a
replacement for either. I agree that testing series 168651 together with
the priority bands is still useful for general working-set preservation.
It can reduce accidental reclaim, while an accounted NO_SHRINK contract
handles the smaller set for which a missed presentation deadline is not an
acceptable reclaim tradeoff.

Thanks,
Neil