Re: [Alsa-user] new source of MIDI playback slow-down identified- 5a03b051ed87e72b959f32a86054e1142ac4cf55 thp: use compaction in kswapdfor GFP_ATOMIC order > 0
From: Arthur Marsh
Date: Wed Feb 23 2011 - 06:41:44 EST
Mel Gorman wrote, on 23/02/11 19:45:
On Tue, Feb 22, 2011 at 08:43:25PM +0100, Andrea Arcangeli wrote:
On Wed, Feb 23, 2011 at 04:17:44AM +1030, Arthur Marsh wrote:
OK, these patches applied together against upstream didn't cause a crash
but I did observe:
significant slowdowns of MIDI playback (moreso than in previous cases,
and with less than 20 Meg of swap file in use);
Can this be replicated with on-board audio hardware or is there something
specific about the card? e.g. those typically driven by snd-intel8x0 or
snd_hda_intel
This card (Soundblaster Audigy 2 ZS - SB0350) has an on-board wavetable
synthesiser. To perform a test with sound hardware included on a
motherboard, one would need to connect the pc to an external synthesiser
to play the MIDI signal from the sound hardware on the motherboard.
To quote an earlier posting from Clemens Ladisch:
"The ALSA sequencer uses either the system timer or an HR timer at 1 kHz
to deliver MIDI commands (notes); the wavetable driver requires its own
interrupts in regular 5.3 ms intervals."
The wavetable interrupts are apparently being lost. I am not getting
lost MIDI events (ie notes going missing or being stuck on), but
slowdown in play-back.
Unfortunately the original mail is a bit light on details on how this was
reproduced and I didn't find a thread with more details. It looks like it's
simply playing a midi file while the system is under load but less clear
on what the symptoms are (audio skipping maybe?). I'll start with using
irqs-off tracer to see can I replicate a similar style of issue without
depending on sound.
Clemens, would this work to identify the problem without relying on a
device such as a sound card with a wavetable synthesiser or external
synthesiser receiving MIDI signals from the pc?
kswapd0 sharing equal top place in CPU usage at times (e.g. 20 percent).
If I should try only one of the patches or something else entirely,
please let me know.
For Mel: with z1, kswapd used only 0.1-3.9 percent of CPU while he
loaded other applications.
What's the usage otherwise?
As that patch has been NAKd by Rik on the grounds it eliminates the "balance
gap" entirely, can it be checked if the patch below keeps the CPU usage low
as well please?
Unfortunately, when I loaded KDE 3.5.X, konversation, aptitude -u, and
icedove with Mel's patch in the last few minutes, kswapd0 ran up to 17.5
percent CPU usage and the playback using aplaymidi slowed down. Load was
around 6 and swap usage had barely started when I noticed the slowdown
in MIDI playback.
By contrast, with the "z1" patch I could load the pc as above and also
starting iceweasel with a few dozen tabs open, and play the MIDI file by
using the QT application Rosegarden instead of aplaymidi, and not have
any slowdown, even though the swap usage was up around 200 MB and load
was over 10.0.
We may need a way to put kswapd in all uncompactable mode to solve
this, logic 3 just trying not to disable the all unreclaimable logic
seems not enough. I.e. if compact_zone_order doesn't return
COMPACT_COMPLETE, stop the compaction loop in kswapd. Then we can put
back in the COMPACT_MODE_KSWAPD return COMPACT_CONTINUE in
compact_finished as the caller will throttle it (and it won't run more
than one scan before putting kswapd to sleep in all uncompactable mode).
Should be possible. I'm not going to develop such a patch now though and
instead have a stab at replicating some of the exhibited problems (high
kswapd CPU usage, long interrupt disabled times etc).
Can the following patch be tested as a potential replacement for z1
please?
==== CUT HERE ====
vmscan: kswapd should not free an excessive number of pages when balancing small zones
When reclaiming for order-0 pages, kswapd requires that all zones be
balanced. Each cycle through balance_pgdat() does background ageing on all
zones if necessary and applies equal pressure on the inactive zone unless
a lot of pages are free already.
A "lot of free pages" is defined as a "balance gap" above the high watermark
which is currently 7*high_watermark. Historically this was reasonable as
min_free_kbytes was small. However, on systems using huge pages, it is
recommended that min_free_kbytes is higher and it is tuned with hugeadm
--set-recommended-min_free_kbytes. With the introduction of transparent
huge page support, this recommended value is also applied. On X86-64 with
4G of memory, min_free_kbytes becomes 67584 so one would expect around 68M
of memory to be free. The Normal zone is approximately 35000 pages so under
even normal memory pressure such as copying a large file, it gets exhausted
quickly. As it is getting exhausted, kswapd applies pressure equally to all
zones, including the DMA32 zone. DMA32 is approximately 700,000 pages with
a high watermark of around 23,000 pages. In this situation, kswapd will
reclaim around (23000*8 where 8 is the high watermark + balance gap of 7 *
high watermark) pages or 718M of pages before the zone is ignored. What
the user sees is that free memory far higher than it should be.
To avoid an excessive number of pages being reclaimed from the larger zones,
explicitely defines the "balance gap" to be either 1% of the zone or the
low watermark for the zone, whichever is smaller. While kswapd will check
all zones to apply pressure, it'll ignore zones that meets the (high_wmark +
balance_gap) watermark.
To test this, 80G were copied from a paritition and the amount of memory
being used was recorded. A comparison of a patch and unpatched kernel
can be seen at
http://www.csn.ul.ie/~mel/postings/minfree-20110222/memory-usage-hydra.ps
and shows that kswapd is not reclaiming as much memory with the patch
applied.
Signed-off-by: Andrea Arcangeli<aarcange@xxxxxxxxxx>
Signed-off-by: Mel Gorman<mel@xxxxxxxxx>
---
include/linux/swap.h | 9 +++++++++
mm/vmscan.c | 16 +++++++++++++---
2 files changed, 22 insertions(+), 3 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 4d55932..a57c6e7 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -155,6 +155,15 @@ enum {
#define SWAP_CLUSTER_MAX 32
#define COMPACT_CLUSTER_MAX SWAP_CLUSTER_MAX
+/*
+ * Ratio between the present memory in the zone and the "gap" that
+ * we're allowing kswapd to shrink in addition to the per-zone high
+ * wmark, even for zones that already have the high wmark satisfied,
+ * in order to provide better per-zone lru behavior. We are ok to
+ * spend not more than 1% of the memory for this zone balancing "gap".
+ */
+#define KSWAPD_ZONE_BALANCE_GAP_RATIO 100
+
#define SWAP_MAP_MAX 0x3e /* Max duplication count, in first swap_map */
#define SWAP_MAP_BAD 0x3f /* Note pageblock is bad, in first swap_map */
#define SWAP_HAS_CACHE 0x40 /* Flag page is cached, in first swap_map */
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 17497d0..0c83530 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2388,6 +2388,7 @@ loop_again:
int compaction;
struct zone *zone = pgdat->node_zones + i;
int nr_slab;
+ unsigned long balance_gap;
if (!populated_zone(zone))
continue;
@@ -2404,11 +2405,20 @@ loop_again:
mem_cgroup_soft_limit_reclaim(zone, order, sc.gfp_mask);
/*
- * We put equal pressure on every zone, unless one
- * zone has way too many pages free already.
+ * We put equal pressure on every zone, unless
+ * one zone has way too many pages free
+ * already. The "too many pages" is defined
+ * as the high wmark plus a "gap" where the
+ * gap is either the low watermark or 1%
+ * of the zone, whichever is smaller.
*/
+ balance_gap = min(low_wmark_pages(zone),
+ (zone->present_pages +
+ KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
+ KSWAPD_ZONE_BALANCE_GAP_RATIO);
if (!zone_watermark_ok_safe(zone, order,
- 8*high_wmark_pages(zone), end_zone, 0))
+ high_wmark_pages(zone) + balance_gap,
+ end_zone, 0))
shrink_zone(priority, zone,&sc);
reclaim_state->reclaimed_slab = 0;
nr_slab = shrink_slab(sc.nr_scanned, GFP_KERNEL,
Regards,
Arthur.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/