Re: arm64 + ARM64_64K_PAGES=y

[Andre Przywara]

On Fri, 9 Nov 2018 13:15:47 -0600
Grygorii Strashko <grygorii.strashko@xxxxxx> wrote:

Hi,

> On 11/8/18 12:14 PM, Grygorii Strashko wrote:
> > 
> > 
> > On 11/8/18 6:00 AM, Sebastian Andrzej Siewior wrote:  
> >> On 2018-11-06 15:34:55 [-0600], Grygorii Strashko wrote:  
> >>> Hi All,  
> >> Hi,
> >>  
> >>> Do anybody tried to use ARM64 RT with 76K pages enabled?  
> >>
> >> 75 would be an off by one but this :)  
> > 
> > Ops 8-). at least subj is correct.
> >   
> >>  
> >>> My attempt shows that enabling CONFIG_ARM64_64K_PAGES=y
> >>> increases latencies by ~30%

That's not really surprising. Performance on systems using a bigger page
size granules might have some trade-offs (bigger memory overhead, worse
cache utilization), so 64K pages might not be really great for your
particular workload. You would probably need a real performance
analysis (using perf, for instance) to pinpoint TLB misses as your
bottleneck.

> >>> cyclictest -n -m -Sp98 -q -D2m with  =y
> >>>
> >>>
> >>> T: 0 (Â 772) P:98 I:1000 C: 120000 Min:ÂÂÂÂÂ 7 Act:ÂÂ 13 Avg:
> >>> 10 Max:ÂÂÂÂÂ 85 T: 1 (Â 773) P:98 I:1500 C:Â 79998 Min:ÂÂÂÂÂ 7
> >>> Act:ÂÂ 13 Avg:ÂÂ 10 Max:ÂÂÂÂÂ 71 T: 2 (Â 774) P:98 I:2000 C:
> >>> 59997 Min:ÂÂÂÂÂ 7 Act:ÂÂ 11 Avg:ÂÂ 11 Max:ÂÂÂÂÂ 64 T: 3 (Â 775)
> >>> P:98 I:2500 C:Â 47996 Min:ÂÂÂÂÂ 7 Act:ÂÂ 14 Avg:ÂÂ 12 Max:ÂÂÂÂÂ 66
> >>>
> >>>
> >>> cyclictest -n -m -Sp98 -q -D2m with CONFIG_ARM64_64K_PAGES=n
> >>>
> >>>
> >>> T: 0 (Â 697) P:98 I:1000 C: 120000 Min:ÂÂÂÂÂ 7 Act:ÂÂ 10 Avg:
> >>> 9 Max:ÂÂÂÂÂ 38 T: 1 (Â 698) P:98 I:1500 C:Â 79987 Min:ÂÂÂÂÂ 7
> >>> Act:ÂÂ 10 Avg:ÂÂ 10 Max:ÂÂÂÂÂ 32 T: 2 (Â 699) P:98 I:2000 C:
> >>> 59981 Min:ÂÂÂÂÂ 7 Act:ÂÂ 14 Avg:ÂÂ 11 Max:ÂÂÂÂÂ 46 T: 3 (Â 700)
> >>> P:98 I:2500 C:Â 47977 Min:ÂÂÂÂÂ 6 Act:ÂÂ 11 Avg:ÂÂ 10 Max:
> >>> 45  
> >>
> >> So this is an idle system?  
> > 
> > Yes (in general) - it's collected with systemd, so some daemons are
> > active. 
> >> The Kconfig help says "faster TLB lookup". Interesting.
> >> Are the 16k pages in between (latency wise) by any chance?  
> > 
> > I'll try it.  
> 
> no i'll not, at least not fast. with 16k pages enabled I can't boot
> TI 4.14 kernel
> -  4.14.71-rt44.
> No msg in log, just "Starting kernel ..."

You need a core that actually supports 16K pages (supporting
certain page size granules is architecturally optional).
>From the Arm Ltd. cores it's Cortex-A73, A75 or A55, possibly other
newer ones as well. Cortex-A53, A57 and A72 do not support 16k pages.

Cheers,
Andre.