Re: [RFC PATCH 00/16] 1GB THP support on x86_64

[Jason Gunthorpe]

On Thu, Sep 03, 2020 at 09:25:27AM -0700, Roman Gushchin wrote:
> On Thu, Sep 03, 2020 at 09:32:54AM +0200, Michal Hocko wrote:
> > On Wed 02-09-20 14:06:12, Zi Yan wrote:
> > > From: Zi Yan <ziy@xxxxxxxxxx>
> > > 
> > > Hi all,
> > > 
> > > This patchset adds support for 1GB THP on x86_64. It is on top of
> > > v5.9-rc2-mmots-2020-08-25-21-13.
> > > 
> > > 1GB THP is more flexible for reducing translation overhead and increasing the
> > > performance of applications with large memory footprint without application
> > > changes compared to hugetlb.
> > 
> > Please be more specific about usecases. This better have some strong
> > ones because THP code is complex enough already to add on top solely
> > based on a generic TLB pressure easing.
> 
> Hello, Michal!
> 
> We at Facebook are using 1 GB hugetlbfs pages and are getting noticeable
> performance wins on some workloads.

At least from a RDMA NIC perspective I've heard from a lot of users
that higher order pages at the DMA level is giving big speed ups too.

It is basically the same dynamic as CPU TLB, except missing a 'TLB'
cache in a PCI-E device is dramatically more expensive to refill. With
200G and soon 400G networking these misses are a growing problem.

With HPC nodes now pushing 1TB of actual physical RAM and single
applications basically using all of it, there is definately some
meaningful return - if pages can be reliably available.

At least for HPC where the node returns to an idle state after each
job and most of the 1TB memory becomes freed up again, it seems more
believable to me that a large cache of 1G pages could be available?

Even triggering some kind of cleaner between jobs to defragment could
be a reasonable approach..

Jason