Re: [PATCH, RFC 00/16] Transparent huge page cache

[Kirill A. Shutemov]

Hugh Dickins wrote:
> On Tue, 29 Jan 2013, Kirill A. Shutemov wrote:
> > Hugh Dickins wrote:
> > > 
> > > Interesting.
> > > 
> > > I was starting to think about Transparent Huge Pagecache a few
> > > months ago, but then got washed away by incoming waves as usual.
> > > 
> > > Certainly I don't have a line of code to show for it; but my first
> > > impression of your patches is that we have very different ideas of
> > > where to start.
> 
> A second impression confirms that we have very different ideas of
> where to start.  I don't want to be dismissive, and please don't let
> me discourage you, but I just don't find what you have very interesting.

The main reason for publishing the patchset in current
(not-really-useful) state is to start discussion early.
Looks like it works :)

> I'm sure you'll agree that the interesting part, and the difficult part,
> comes with mmap(); and there's no point whatever to THPages without mmap()
> (of course, I'm including exec and brk and shm when I say mmap there).
> 
> (There may be performance benefits in working with larger page cache
> size, which Christoph Lameter explored a few years back, but that's a
> different topic: I think 2MB - if I may be x86_64-centric - would not be
> the unit of choice for that, unless SSD erase block were to dominate.)
> 
> I'm interested to get to the point of prototyping something that does
> support mmap() of THPageCache: I'm pretty sure that I'd then soon learn
> a lot about my misconceptions, and have to rework for a while (or give
> up!); but I don't see much point in posting anything without that.
> I don't know if we have 5 or 50 places which "know" that a THPage
> must be Anon: some I'll spot in advance, some I sadly won't.
> 
> It's not clear to me that the infrastructural changes you make in this
> series will be needed or not, if I pursue my approach: some perhaps as
> optimizations on top of the poorly performing base that may emerge from
> going about it my way.  But for me it's too soon to think about those.
> 
> Something I notice that we do agree upon: the radix_tree holding the
> 4k subpages, at least for now.  When I first started thinking towards
> THPageCache, I was fascinated by how we could manage the hugepages in
> the radix_tree, cutting out unnecessary levels etc; but after a while
> I realized that although there's probably nice scope for cleverness
> there (significantly constrained by RCU expectations), it would only
> be about optimization.

One more point: you have still preserve memory for these levels anyway,
since we must have never-fail split_huge_page().

> Let's be simple and stupid about radix_tree
> for now, the problems that need to be worked out lie elsewhere.
> 
> > > 
> > > Perhaps that's good complementarity, or perhaps I'll disagree with
> > > your approach.  I'll be taking a look at yours in the coming days,
> > > and trying to summon back up my own ideas to summarize them for you.
> > 
> > Yeah, it would be nice to see alternative design ideas. Looking forward.
> > 
> > > Perhaps I was naive to imagine it, but I did intend to start out
> > > generically, independent of filesystem; but content to narrow down
> > > on tmpfs alone where it gets hard to support the others (writeback
> > > springs to mind).  khugepaged would be migrating little pages into
> > > huge pages, where it saw that the mmaps of the file would benefit
> > > (and for testing I would hack mmap alignment choice to favour it).
> > 
> > I don't think all fs at once would fly, but it's wonderful, if I'm
> > wrong :)
> 
> You are imagining the filesystem putting huge pages into its cache.
> Whereas I'm imagining khugepaged looking around at mmaped file areas,
> seeing which would benefit from huge pagecache (let's assume offset 0
> belongs on hugepage boundary - maybe one day someone will want to tune
> some files or parts differently, but that's low priority), migrating 4k
> pages over to 2MB page (wouldn't have to be done all in one pass), then
> finally slotting in the pmds for that.

I had file huge page consolidation on todo list, but much later. I feel
that our approaches are complimentary.

> But going this way, I expect we'd have to split at page_mkwrite():
> we probably don't want a single touch to dirty 2MB at a time,
> unless tmpfs or ramfs.

Hm.. Splitting is rather expensive. I think it makes sense for fs with
backing device to consolidate only pages which mapped without PROT_WRITE.
This way we can avoid consolidate-split loops.

> > > I had arrived at a conviction that the first thing to change was
> > > the way that tail pages of a THP are refcounted, that it had been a
> > > mistake to use the compound page method of holding the THP together.
> > > But I'll have to enter a trance now to recall the arguments ;)
> > 
> > THP refcounting looks reasonable for me, if take split_huge_page() in
> > account.
> 
> I'm not claiming that the THP refcounting is wrong in what it's doing
> at present; but that I suspect we'll want to rework it for THPageCache.
> 
> Something I take for granted, I think you do too but I'm not certain:
> a file with transparent huge pages in its page cache can also have small
> pages in other extents of its page cache; and can be mapped hugely (2MB
> extents) into one address space at the same time as individual 4k pages
> from those extents are mapped into another (or the same) address space.
> 
> One can certainly imagine sacrificing that principle, splitting whenever
> there's such a "conflict"; but it then becomes uninteresting to me, too
> much like hugetlbfs.  Splitting an anonymous hugepage in all address
> spaces that hold it when one of them needs it split, that has been a
> pragmatic strategy: it's not a common case for forks to diverge like
> that; but files are expected to be more widely shared.
> 
> At present THP is using compound pages, with mapcount of tail pages
> reused to track their contribution to head page count; but I think we
> shall want to be able to use the mapcount, and the count, of TH tail
> pages for their original purpose if huge mappings can coexist with tiny.
> Not fully thought out, but that's my feeling.
> 
> The use of compound pages, in particular the redirection of tail page
> count to head page count, was important in hugetlbfs: a get_user_pages
> reference on a subpage must prevent the containing hugepage from being
> freed, because hugetlbfs has its own separate pool of hugepages to
> which freeing returns them.
> 
> But for transparent huge pages?  It should not matter so much if the
> subpages are freed independently.  So I'd like to devise another glue
> to hold them together more loosely (for prototyping I can certainly
> pretend we have infinite pageflag and pagefield space if that helps):
> I may find in practice that they're forever falling apart, and I run
> crying back to compound pages; but at present I'm hoping not.

Looks interesting. But I'm not sure whether it will work. It would be nice
to summon Andrea to the thread.
 
> This mail might suggest that I'm about to start coding: I wish that
> were true, but in reality there's always a lot of unrelated things
> I have to look at, which dilute my focus.  So if I've said anything
> that sparks ideas for you, go with them.

I want get my current approach work first. Will see.

-- 
 Kirill A. Shutemov
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/