Re: [PATCH 06/17] arm: mmu_gather rework

From: Russell King
Date: Mon Feb 28 2011 - 07:01:16 EST

On Mon, Feb 28, 2011 at 12:44:47PM +0100, Peter Zijlstra wrote:
> Right, so the normal case is:
> unmap_region()
> tlb_gather_mmu()

The fullmm argument is important here as it specifies the mode.
tlb_gather_mmu(, 0)

> unmap_vmas()
> for (; vma; vma = vma->vm_next)
> unmao_page_range()
> tlb_start_vma() -> flush cache range
> zap_*_range()
> ptep_get_and_clear_full() -> batch/track external tlbs
> tlb_remove_tlb_entry() -> batch/track external tlbs
> tlb_remove_page() -> track range/batch page
> tlb_end_vma() -> flush tlb range

tlb_finish_mmu() -> nothing

> [ for architectures that have hardware page table walkers
> concurrent faults can still load the page tables ]
> free_pgtables()

tlb_gather_mmu(, 1)

> while (vma)
> unlink_*_vma()
> free_*_range()
> *_free_tlb()
> tlb_finish_mmu()

tlb_finish_mmu() -> flush tlb mm

> free vmas

So this is all fine. Note that we *don't* use the range stuff here.

> Now, if we want to track ranges _and_ have hardware page table walkers
> (ARM seems to be one such), we must flush TLBs at tlb_end_vma() because
> flush_tlb_range() requires a vma pointer (ARM and TILE actually use more
> than ->vm_mm), and on tlb_finish_mmu() issue a full mm wide invalidate
> because the hardware walker could have re-populated the cache after
> clearing the PTEs but before freeing the page tables.

No. The hardware walker won't re-populate the TLB after the page table
entries have been cleared - where would it get this information from if
not from the page tables?

> What ARM does is it retains the last vma pointer and tracks
> pte_free_tlb() range and uses that in tlb_finish_mmu(), which is a tad
> hacky.

It may be hacky but then the TLB shootdown interface is hacky too. We
don't keep the vma around to re-use after tlb_end_vma() - if you think
that then you misunderstand what's going on. The vma pointer is kept
around as a cheap way of allowing tlb_finish_mmu() to distinguish
between the unmap_region() mode and the shift_arg_pages() mode.

> Mostly because of shift_arg_pages(), where we have:
> shift_arg_pages()
> tlb_gather_mmu()

tlb_gather_mmu(, 0)

> free_*_range()
> tlb_finish_mmu()

tlb_finish_mmu() does nothing without the ARM change.
tlb_finish_mmu() -> flush_tlb_mm() with the ARM change.

And this is where the bug was - these page table entries could find
their way into the TLB and persist after they've been torn down.

> For which ARM now punts and does a full tlb invalidate (no vma pointer).
> But also because it drags along that vma pointer, which might not at all
> match the range its actually going to invalidate (and hence its vm_flags
> might not accurately reflect things -- at worst more expensive than
> needed).

Where do you get that from? Where exactly in the above code would the
VMA pointer get set? In this case, it will be NULL, so we do a
flush_tlb_mm() for this case. We have to - we don't have any VMA to
deal with at this point.

> The reason I wanted flush_tlb_range() to take an mm_struct and not the
> current vm_area_struct is because we can avoid doing the
> flush_tlb_range() from tlb_end_vma() and delay the thing until
> tlb_finish_mmu() without having to resort to such games as above. We
> could simply track the full range over all VMAs and free'd page-tables
> and do one range invalidate.

No. That's stupid. Consider the case where you have to loop one page
at a time over the range (we do on ARM.) If we ended up with your
suggestion above, that means we could potentially have to loop 4K at a
time over 3GB of address space. That's idiotic when we have an
instruction which can flush the entire TLB for a particular thread.

Russell King
Linux kernel 2.6 ARM Linux -
maintainer of:
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at