Re: [PATCH 4/6] mm: hugetlb_vmemmap: add missing smp_wmb() before set_pte_at()

[Muchun Song]

> On Aug 18, 2022, at 20:58, Miaohe Lin <linmiaohe@xxxxxxxxxx> wrote:
> 
> On 2022/8/18 17:18, Muchun Song wrote:
>> 
>> 
>>> On Aug 18, 2022, at 16:54, Yin, Fengwei <fengwei.yin@xxxxxxxxx> wrote:
>>> 
>>> 
>>> 
>>> On 8/18/2022 4:40 PM, Muchun Song wrote:
>>>> 
>>>> 
>>>>> On Aug 18, 2022, at 16:32, Yin, Fengwei <fengwei.yin@xxxxxxxxx> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> On 8/18/2022 3:59 PM, Muchun Song wrote:
>>>>>> 
>>>>>> 
>>>>>>> On Aug 18, 2022, at 15:52, Miaohe Lin <linmiaohe@xxxxxxxxxx> wrote:
>>>>>>> 
>>>>>>> On 2022/8/18 10:47, Muchun Song wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Aug 18, 2022, at 10:00, Yin, Fengwei <fengwei.yin@xxxxxxxxx> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 8/18/2022 9:55 AM, Miaohe Lin wrote:
>>>>>>>>>>>>> 	/*
>>>>>>>>>>>>> 	 * The memory barrier inside __SetPageUptodate makes sure that
>>>>>>>>>>>>> 	 * preceding stores to the page contents become visible before
>>>>>>>>>>>>> 	 * the set_pte_at() write.
>>>>>>>>>>>>> 	 */
>>>>>>>>>>>>> 	__SetPageUptodate(page);
>>>>>>>>>>>> IIUC, the case here we should make sure others (CPUs) can see new page’s
>>>>>>>>>>>> contents after they have saw PG_uptodate is set. I think commit 0ed361dec369
>>>>>>>>>>>> can tell us more details.
>>>>>>>>>>>> 
>>>>>>>>>>>> I also looked at commit 52f37629fd3c to see why we need a barrier before
>>>>>>>>>>>> set_pte_at(), but I didn’t find any info to explain why. I guess we want
>>>>>>>>>>>> to make sure the order between the page’s contents and subsequent memory
>>>>>>>>>>>> accesses using the corresponding virtual address, do you agree with this?
>>>>>>>>>>> This is my understanding also. Thanks.
>>>>>>>>>> That's also my understanding. Thanks both.
>>>>>>>>> I have an unclear thing (not related with this patch directly): Who is response
>>>>>>>>> for the read barrier in the read side in this case?
>>>>>>>>> 
>>>>>>>>> For SetPageUptodate, there are paring write/read memory barrier.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> I have the same question. So I think the example proposed by Miaohe is a little
>>>>>>>> difference from the case (hugetlb_vmemmap) here.
>>>>>>> 
>>>>>>> Per my understanding, memory barrier in PageUptodate() is needed because user might access the
>>>>>>> page contents using page_address() (corresponding pagetable entry already exists) soon. But for
>>>>>>> the above proposed case, if user wants to access the page contents, the corresponding pagetable
>>>>>>> should be visible first or the page contents can't be accessed. So there should be a data dependency
>>>>>>> acting as memory barrier between pagetable entry is loaded and page contents is accessed.
>>>>>>> Or am I miss something?
>>>>>> 
>>>>>> Yep, it is a data dependency. The difference between hugetlb_vmemmap and PageUptodate() is that
>>>>>> the page table (a pointer to the mapped page frame) is loaded by MMU while PageUptodate() is
>>>>>> loaded by CPU. Seems like the data dependency should be inserted between the MMU access and the CPU
>>>>>> access. Maybe it is hardware’s guarantee?
>>>>> I just found the comment in pmd_install() explained why most arch has no read
>>>> 
>>>> I think pmd_install() is a little different as well. We should make sure the
>>>> page table walker (like GUP) see the correct PTE entry after they see the pmd
>>>> entry.
>>> 
>>> The difference I can see is that pmd/pte thing has both hardware page walker and
>>> software page walker (like GUP) as read side. While the case here only has hardware
>>> page walker as read side. But I suppose the memory barrier requirement still apply
>>> here.
>> 
>> I am not against this change. Just in order to make me get a better understanding of
>> hardware behavior.
>> 
>>> 
>>> Maybe we could do a test: add large delay between reset_struct_page() and set_pte_at?
>> 
>> Hi Miaohe,
>> 
>> Would you mind doing this test? One thread do vmemmap_restore_pte(), another thread
>> detect if it can see a tail page with PG_head after the previous thread has executed
>> set_pte_at().
> 
> Will it be easier to construct the memory reorder manually like below?
> 
> vmemmap_restore_pte()
> 	...
> 	set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot));
> 	/* might a delay. */
> 	copy_page(to, (void *)walk->reuse_addr);
> 	reset_struct_pages(to);


Well, you have changed the code ordering. I thought we don’t change the code
ordering. Just let the hardware do reordering. The ideal scenario would be
as follows.


CPU0:						CPU1:

vmemmap_restore_pte()
	copy_page(to, (void *)walk->reuse_addr);
        reset_struct_pages(to); // clear the tail page’s PG_head
	set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot));
						// Detect if it can see a tail page with PG_head.

I should admit it is a little difficult to construct the scenario. After more
thought, I think here should be inserted a barrier. So:

Reviewed-by: Muchun Song <songmuchun@xxxxxxxxxxxxx>

Thanks.

> 
> And another thread detects whether it can see a tail page with some invalid fields? If so,
> it seems the problem will always trigger? If not, we depend on the observed meory reorder
> and set_pte_at doesn't contain a memory barrier?
> 
> Thanks,
> Miaohe Lin