Re: [PATCH] Repeated fork() causes SLAB to grow without bound

[Konstantin Khlebnikov]

On Thu, Nov 20, 2014 at 6:03 PM, Konstantin Khlebnikov <koct9i@xxxxxxxxx> wrote:
> On Thu, Nov 20, 2014 at 5:50 PM, Rik van Riel <riel@xxxxxxxxxx> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 11/20/2014 09:42 AM, Konstantin Khlebnikov wrote:
>>
>>> I'm thinking about limitation for reusing anon_vmas which might
>>> increase performance without breaking asymptotic estimation of
>>> count anon_vma in the worst case. For example this heuristic: allow
>>> to reuse only anon_vma with single direct descendant. It seems
>>> there will be arount up to two times more anon_vmas but
>>> false-aliasing must be much lower.

Done. RFC patch in attachment.

This patch adds heuristic which decides to reuse existing anon_vma instead
of forking new one. It counts vmas and direct descendants for each anon_vma.
Anon_vma with degree lower than two will be reused at next fork.
As a result each anon_vma has either alive vma or at least two descendants,
endless chains are no longer possible and count of anon_vmas is no more than
two times more than count of vmas.


>>
>> It may even be possible to not create a child anon_vma for the
>> first child a parent forks, but only create a new anon_vma once
>> the parent clones a second child (alive at the same time as the
>> first child).
>>
>> That still takes care of things like apache or sendmail, but
>> would not create infinite anon_vmas for a task that keeps forking
>> itself to infinite depth without calling exec...
>
> But this scheme is still exploitable. Malicious software easily could create
> sequence of forks and exits which leads to infinite chain of anon_vmas.
>
>>
>> - --
>> All rights reversed
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1
>>
>> iQEcBAEBAgAGBQJUbf+hAAoJEM553pKExN6DxhQH/1QL+9GdhaSx7EQnRcbDRcHi
>> GuEfMU0g9Kv4ad+oPSQnH/L7vJMJAYeh5ZJGH+rOykWHp3sGReqDZOnzpXRAe11z
>> 1cSC1BJsndzrv9wX8niFpuKpYbF0IP+ckv3qaEzWtm5yCRyhHVZfr6b794Y4K9jF
>> z2EPPu1vAAldbkx1VlYTwofBA5lESL5UmrFvH4ouI7BeWYSEe6BgVCbvK+K5fANT
>> ketdA5R08xyUAcXDa+28qpBYkdWnxNhwqseDoXCW8SOFNwWbLDI6GRfrsCNku13i
>> Gi41h3uEuIAGDf+AU/GMjiymgwutCOGq+cfZlszELaRvHmDpNGYdPv1llghNg7Q=
>> =Vk+H
>> -----END PGP SIGNATURE-----

Attachment: mm-prevent-endless-growth-of-anon_vma-hierarchy
Description: Binary data