Re: [PATCH v3] binfmt_elf: Dump smaller VMAs first in ELF cores

[Linus Torvalds]

On Wed, 19 Feb 2025 at 11:52, Kees Cook <kees@xxxxxxxxxx> wrote:
>
> Yeah, I think we need to make this a tunable. Updating the kernel breaks
> elftools, which isn't some weird custom corner case. :P

I wonder if we could also make the default be "no sorting" if the
vma's are all fairly small...

IOW, only trigger the new behavior when nity actually *matters*.

We already have the code to count how big the core dump is, it's that

                cprm->vma_data_size += m->dump_size;

in dump_vma_snapshot() thing, so I think this could all basically be a
one-liner that does the sort() call only if that vma_data_size is
larger than the core-dump limit, or something like that?

That way, the normal case could basically work for everybody, and the
system tunable would be only for people who want to force a certain
situation.

Something trivial like this (ENTIRELY UNTESTED) patch, perhaps:

  --- a/fs/coredump.c
  +++ b/fs/coredump.c
  @@ -1256,6 +1256,10 @@ static bool dump_vma_snapshot(struct
coredump_params *cprm)
                  cprm->vma_data_size += m->dump_size;
          }

  +       /* Only sort the vmas by size if they don't all fit in the
core dump */
  +       if (cprm->vma_data_size < cprm->limit)
  +               return true;
  +
          sort(cprm->vma_meta, cprm->vma_count, sizeof(*cprm->vma_meta),
                  cmp_vma_size, NULL);

Hmm?

             Linus