Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Sat, Jan 19, 2019 at 03:44:48PM -0800, hpa@xxxxxxxxx wrote:
> On January 19, 2019 3:25:03 PM PST, Joel Fernandes <joel@xxxxxxxxxxxxxxxxx> wrote:
> >On Sat, Jan 19, 2019 at 12:43:35PM -0500, Daniel Colascione wrote:
> >> On Sat, Jan 19, 2019 at 11:27 AM Joel Fernandes
> ><joel@xxxxxxxxxxxxxxxxx> wrote:
> >> >
> >> > On Sat, Jan 19, 2019 at 09:25:32AM +0100, Greg KH wrote:
> >> > > On Fri, Jan 18, 2019 at 05:55:43PM -0500, Joel Fernandes wrote:
> >> > > > --- /dev/null
> >> > > > +++ b/kernel/kheaders.c
> >> 
> >> Thanks a ton for this work. It'll make it much easier to do cool
> >> things with BPF.
> >
> >You're welcome, thanks.
> >
> >> One question: I can imagine wanting to probe
> >> structures that are defined, not in headers, but in random
> >> implementation files. Would it be possible to optionally include
> >*all*
> >> kernel source files?
> >
> >That would be prohibitively too large to justify keeping it in memory,
> >even
> >compressed. Arguably, such structures should be moved into include/ if
> >modules or whatever is extending the kernel like eBPF needs them.
> >
> >> If not, what about a hash, so we could at least
> >> do precise correlation between a candidate local tree and what's
> >> actually on device?
> >
> >That would make a tool too difficult to write wouldn't it, since they
> >you have to
> >correlate every possible hash and keep updating eBPF tools with new
> >hashes -
> >probably not scalable. I think what you want is to use the kernel
> >version to
> >assume what such internal structures look like although that's still
> >not
> >robust.
> >
> >> BTW, I'm not sure that the magic constants you've defined are long
> >> enough.  I'd feel more comfortable with two UUIDs (16 bytes each).
> >
> >Ok, I'll expand it.
> >
> >> I'd also strongly consider LZMA compression: xz -9 on the kernel
> >> headers (with comments) brings the size down to 5MB, compared to the
> >> 7MB I get for gzip -9. Considering that this feature is optional, I
> >> think it's okay to introduce a dependency on widespread modern
> >> compression tools. (For comparison, bzip2 -9 gets us 6MB.)
> >
> >Ok, I'll look into LZMA. Thanks for checking the compression sizes.
> >
> >- Joel
> 
> Don't use lzma, use xz if you are going to do something.

Ok, sounds good.

> However, it seems unlikely to me that someone not willing to spend the space in the filesystem will spend unswappable kernel memory.
> 
> It would seem that a far saner way to do this is to use inittmpfs or perhaps an auxiliary "ktmpfs" so it can at least be swapped out if you have swap.

But this is already possible with the proposed solution, you would load the
module, extract it into a tmpfs, and unload the module. TMPFS pages can
already be swapped.

thanks,

 - Joel