Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Wed, Mar 06, 2019 at 04:07:13PM -0800, H. Peter Anvin wrote:
> On 3/6/19 3:37 PM, Daniel Colascione wrote:
> > 
> > I just don't get the opposition to Joel's work. The rest of the thread
> > already goes into detail about the problems with pure-filesystem
> > solutions, and you and others are just totally ignoring those
> > well-thought-out rationales for the module approach and doing
> > inflooping on "lol just use a tarball". That's not productive.
> > 
> > Look; here's the bottom line: without this work, doing certain kinds
> > of system tracing is a nightmare, and with this patch, it Just Works.
> > You're arguing that various tools should do a better job of keeping
> > the filesystem in sync with the kernel. Maybe you're right. But we
> > don't live in a world where they will, because if this coherence were
> > going to happen, it'd work already. But this work solves the problem:
> > by necessity, anything that changes a kernel image *must* update
> > modules coherently, whether the kernel image and module come from the
> > filesystem, network boot, some kind of SQL database, or carrier
> > pigeon. There's nothing wrong with work that very cheaply makes the
> > kernel self-describing (introspection is elegant) and that takes
> > advantage of *existing* kernel tooling infrastructure to transparently
> > do a new thing.
> > 
> > You don't have to use this patch if you don't want to. Please stop
> > trying to block it.
> > 
> 
> No, that's not how this works. It is far worse to do something the wrong
> way than not doing it at all, when it affects the kernel-user space
> interactions.
> 
> Experience -- and we have almost 30 years of it -- has shown that hacks
> of this nature become engrained and all of a sudden is "mandatory". At
> the *very least* it needs to comply with the zero-one-infinity rule
> rather than being yet another ad hoc hack.
> 
> More fundamentally, we already have multiple ways to handle objects that
> need to go into the filesystem: they can be installed with (or as)
> modules, they can use the firmware interface, and so on.
> 
> Saying "it can be a module" is worse than a copout: even if dynamically
> loaded -- and many setups lock out post-boot module loadings for
> security reasons -- there is nothing to cause it to unload.
> 
> The bottom line is that in the end there is no difference between making
> this an archive of some sort and a module, except that to use the module
> you need privilege that you otherwise may not need. If your argument is
> that someone may not be providing the whole set of items provided by
> "make modules_install", what is there to say that they would include
> your specific module?

They would include the module because we can enforce that certain
config options in the kernel need to be enabled for Android features to work.
People can obviously mess up their kernel config in any number of ways and
misconfigure their kernel such that user space may even fail to boot. So how
is it relevant that people would include the module or not? If they want a
feature to work, their kernel has to enable the feature, right?

About disabling post-boot module loading, we don't do it on Android and
that's really the target system here. Can we not deal with such post-boot
module load disabling issue at a later time if such user really wants this?
If there is a CONFIG option that disables post-boot module loading, then we
can make this feature depend on that CONFIG being disabled.

> Here are some better ways of implementation I can see:
> 
> 1. Include an archive in "make modules_install". Most trivial; no kernel
>    changes at all.

As I said in previous posts, some people want to boot a kernel image without
any update to FS, for debug purposes. In the Android/embedded world,
developers want to build and boot a single binary blob (for debugging). For
such usecases, hacking modules_install would not help.

Once they are done debugging, they can switch the CONFIG option back to
module instead of building it in and consuming memory, so that it is not
loaded into memory until needed.

> 2. Generalize the initramfs code to be able to create a pre-populated
>    tmpfs at any time, that can be populated from an archive provided by
>    the firmware loading mechanism; like all firmware allows it to either
>    be built in or fetched from the filesystem. This allows it to be
>    built in to the kernel image if that becomes necessary; using tmpfs
>    means that it can be pushed out to swap rather than permanently
>    stored in kernel memory, and this filesystem can be unmounted freeing
>    its memory.

I see your points about how building it in can make it permanently lost
memory, but would that really be an issue if it is built-in by a user ONLY
for debugging?

Anyway, I'll try to think more about how I can make it memory that can be
unloaded/removed even if built-in into the kernel.

About making it a firmware built-in, and then populating tmpfs with the
firmware's contents, isn't that pointless if the firmware is built-in into
the kernel, and the memory that the firmware blob itself consumes is
permanently lost? Whether the tmpfs copy is swapped out or not, would not
help save the built-in firmware's memory.

> 3. Use a squashfs image instead of an archive.

This point number 3 sounds like a non-starter because it will make the kernel
build system fail if squashfs-tools is not available. So it is not an option
I think. I also don't see how it would solve the above swapping issue you
mentioned.  Whether it is an archive or squashfs image doesn't matter as much
as where the image lives.

thanks,

 - Joel