Re: LLM based rewrites

[Dr. Greg]

On Tue, Mar 10, 2026 at 08:47:21AM -0400, Theodore Tso wrote:

Good morning, I hope the week is going well for everyone.

> On Mon, Mar 09, 2026 at 10:15:28PM -0700, EJ Stinson wrote:
> > Imagine if a rouge AI got access to rewriting the kernel, or was
> > exploited, this would lead to near certain catastrophe. LLM's
> > should not rewrite the code, as if somehow a AI were to achieve
> > singularity or go rouge/be attacked by an anarchistic/foreign
> > actor, think about the amount of code it could sneak in without
> > human suspicion, or just lead to human ignorance. I think for the
> > time being until we know for certain, there should be no reason to
> > use LLM's to help rewrite at scale any sort of code. Even if we
> > were able to prove it wasn???t stolen code; the time spent on
> > proving such fact, and ensuring the security, would already take
> > way too long tomerit this sort of use.

> And the third is whether it would really result in more secure code
> (which was their premise for why some companies might do this, since
> the people giving the presentation at FOSDEM were security
> researchers).  Given that AI generated code is generally *more* likely
> to have security vulnerabilities than human written code, this
> assumption seems dubious to me.  Also if the security vulnerability is
> inherent in the software architecture, having the first LLM generate a
> spec might result in a *spec* which is buggy / vulnerable, and so when
> the second LLM translates that spec back into C code, not only might
> it introduce new security vulnerabiities, the original security
> vulnerability present in the source implementaiton might be preserved.

It would seem that if some enterprising individual or more likely a
major technology company, with sufficient resources, told an LLM to
simply convert the entire kernel to Rust, that would be the end of
kernel security vulnerabilities as we know it, not?

Then, if said enterprising individual or corporation slapped the GPL
on the result and pushed it to GitHub, mankind would be saved as we
know it.

In the spirit of Christian's intention to inspire conversation... :-)

> Cheers,
> 
> 						- Ted

Have a good remainder of the week.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project