Re: [PATCH v3 09/14] vfs: add O_CREAT|O_DIRECTORY to open*(2)

[Jori Koolstra]

> Op 07-07-2026 15:55 CEST schreef Pedro Falcato <pfalcato@xxxxxxx>:
> 
>  
> On Tue, Jul 07, 2026 at 12:06:44PM +0200, Jori Koolstra wrote:
> > Hi Pedro,
> > 
> > > Op 07-07-2026 10:35 CEST schreef Pedro Falcato <pfalcato@xxxxxxx>:
> > > 
> > > Ok so I genuinely don't see how users are supposed to use O_CREAT | O_DIRECTORY.
> > > Lets think about the portability here. This is undefined by POSIX (meaning you
> > > cannot rely on a O_CREAT | O_DIRECTORY having particular semantics). FreeBSD
> > > and NetBSD each have different semantics (my investigation stopped there
> > > ~3 years ago; I vaguely remember that macOS/Darwin does not/did not support
> > > O_DIRECTORY. I don't remember what's the deal with OpenBSD/Solaris/Illumos
> > > here).
> > > 
> > > So, essentially you can't rely on _any_ common behavior between platforms.
> > > Ok. Now, you may say "it's 2026 lol who cares about BSD". So lets look at Linux:
> > > - Linux for most of its lifetime had "return open directory, or create
> > >   a regular file".
> > > - It later had the above, with the added caveat that it always returned
> > >   ENOTDIR.
> > > - In 2023 I then found the bug and Christian changed it to always return
> > >   EINVAL on that combination.
> > > 
> > > Ok. So Linux had 3 behaviors and 2 of them do not make any kind of sense. But
> > > that's fine, we return EINVAL now. Except that the other behaviors existed.
> > > 
> > > So, lets say this feature lands as-is. If you're writing something *for Linux*,
> > > you have to contend with the previous *3 behaviors* for a grand total of 4
> > > behaviors. Including bizarre runtime tests, lest you are running on something
> > > pre-6.4 (system calls can be similarly problematic, but at least there its a
> > > simple ENOSYS test).
> > > 
> > 
> > But isn't this a problem with all UAPI design? You introduce a new feature that
> > does not exist on old kernels, then userspace can use that feature but must
> > provide fallback behaviors for any older kernel that the userspace program wants
> > to support. Ideally, this is abstracted away by some library. Then, at some point
> > this library raises its minimal supported kernel version bar, and it can clean-up
> > the now obsolete fallback code.
> 
> Correct, but usually most new UAPIs have very simple checks:
> 
> if (syscall(SYS_ponies) < 0 && errno == ENOSYS) {
> 	fallback();
> }
> 
> or
> 
> if (ponies(NEW_FLAG) < 0 && errno == EINVAL) {
> 	fallback();
> }
> 
> (in fact, many system calls out there can even be silently emulated by the
> libc if need be; this one can't, but that's besides the point)
> 

Forgive my ignorance, but why can't we use uname(2) or something to detect
kernel version and rely on that? Why must we check behavior as if it is a
black box? Is uname(2) blocked on systems for security reasons or something?

> My problem isn't with adding a new uapi, it's that the flag combination has
> so horrifically overloaded with weird semantics that it's pretty non-trivial
> to find out conclusively whether or not you have the feature, across the recent
> lifetime of Linux. I'm not even concerned with repurposing the flags because the
> old semantics were so obviously BS.
> 

How bad would it be to backport -EINVAL to the last stable kernels? I mean we know
people didn't really rely on the weird semantics prior to that because of your
find.

> 
> FWIW, thinking about this a bit more, it's possible that openat2() could serve
> as a stricter check for the UAPI in open() in general. Making this less
> footgunny to use.
> 

Not sure if I follow what you mean here.

> 
> -- 
> Pedro

Best,
Jori.