Re: [PATCH v3 09/14] vfs: add O_CREAT|O_DIRECTORY to open*(2)

[Pedro Falcato]

Hi,

Sorry for comenting so late...

On Sat, Jul 04, 2026 at 06:41:43PM +0200, Jori Koolstra wrote:
> Currently there is no way to race-freely create and open a directory.
> For regular files we have open(O_CREAT) for creating a new file inode,
> and returning a pinning fd to it. The lack of such functionality for
> directories means that when populating a directory tree there's always
> a race involved: the inodes first need to be created, and then opened
> to adjust their permissions/ownership/labels/timestamps/acls/xattrs/...,
> but in the time window between the creation and the opening they might
> be replaced by something else.
> 
> Addressing this race without proper APIs is possible (by immediately
> fstat()ing what was opened, to verify that it has the right inode type),
> but difficult to get right. Hence, adding support for a new flag combo
> O_CREAT|O_DIRECTORY to open*(2) that creates a directory (if it does not
> exist already) and returns an O_DIRECTORY fd is very useful.
> 
> Historically, the O_CREAT|O_DIRECTORY behaviour was to return ENOTDIR if
> a regular file exists at the open path; EISDIR if a directory exists at
> the path; and to create a regular file if no file exists at the path.
> This behaviour changed accidentally with 973d4b73fbaf ("do_last(): rejoin
> the common path even earlier in FMODE_{OPENED,CREATED} case") causing
> ENOTDIR to return in the last case while still creating the file. As
> this change was not detected for a long time, Brauner proposed to adopt
> the more consistent NetBSD behaviour, i.e. to return EINVAL on the the
> O_CREAT|O_DIRECTORY combination. This change was applied in 43b450632676
> ("open: return EINVAL for O_DIRECTORY | O_CREAT") in March, 2023. As
> the EINVAL behaviour has been in the kernel for about 3 year now, no
> rollback is expected as a result of userspace reliance on old
> behaviour, leaving us free to reassign the O_CREAT|O_DIRECTORY semantics.

Ok so I genuinely don't see how users are supposed to use O_CREAT | O_DIRECTORY.
Lets think about the portability here. This is undefined by POSIX (meaning you
cannot rely on a O_CREAT | O_DIRECTORY having particular semantics). FreeBSD
and NetBSD each have different semantics (my investigation stopped there
~3 years ago; I vaguely remember that macOS/Darwin does not/did not support
O_DIRECTORY. I don't remember what's the deal with OpenBSD/Solaris/Illumos
here).

So, essentially you can't rely on _any_ common behavior between platforms.
Ok. Now, you may say "it's 2026 lol who cares about BSD". So lets look at Linux:
- Linux for most of its lifetime had "return open directory, or create
  a regular file".
- It later had the above, with the added caveat that it always returned
  ENOTDIR.
- In 2023 I then found the bug and Christian changed it to always return
  EINVAL on that combination.

Ok. So Linux had 3 behaviors and 2 of them do not make any kind of sense. But
that's fine, we return EINVAL now. Except that the other behaviors existed.

So, lets say this feature lands as-is. If you're writing something *for Linux*,
you have to contend with the previous *3 behaviors* for a grand total of 4
behaviors. Including bizarre runtime tests, lest you are running on something
pre-6.4 (system calls can be similarly problematic, but at least there its a
simple ENOSYS test).

If you're writing something that wants to be portable, or that intends to
standardize on something, you have some 5 different behaviors all across
the FOSS UNIX landscape, not considering everything else. It's also something
that will, FWIW, probably never be included in POSIX because no one can agree
on the semantics here.

TL;DR I don't see, given the reasons above, how users are supposed to use
this without it being a total minefield.

If you're intent on the open() path, I would suggest that perhaps openat2()
would be a nice alternative, although definitely not as beautiful as
open(O_CREAT | O_DIRECTORY).

Just my 2c. And I feel horrible typing this out because you clearly spent
a great amount of time and effort between v2 and v3 as you redid this :/

-- 
Pedro