[PATCH v3] coredump: exit_files() in coredump_wait() if MMF_DUMP_MAPPED_SHARED is not set

[Xin Zhao]

A coredump typically takes some time to complete. If we happen to hold a
write lock with flock just before triggering the coredump, that write lock
will not be released during the entire coredump process. As a result,
other processes attempting to acquire the same write lock may experience
significant delays.

To address this, call exit_files() in the end of coredump_wait(), if
MMF_DUMP_MAPPED_SHARED is not set. Note that early unlocking a flock on a
file allows other processes to lock and modify the mapped data protected
by the flock.

Signed-off-by: Xin Zhao <jackzxcui1989@xxxxxxx>
---

Change in v3:
- Add comment and commit-log to explain why do the MMF_DUMP_MAPPED_SHARED
  mm_flags_test() check, note that memory mapped files keep their own
  separate references to the files. The case to work around is that early
  unlocking a flock on a file allows other processes to lock and modify
  the mapped data protected by the flock,
  as suggested by Pedro Falcato.

Change in v2:
- Get rid of the implement of adding new fcntl API, the issue does not
  worth inflicting the cost on everyone,
  as suggested by Al Viro.
- Call exit_files() in coredump_wait(),
  as suggested by Eric W. Biederman.
  Add MMF_DUMP_MAPPED_SHARED mm_flags_test() check to filter cases that
  need to dump file-backed shared memory.
- Link to v2: https://lore.kernel.org/lkml/20260618150301.3226517-1-jackzxcui1989@xxxxxxx/

v1:
- Link to v1: https://lore.kernel.org/all/20260618030700.2511668-1-jackzxcui1989@xxxxxxx/
---
 fs/coredump.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/fs/coredump.c b/fs/coredump.c
index bb6fdb1f4..70698d06e 100644
--- a/fs/coredump.c
+++ b/fs/coredump.c
@@ -548,6 +548,13 @@ static int coredump_wait(int exit_code, struct core_state *core_state)
 		}
 	}
 
+	/*
+	 * Early unlocking a flock on a file allows other processes
+	 * to lock and modify the mapped data protected by the flock.
+	 */
+	if (!mm_flags_test(MMF_DUMP_MAPPED_SHARED, tsk->mm))
+		exit_files(tsk);
+
 	return core_waiters;
 }
 
-- 
2.34.1