[GIT PULL] bcachefs updates for 6.8

From: Kent Overstreet
Date: Wed Jan 10 2024 - 14:36:58 EST

Hi Linus, here's the main bcachefs updates for 6.8.

Cheers,
Kent


The following changes since commit 0d72ab35a925d66b044cb62b709e53141c3f0143:

bcachefs: make RO snapshots actually RO (2024-01-01 11:47:07 -0500)

are available in the Git repository at:

https://evilpiepirate.org/git/bcachefs.git tags/bcachefs-2024-01-10

for you to fetch changes up to 169de41985f53320580f3d347534966ea83343ca:

bcachefs: eytzinger0_find() search should be const (2024-01-05 23:24:46 -0500)

----------------------------------------------------------------
bcachefs updates for 6.8:

- btree write buffer rewrite: instead of adding keys to the btree write
buffer at transaction commit time, we know journal them with a
different journal entry type and copy them from the journal to the
write buffer just prior to journal write.

This reduces the number of atomic operations on shared cachelines
in the transaction commit path and is a signicant performance
improvement on some workloads: multithreaded 4k random writes went
from ~650k iops to ~850k iops.

- Bring back optimistic spinning for six locks: the new implementation
doesn't use osq locks; instead we add to the lock waitlist as normal,
and then spin on the lock_acquired bit in the waitlist entry, _not_
the lock itself.

- BCH_IOCTL_DEV_USAGE_V2, which allows for new data types
- BCH_IOCTL_OFFLINE_FSCK, which runs the kernel implementation of fsck
but without mounting: useful for transparently using the kernel
version of fsck from 'bcachefs fsck' when the kernel version is a
better match for the on disk filesystem.

- BCH_IOCTL_ONLINE_FSCK: online fsck. Not all passes are supported yet,
but the passes that are supported are fully featured - errors may be
corrected as normal.

The new ioctls use the new 'thread_with_file' abstraction for kicking
off a kthread that's tied to a file descriptor returned to userspace
via the ioctl.

- btree_paths within a btree_trans are now dynamically growable,
instead of being limited to 64. This is important for the
check_directory_structure phase of fsck, and also fixes some issues
we were having with btree path overflow in the reflink btree.

- Trigger refactoring; prep work for the upcoming disk space accounting
rewrite

- Numerous bugfixes :)

----------------------------------------------------------------
Brian Foster (3):
bcachefs: remove sb lock and flags update on explicit shutdown
bcachefs: return from fsync on writeback error to avoid early shutdown
bcachefs: clean up some dead fallocate code

Daniel Hill (6):
bcachefs: add a quieter bch2_read_super
bcachefs: remove dead bch2_evacuate_bucket()
bcachefs: rebalance should wakeup on shutdown if disabled
bcachefs: copygc should wakeup on shutdown if disabled
bcachefs: copygc shouldn't try moving buckets on error
bcachefs: remove redundant condition from data_update_index_update

Gustavo A. R. Silva (3):
bcachefs: Replace zero-length arrays with flexible-array members
bcachefs: Use array_size() in call to copy_from_user()
bcachefs: Replace zero-length array with flex-array member and use __counted_by

Kent Overstreet (210):
bcachefs: Flush fsck errors before running twice
bcachefs: Add extra verbose logging for ro path
bcachefs: Improved backpointer messages in fsck
bcachefs: kill INODE_LOCK, use lock_two_nondirectories()
bcachefs: Check for unlinked inodes not on deleted list
bcachefs: Fix locking when checking freespace btree
bcachefs: Print old version when scanning for old metadata
bcachefs: Fix warning when building in userspace
bcachefs: Include average write size in sysfs journal_debug
bcachefs: Add an assertion in bch2_journal_pin_set()
bcachefs: Journal pins must always have a flush_fn
bcachefs: track_event_change()
bcachefs: Clear k->needs_whitout earlier in commit path
bcachefs: BTREE_INSERT_JOURNAL_REPLAY now "don't init trans->journal_res"
bcachefs: Kill BTREE_UPDATE_PREJOURNAL
bcachefs: Go rw before journal replay
bcachefs: Make journal replay more efficient
bcachefs: Avoiding dropping/retaking write locks in bch2_btree_write_buffer_flush_one()
bcachefs: Fix redundant variable initialization
bcachefs: Kill dead BTREE_INSERT flags
bcachefs: bch_str_hash_flags_t
bcachefs: Rename BTREE_INSERT flags
bcachefs: Improve btree_path_dowgrade tracepoint
bcachefs: backpointers fsck no longer uses BTREE_ITER_ALL_LEVELS
bcachefs: Kill BTREE_ITER_ALL_LEVELS
bcachefs: Fix userspace bch2_prt_datetime()
bcachefs: Don't rejournal keys in key cache flush
bcachefs: Don't flush journal after replay
bcachefs: Add a tracepoint for journal entry close
bcachefs: Kill memset() in bch2_btree_iter_init()
bcachefs: Kill btree_iter->journal_pos
bcachefs: Rename bch_replicas_entry -> bch_replicas_entry_v1
bcachefs: Don't use update_cached_sectors() in bch2_mark_alloc()
bcachefs: x-macro-ify bch_data_ops enum
bcachefs: Convert bch2_move_btree() to bbpos
bcachefs: BCH_DATA_OP_drop_extra_replicas
powerpc: Export kvm_guest static key, for bcachefs six locks
bcachefs: six locks: Simplify optimistic spinning
bcachefs: Simplify check_bucket_ref()
bcachefs: BCH_IOCTL_DEV_USAGE_V2
bcachefs: New bucket sector count helpers
bcachefs: bch2_dev_usage_to_text()
bcachefs: Kill dev_usage->buckets_ec
bcachefs: Improve sysfs compression_stats
bcachefs: Print durability in member_to_text()
bcachefs: Add a rebalance, data_update tracepoints
bcachefs: Refactor bch2_check_alloc_to_lru_ref()
bcachefs: Kill journal_seq/gc args to bch2_dev_usage_update_m()
bcachefs: convert bch_fs_flags to x-macro
bcachefs: No need to allocate keys for write buffer
bcachefs: Improve btree write buffer tracepoints
bcachefs: kill journal->preres_wait
bcachefs: delete useless commit_do()
bcachefs: Clean up btree write buffer write ref handling
bcachefs: bch2_btree_write_buffer_flush_locked()
bcachefs: bch2_btree_write_buffer_flush() -> bch2_btree_write_buffer_tryflush()
bcachefs: count_event()
bcachefs: Improve trace_trans_restart_too_many_iters()
bcachefs: Improve trace_trans_restart_would_deadlock
bcachefs: Don't open code bch2_dev_exists2()
bcachefs: ONLY_SPECIFIED_DEVS doesn't mean ignore durability anymore
bcachefs: wb_flush_one_slowpath()
bcachefs: more write buffer refactoring
bcachefs: Explicity go RW for fsck
bcachefs: On missing backpointer to interior node, flush interior updates
bcachefs: Make backpointer fsck wb flush check more rigorous
bcachefs: Include btree_trans in more tracepoints
bcachefs: Move reflink_p triggers into reflink.c
bcachefs: Refactor trans->paths_allocated to be standard bitmap
bcachefs: BCH_ERR_opt_parse_error
bcachefs: Improve error message when finding wrong btree node
bcachefs: c->ro_ref
bcachefs: thread_with_file
bcachefs: Add ability to redirect log output
bcachefs: Mark recovery passses that are safe to run online
bcachefs: bch2_run_online_recovery_passes()
bcachefs: BCH_IOCTL_FSCK_OFFLINE
bcachefs: BCH_IOCTL_FSCK_ONLINE
bcachefs: Fix open coded set_btree_iter_dontneed()
bcachefs: Fix bch2_read_btree()
bcachefs: continue now works in for_each_btree_key2()
bcachefs: Kill for_each_btree_key()
bcachefs: Rename for_each_btree_key2() -> for_each_btree_key()
bcachefs: reserve path idx 0 for sentinal
bcachefs: Fix snapshot.c assertion for online fsck
bcachefs: kill btree_path->(alloc_seq|downgrade_seq)
bcachefs; kill bch2_btree_key_cache_flush()
bcachefs: Improve trans->extra_journal_entries
bcachefs: bch2_trans_node_add no longer uses trans_for_each_path()
bcachefs: Unwritten journal buffers are always dirty
bcachefs: journal->buf_lock
bcachefs: btree write buffer now slurps keys from journal
bcachefs: Inline btree write buffer sort
bcachefs: check_root() can now be run online
bcachefs: kill btree_trans->wb_updates
bcachefs: Drop journal entry compaction
bcachefs: fix userspace build errors
bcachefs: bch_err_(fn|msg) check if should print
bcachefs: qstr_eq()
bcachefs: drop extra semicolon
bcachefs: Make sure allocation failure errors are logged
MAINTAINERS: Update my email address
bcachefs: Delete dio read alignment check
bcachefs: Fixes for rust bindgen
bcachefs: check for failure to downgrade
bcachefs: Use GFP_KERNEL for promote allocations
bcachefs: Improve the nopromote tracepoint
bcachefs: trans_for_each_update() now declares loop iter
bcachefs: darray_for_each() now declares loop iter
bcachefs: simplify bch_devs_list
bcachefs: better error message in btree_node_write_work()
bcachefs: add more verbose logging
bcachefs: fix warning about uninitialized time_stats
bcachefs: use track_event_change() for allocator blocked stats
bcachefs: bch2_trans_srcu_lock() should be static
bcachefs: bch2_dirent_lookup() -> lockrestart_do()
bcachefs: for_each_btree_key_upto() -> for_each_btree_key_old_upto()
bcachefs: kill for_each_btree_key_old_upto()
bcachefs: kill for_each_btree_key_norestart()
bcachefs: for_each_btree_key() now declares loop iter
bcachefs: for_each_member_device() now declares loop iter
bcachefs: for_each_member_device_rcu() now declares loop iter
bcachefs: vstruct_for_each() now declares loop iter
bcachefs: fsck -> bch2_trans_run()
bcachefs: kill __bch2_btree_iter_peek_upto_and_restart()
bcachefs: bkey_for_each_ptr() now declares loop iter
bcachefs: for_each_keylist_key() declares loop iter
bcachefs: skip journal more often in key cache reclaim
bcachefs: Convert split_devs() to darray
bcachefs: Kill GFP_NOFAIL usage in readahead path
bcachefs: minor bch2_btree_path_set_pos() optimization
bcachefs: bch2_path_get() -> btree_path_idx_t
bcachefs; bch2_path_put() -> btree_path_idx_t
bcachefs: bch2_btree_path_set_pos() -> btree_path_idx_t
bcachefs: bch2_btree_path_make_mut() -> btree_path_idx_t
bcachefs: bch2_btree_path_traverse() -> btree_path_idx_t
bcachefs: btree_path_alloc() -> btree_path_idx_t
bcachefs: btree_iter -> btree_path_idx_t
bcachefs: btree_insert_entry -> btree_path_idx_t
bcachefs: struct trans_for_each_path_inorder_iter
bcachefs: bch2_btree_path_to_text() -> btree_path_idx_t
bcachefs: kill trans_for_each_path_from()
bcachefs: trans_for_each_path() no longer uses path->idx
bcachefs: trans_for_each_path_with_node() no longer uses path->idx
bcachefs: bch2_path_get() no longer uses path->idx
bcachefs: bch2_btree_iter_peek_prev() no longer uses path->idx
bcachefs: get_unlocked_mut_path() -> btree_path_idx_t
bcachefs: kill btree_path.idx
bcachefs: Clean up btree_trans
bcachefs: rcu protect trans->paths
bcachefs: optimize __bch2_trans_get(), kill DEBUG_TRANSACTIONS
bcachefs: trans->updates will also be resizable
bcachefs: trans->nr_paths
bcachefs: Fix interior update path btree_path uses
bcachefs: growable btree_paths
bcachefs: bch2_btree_trans_peek_updates
bcachefs: bch2_btree_trans_peek_prev_updates
bcachefs: bch2_btree_trans_peek_slot_updates
bcachefs: Fix reattach_inode() for snapshots
bcachefs: check_directory_structure() can now be run online
bcachefs: Check journal entries for invalid keys in trans commit path
bcachefs: Fix nochanges/read_only interaction
bcachefs: bch_member->seq
bcachefs: Split brain detection
bcachefs: btree_trans always has stats
bcachefs: track transaction durations
bcachefs: wb_key_cmp -> wb_key_ref_cmp
bcachefs: __journal_keys_sort() refactoring
bcachefs: __bch2_journal_key_to_wb -> bch2_journal_key_to_wb_slowpath
bcachefs: Fix printing of device durability
bcachefs: factor out thread_with_file, thread_with_stdio
bcachefs: Upgrading uses bch_sb.recovery_passes_required
bcachefs: trans_mark now takes bkey_s
bcachefs: mark now takes bkey_s
bcachefs: Kill BTREE_TRIGGER_NOATOMIC
bcachefs: BTREE_TRIGGER_TRANSACTIONAL
bcachefs: kill mem_trigger_run_overwrite_then_insert()
bcachefs: unify inode trigger
bcachefs: unify reflink_p trigger
bcachefs: unify reservation trigger
bcachefs: move bch2_mark_alloc() to alloc_background.c
bcachefs: unify alloc trigger
bcachefs: move stripe triggers to ec.c
bcachefs: unify stripe trigger
bcachefs: bch2_trigger_pointer()
bcachefs: Online fsck can now fix errors
bcachefs: bch2_trigger_stripe_ptr()
bcachefs: unify extent trigger
bcachefs: Combine .trans_trigger, .atomic_trigger
bcachefs: kill useless return ret
bcachefs: Add an option to control btree node prefetching
bcachefs: don't clear accessed bit in btree node fill
bcachefs: add time_stats for btree_node_read_done()
bcachefs: increase max_active on io_complete_wq
bcachefs: add missing bch2_latency_acct() call
bcachefs: Don't autofix errors we can't fix
bcachefs: no thread_with_file in userspace
bcachefs: Upgrades now specify errors to fix, like downgrades
bcachefs: fsck_err()s don't need to manually check c->sb.version anymore
bcachefs: Improve would_deadlock trace event
bcachefs: %pg is banished
bcachefs: __bch2_sb_field_to_text()
bcachefs: print sb magic when relevant
bcachefs: improve validate_bset_keys()
bcachefs: improve checksum error messages
bcachefs: bch2_dump_bset() doesn't choke on u64s == 0
bcachefs: Restart recovery passes more reliably
bcachefs: fix simulateously upgrading & downgrading
bcachefs: move "ptrs not changing" optimization to bch2_trigger_extent()
bcachefs: eytzinger0_find() search should be const

Randy Dunlap (2):
bcachefs: six lock: fix typos
bcachefs: mean and variance: fix kernel-doc for function params

Richard Davies (1):
bcachefs: Remove obsolete comment about zstd

Yang Li (1):
bcachefs: clean up one inconsistent indenting

MAINTAINERS | 2 +-
arch/powerpc/kernel/firmware.c | 2 +
fs/bcachefs/Kconfig | 18 +-
fs/bcachefs/Makefile | 1 +
fs/bcachefs/alloc_background.c | 484 +++++-----
fs/bcachefs/alloc_background.h | 39 +-
fs/bcachefs/alloc_foreground.c | 46 +-
fs/bcachefs/backpointers.c | 199 +++--
fs/bcachefs/backpointers.h | 27 +-
fs/bcachefs/bcachefs.h | 192 +++-
fs/bcachefs/bcachefs_format.h | 123 ++-
fs/bcachefs/bcachefs_ioctl.h | 60 +-
fs/bcachefs/bkey_methods.h | 82 +-
fs/bcachefs/bset.c | 6 +
fs/bcachefs/btree_cache.c | 28 +-
fs/bcachefs/btree_cache.h | 4 +-
fs/bcachefs/btree_gc.c | 327 +++----
fs/bcachefs/btree_io.c | 132 ++-
fs/bcachefs/btree_io.h | 2 +-
fs/bcachefs/btree_iter.c | 945 ++++++++++----------
fs/bcachefs/btree_iter.h | 407 ++++-----
fs/bcachefs/btree_journal_iter.c | 25 +-
fs/bcachefs/btree_key_cache.c | 63 +-
fs/bcachefs/btree_key_cache.h | 2 -
fs/bcachefs/btree_locking.c | 111 ++-
fs/bcachefs/btree_locking.h | 16 +-
fs/bcachefs/btree_trans_commit.c | 313 +++----
fs/bcachefs/btree_types.h | 136 +--
fs/bcachefs/btree_update.c | 245 ++----
fs/bcachefs/btree_update.h | 111 ++-
fs/bcachefs/btree_update_interior.c | 322 +++----
fs/bcachefs/btree_update_interior.h | 11 +-
fs/bcachefs/btree_write_buffer.c | 668 +++++++++-----
fs/bcachefs/btree_write_buffer.h | 53 +-
fs/bcachefs/btree_write_buffer_types.h | 63 +-
fs/bcachefs/buckets.c | 1511 ++++++++------------------------
fs/bcachefs/buckets.h | 45 +-
fs/bcachefs/buckets_types.h | 2 -
fs/bcachefs/chardev.c | 363 ++++++--
fs/bcachefs/checksum.h | 23 +
fs/bcachefs/compress.c | 4 -
fs/bcachefs/darray.h | 8 +-
fs/bcachefs/data_update.c | 30 +-
fs/bcachefs/debug.c | 141 ++-
fs/bcachefs/dirent.c | 51 +-
fs/bcachefs/dirent.h | 7 +-
fs/bcachefs/disk_groups.c | 13 +-
fs/bcachefs/ec.c | 406 +++++++--
fs/bcachefs/ec.h | 5 +-
fs/bcachefs/ec_types.h | 2 +-
fs/bcachefs/errcode.h | 7 +-
fs/bcachefs/error.c | 103 ++-
fs/bcachefs/extent_update.c | 2 +-
fs/bcachefs/extents.c | 4 -
fs/bcachefs/extents.h | 24 +-
fs/bcachefs/eytzinger.h | 10 +-
fs/bcachefs/fs-common.c | 36 +-
fs/bcachefs/fs-io-buffered.c | 38 +-
fs/bcachefs/fs-io-direct.c | 3 -
fs/bcachefs/fs-io.c | 20 +-
fs/bcachefs/fs-ioctl.c | 12 +-
fs/bcachefs/fs.c | 100 +--
fs/bcachefs/fs.h | 9 +-
fs/bcachefs/fsck.c | 630 ++++++-------
fs/bcachefs/inode.c | 129 ++-
fs/bcachefs/inode.h | 15 +-
fs/bcachefs/io_misc.c | 55 +-
fs/bcachefs/io_read.c | 50 +-
fs/bcachefs/io_write.c | 45 +-
fs/bcachefs/journal.c | 108 ++-
fs/bcachefs/journal.h | 4 +-
fs/bcachefs/journal_io.c | 153 ++--
fs/bcachefs/journal_reclaim.c | 120 ++-
fs/bcachefs/journal_reclaim.h | 16 +-
fs/bcachefs/journal_seq_blacklist.c | 2 +-
fs/bcachefs/journal_types.h | 16 +-
fs/bcachefs/keylist.c | 2 -
fs/bcachefs/keylist.h | 4 +-
fs/bcachefs/logged_ops.c | 18 +-
fs/bcachefs/lru.c | 11 +-
fs/bcachefs/mean_and_variance.c | 10 +-
fs/bcachefs/mean_and_variance.h | 5 +-
fs/bcachefs/migrate.c | 9 +-
fs/bcachefs/move.c | 187 ++--
fs/bcachefs/move.h | 13 +-
fs/bcachefs/movinggc.c | 49 +-
fs/bcachefs/opts.c | 4 +-
fs/bcachefs/opts.h | 20 +-
fs/bcachefs/quota.c | 28 +-
fs/bcachefs/rebalance.c | 38 +-
fs/bcachefs/recovery.c | 291 +++---
fs/bcachefs/recovery.h | 1 +
fs/bcachefs/recovery_types.h | 25 +-
fs/bcachefs/reflink.c | 224 ++++-
fs/bcachefs/reflink.h | 22 +-
fs/bcachefs/replicas.c | 66 +-
fs/bcachefs/replicas.h | 22 +-
fs/bcachefs/replicas_types.h | 6 +-
fs/bcachefs/sb-clean.c | 20 +-
fs/bcachefs/sb-downgrade.c | 90 +-
fs/bcachefs/sb-downgrade.h | 1 +
fs/bcachefs/sb-errors_types.h | 4 +-
fs/bcachefs/sb-members.c | 18 +-
fs/bcachefs/sb-members.h | 100 ++-
fs/bcachefs/six.c | 117 +--
fs/bcachefs/six.h | 13 +-
fs/bcachefs/snapshot.c | 174 ++--
fs/bcachefs/snapshot.h | 8 +-
fs/bcachefs/str_hash.h | 25 +-
fs/bcachefs/subvolume.c | 31 +-
fs/bcachefs/subvolume_types.h | 4 +
fs/bcachefs/super-io.c | 168 ++--
fs/bcachefs/super-io.h | 7 +-
fs/bcachefs/super.c | 388 ++++----
fs/bcachefs/super.h | 6 +-
fs/bcachefs/super_types.h | 2 +-
fs/bcachefs/sysfs.c | 160 ++--
fs/bcachefs/tests.c | 193 ++--
fs/bcachefs/thread_with_file.c | 299 +++++++
fs/bcachefs/thread_with_file.h | 41 +
fs/bcachefs/thread_with_file_types.h | 16 +
fs/bcachefs/trace.h | 278 ++++--
fs/bcachefs/util.c | 191 ++--
fs/bcachefs/util.h | 56 +-
fs/bcachefs/vstructs.h | 10 +-
125 files changed, 7101 insertions(+), 5961 deletions(-)
create mode 100644 fs/bcachefs/thread_with_file.c
create mode 100644 fs/bcachefs/thread_with_file.h
create mode 100644 fs/bcachefs/thread_with_file_types.h