Re: [PATCH v2] f2fs: Add option to limit required GC for checkpoint=disable

[Chao Yu]

On 2019/5/2 9:42, Daniel Rosenberg wrote:
> This extends the checkpoint option to allow checkpoint=disable:%u
> This allows you to specify what percent of the drive you are willing
> to lose access to while mounting with checkpoint=disable. If the amount
> lost would be higher, the mount will return -EAGAIN.
> Currently, we need to run garbage collection until the amount of holes
> is smaller than the OVP space. With the new option, f2fs can mark
> space as unusable up front instead of requiring that
> the space be freed up with garbage collection.
> 
> Change-Id: Ib8c3cb46085bf4aac3cdc06d4b64d6a20db87028
> Signed-off-by: Daniel Rosenberg <drosen@xxxxxxxxxx>
> ---
> 
>> If we allow a high unusable blocks, I doubt in inc_valid_{node,block}_count(),
>> we may overflow in below calculation:
>>
>>         if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
>>                 valid_block_count += sbi->unusable_block_count;
>>
>>         if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
>>                 avail_user_block_count -= sbi->unusable_block_count;
> The starting value of unusable_block_count is limited.
> Considering the case where the disk is full, we have
> total blocks = used + OVP
> OVP = reserved + holes
> So, the holes must be less than OVP, and unusable = 0
> 
> In general, total = used + available + OVP
> total-used-OVP=avail
> 
> (free being blocks from free segments,
> available being blocks available to the user)
> reserved + holes + free = OVP + available
> holes - OVP = available - reserved - free
> 
> unusable = max(data_holes, node_holes)-OVP
> unusable <= holes-OVP
> 
> unusable <= available-reserved - free segs
> unusable <= available
> 
> similarly,
> total = used + available + OVP
> reserved + holes + free = OVP + available
> available = reserved + holes + free - OVP
> total = used + reserved + holes + free - OVP
> unusable<=holes-OVP
> total >= used + reserved + unusable + free
> total >= used + unusable

Thanks for your detailed explanation. :)

> 
> So the unusable block count is lower than the available block count when we
> first mount with checkpoint=disable, and we only grow unusable when we grow
> available blocks. We also have an upper bound on used+unusable which is less
> the total block count. So unless I've missed something elsewhere, we should
> not be able to underflow or overflow these values.

Look into the code again, I guess with additional sbi->current_reserved_blocks,
we may overflow? as current_reserved_blocks may has included the hole space, right?

	valid_block_count = sbi->total_valid_block_count +
					sbi->current_reserved_blocks + 1;

	if (!__allow_reserved_blocks(sbi, inode, false))
		valid_block_count += F2FS_OPTION(sbi).root_reserved_blocks;
	if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
		valid_block_count += sbi->unusable_block_count;

Thanks,

> 
> v2:
> Changed default to 0 to be in line with current behavior.
> Adjusted description of checkpoint=disable to be more clear, since with that
> default, mounting with checkpoint=disable may return EAGAIN unless you use
> checkpoint=disable:100
> Changed description of /sys/fs/f2fs/<disk>/unusable to be clearer
> 
>  Documentation/ABI/testing/sysfs-fs-f2fs |  8 +++++
>  Documentation/filesystems/f2fs.txt      | 12 ++++++-
>  fs/f2fs/f2fs.h                          |  6 +++-
>  fs/f2fs/segment.c                       | 17 ++++++++--
>  fs/f2fs/super.c                         | 44 +++++++++++++------------
>  fs/f2fs/sysfs.c                         | 16 +++++++++
>  6 files changed, 77 insertions(+), 26 deletions(-)
> 
> diff --git a/Documentation/ABI/testing/sysfs-fs-f2fs b/Documentation/ABI/testing/sysfs-fs-f2fs
> index 91822ce258317..dca326e0ee3e1 100644
> --- a/Documentation/ABI/testing/sysfs-fs-f2fs
> +++ b/Documentation/ABI/testing/sysfs-fs-f2fs
> @@ -243,3 +243,11 @@ Description:
>  		 - Del: echo '[h/c]!extension' > /sys/fs/f2fs/<disk>/extension_list
>  		 - [h] means add/del hot file extension
>  		 - [c] means add/del cold file extension
> +
> +What:		/sys/fs/f2fs/<disk>/unusable
> +Date		April 2019
> +Contact:	"Daniel Rosenberg" <drosen@xxxxxxxxxx>
> +Description:
> +		If checkpoint=disable, it displays the number of blocks that are unusable.
> +                If checkpoint=enable it displays the enumber of blocks that would be unusable
> +                if checkpoint=disable were to be set.
> diff --git a/Documentation/filesystems/f2fs.txt b/Documentation/filesystems/f2fs.txt
> index f7b5e4ff0de3e..d49f5ac8bf9b6 100644
> --- a/Documentation/filesystems/f2fs.txt
> +++ b/Documentation/filesystems/f2fs.txt
> @@ -214,11 +214,21 @@ fsync_mode=%s          Control the policy of fsync. Currently supports "posix",
>                         non-atomic files likewise "nobarrier" mount option.
>  test_dummy_encryption  Enable dummy encryption, which provides a fake fscrypt
>                         context. The fake fscrypt context is used by xfstests.
> -checkpoint=%s          Set to "disable" to turn off checkpointing. Set to "enable"
> +checkpoint=%s[:%u]     Set to "disable" to turn off checkpointing. Set to "enable"
>                         to reenable checkpointing. Is enabled by default. While
>                         disabled, any unmounting or unexpected shutdowns will cause
>                         the filesystem contents to appear as they did when the
>                         filesystem was mounted with that option.
> +                       While mounting with checkpoint=disabled, the filesystem must
> +                       run garbage collection to ensure that all available space can
> +                       be used. If this takes too much time, the mount may return
> +                       EAGAIN. You may optionally add a percentage to indicate what
> +                       percent of the disk you would be willing to temporarily give
> +                       up to avoid additional garbage collection. For instance,
> +                       checkpoint=disable:100 would always succeed, but would
> +                       potentially make the disk appear full. The actual space that
> +                       would be unsuable can be viewed at /sys/fs/f2fs/<disk>/unusable
> +                       This space is reclaimed once checkpoint=enable.
>  
>  ================================================================================
>  DEBUGFS ENTRIES
> diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
> index 30acde08822ef..d0477251cec56 100644
> --- a/fs/f2fs/f2fs.h
> +++ b/fs/f2fs/f2fs.h
> @@ -136,6 +136,9 @@ struct f2fs_mount_info {
>  	int alloc_mode;			/* segment allocation policy */
>  	int fsync_mode;			/* fsync policy */
>  	bool test_dummy_encryption;	/* test dummy encryption */
> +	block_t unusable_percent_cap;   /* Percent of space allowed to be
> +					 * unusable when disabling checkpoint
> +					 */
>  };
>  
>  #define F2FS_FEATURE_ENCRYPT		0x0001
> @@ -3049,7 +3052,8 @@ bool f2fs_issue_discard_timeout(struct f2fs_sb_info *sbi);
>  void f2fs_clear_prefree_segments(struct f2fs_sb_info *sbi,
>  					struct cp_control *cpc);
>  void f2fs_dirty_to_prefree(struct f2fs_sb_info *sbi);
> -int f2fs_disable_cp_again(struct f2fs_sb_info *sbi);
> +block_t f2fs_get_unusable_blocks(struct f2fs_sb_info *sbi);
> +int f2fs_disable_cp_again(struct f2fs_sb_info *sbi, block_t unusable);
>  void f2fs_release_discard_addrs(struct f2fs_sb_info *sbi);
>  int f2fs_npages_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra);
>  void f2fs_allocate_new_segments(struct f2fs_sb_info *sbi);
> diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c
> index a3380d1de6000..704224f4a2866 100644
> --- a/fs/f2fs/segment.c
> +++ b/fs/f2fs/segment.c
> @@ -869,11 +869,12 @@ void f2fs_dirty_to_prefree(struct f2fs_sb_info *sbi)
>  	mutex_unlock(&dirty_i->seglist_lock);
>  }
>  
> -int f2fs_disable_cp_again(struct f2fs_sb_info *sbi)
> +block_t f2fs_get_unusable_blocks(struct f2fs_sb_info *sbi)
>  {
> -	struct dirty_seglist_info *dirty_i = DIRTY_I(sbi);
>  	block_t ovp = overprovision_segments(sbi) << sbi->log_blocks_per_seg;
> +	struct dirty_seglist_info *dirty_i = DIRTY_I(sbi);
>  	block_t holes[2] = {0, 0};	/* DATA and NODE */
> +	block_t unusable;
>  	struct seg_entry *se;
>  	unsigned int segno;
>  
> @@ -887,7 +888,17 @@ int f2fs_disable_cp_again(struct f2fs_sb_info *sbi)
>  	}
>  	mutex_unlock(&dirty_i->seglist_lock);
>  
> -	if (holes[DATA] > ovp || holes[NODE] > ovp)
> +	unusable = holes[DATA] > holes[NODE] ? holes[DATA] : holes[NODE];
> +	if (unusable > ovp)
> +		return unusable - ovp;
> +	return 0;
> +}
> +
> +int f2fs_disable_cp_again(struct f2fs_sb_info *sbi, block_t unusable)
> +{
> +	block_t max_allowed =  (sbi->user_block_count / 100) *
> +			F2FS_OPTION(sbi).unusable_percent_cap;
> +	if (unusable > max_allowed)
>  		return -EAGAIN;
>  	if (is_sbi_flag_set(sbi, SBI_CP_DISABLED_QUICK) &&
>  		dirty_segments(sbi) > overprovision_segments(sbi))
> diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
> index 0b6fa77c35f3e..640753cb3f73f 100644
> --- a/fs/f2fs/super.c
> +++ b/fs/f2fs/super.c
> @@ -136,7 +136,9 @@ enum {
>  	Opt_alloc,
>  	Opt_fsync,
>  	Opt_test_dummy_encryption,
> -	Opt_checkpoint,
> +	Opt_checkpoint_disable,
> +	Opt_checkpoint_disable_cap,
> +	Opt_checkpoint_enable,
>  	Opt_err,
>  };
>  
> @@ -195,7 +197,9 @@ static match_table_t f2fs_tokens = {
>  	{Opt_alloc, "alloc_mode=%s"},
>  	{Opt_fsync, "fsync_mode=%s"},
>  	{Opt_test_dummy_encryption, "test_dummy_encryption"},
> -	{Opt_checkpoint, "checkpoint=%s"},
> +	{Opt_checkpoint_disable, "checkpoint=disable"},
> +	{Opt_checkpoint_disable_cap, "checkpoint=disable:%u"},
> +	{Opt_checkpoint_enable, "checkpoint=enable"},
>  	{Opt_err, NULL},
>  };
>  
> @@ -771,22 +775,17 @@ static int parse_options(struct super_block *sb, char *options)
>  					"Test dummy encryption mount option ignored");
>  #endif
>  			break;
> -		case Opt_checkpoint:
> -			name = match_strdup(&args[0]);
> -			if (!name)
> -				return -ENOMEM;
> -
> -			if (strlen(name) == 6 &&
> -					!strncmp(name, "enable", 6)) {
> -				clear_opt(sbi, DISABLE_CHECKPOINT);
> -			} else if (strlen(name) == 7 &&
> -					!strncmp(name, "disable", 7)) {
> -				set_opt(sbi, DISABLE_CHECKPOINT);
> -			} else {
> -				kvfree(name);
> +		case Opt_checkpoint_disable_cap:
> +			if (args->from && match_int(args, &arg))
>  				return -EINVAL;
> -			}
> -			kvfree(name);
> +			if (arg < 0 || arg > 100)
> +				return -EINVAL;
> +			F2FS_OPTION(sbi).unusable_percent_cap = arg;
> +		case Opt_checkpoint_disable:
> +			set_opt(sbi, DISABLE_CHECKPOINT);
> +			break;
> +		case Opt_checkpoint_enable:
> +			clear_opt(sbi, DISABLE_CHECKPOINT);
>  			break;
>  		default:
>  			f2fs_msg(sb, KERN_ERR,
> @@ -1411,8 +1410,8 @@ static int f2fs_show_options(struct seq_file *seq, struct dentry *root)
>  		seq_printf(seq, ",alloc_mode=%s", "reuse");
>  
>  	if (test_opt(sbi, DISABLE_CHECKPOINT))
> -		seq_puts(seq, ",checkpoint=disable");
> -
> +		seq_printf(seq, ",checkpoint=disable:%u",
> +				F2FS_OPTION(sbi).unusable_percent_cap);
>  	if (F2FS_OPTION(sbi).fsync_mode == FSYNC_MODE_POSIX)
>  		seq_printf(seq, ",fsync_mode=%s", "posix");
>  	else if (F2FS_OPTION(sbi).fsync_mode == FSYNC_MODE_STRICT)
> @@ -1441,6 +1440,7 @@ static void default_options(struct f2fs_sb_info *sbi)
>  	set_opt(sbi, EXTENT_CACHE);
>  	set_opt(sbi, NOHEAP);
>  	clear_opt(sbi, DISABLE_CHECKPOINT);
> +	F2FS_OPTION(sbi).unusable_percent_cap = 0;
>  	sbi->sb->s_flags |= SB_LAZYTIME;
>  	set_opt(sbi, FLUSH_MERGE);
>  	set_opt(sbi, DISCARD);
> @@ -1469,6 +1469,7 @@ static int f2fs_disable_checkpoint(struct f2fs_sb_info *sbi)
>  	struct cp_control cpc;
>  	int err = 0;
>  	int ret;
> +	block_t unusable;
>  
>  	if (s_flags & SB_RDONLY) {
>  		f2fs_msg(sbi->sb, KERN_ERR,
> @@ -1496,7 +1497,8 @@ static int f2fs_disable_checkpoint(struct f2fs_sb_info *sbi)
>  		goto restore_flag;
>  	}
>  
> -	if (f2fs_disable_cp_again(sbi)) {
> +	unusable = f2fs_get_unusable_blocks(sbi);
> +	if (f2fs_disable_cp_again(sbi, unusable)) {
>  		err = -EAGAIN;
>  		goto restore_flag;
>  	}
> @@ -1506,7 +1508,7 @@ static int f2fs_disable_checkpoint(struct f2fs_sb_info *sbi)
>  	set_sbi_flag(sbi, SBI_CP_DISABLED);
>  	f2fs_write_checkpoint(sbi, &cpc);
>  
> -	sbi->unusable_block_count = 0;
> +	sbi->unusable_block_count = unusable;
>  	mutex_unlock(&sbi->gc_mutex);
>  restore_flag:
>  	sbi->sb->s_flags = s_flags;	/* Restore MS_RDONLY status */
> diff --git a/fs/f2fs/sysfs.c b/fs/f2fs/sysfs.c
> index 729f46a3c9ee0..fa184880cff34 100644
> --- a/fs/f2fs/sysfs.c
> +++ b/fs/f2fs/sysfs.c
> @@ -68,6 +68,20 @@ static ssize_t dirty_segments_show(struct f2fs_attr *a,
>  		(unsigned long long)(dirty_segments(sbi)));
>  }
>  
> +static ssize_t unusable_show(struct f2fs_attr *a,
> +		struct f2fs_sb_info *sbi, char *buf)
> +{
> +	block_t unusable;
> +
> +	if (test_opt(sbi, DISABLE_CHECKPOINT))
> +		unusable = sbi->unusable_block_count;
> +	else
> +		unusable = f2fs_get_unusable_blocks(sbi);
> +	return snprintf(buf, PAGE_SIZE, "%llu\n",
> +		(unsigned long long)unusable);
> +}
> +
> +
>  static ssize_t lifetime_write_kbytes_show(struct f2fs_attr *a,
>  		struct f2fs_sb_info *sbi, char *buf)
>  {
> @@ -440,6 +454,7 @@ F2FS_GENERAL_RO_ATTR(dirty_segments);
>  F2FS_GENERAL_RO_ATTR(lifetime_write_kbytes);
>  F2FS_GENERAL_RO_ATTR(features);
>  F2FS_GENERAL_RO_ATTR(current_reserved_blocks);
> +F2FS_GENERAL_RO_ATTR(unusable);
>  
>  #ifdef CONFIG_FS_ENCRYPTION
>  F2FS_FEATURE_RO_ATTR(encryption, FEAT_CRYPTO);
> @@ -495,6 +510,7 @@ static struct attribute *f2fs_attrs[] = {
>  	ATTR_LIST(inject_type),
>  #endif
>  	ATTR_LIST(dirty_segments),
> +	ATTR_LIST(unusable),
>  	ATTR_LIST(lifetime_write_kbytes),
>  	ATTR_LIST(features),
>  	ATTR_LIST(reserved_blocks),
>