Re: [PATCH] blk-iolatency: flush enable work after policy deactivation

[Cen Zhang]

Hi all,

Sorry for the noise, but I wanted to gently follow up on this patch in
case it got buried.

I would really appreciate any feedback when you have a chance. Please
let me know if I should make any changes or provide more testing
results.

Thanks a lot.

Best regards,
Cen Zhang

Cen Zhang <zzzccc427@xxxxxxxxx> 于2026年6月21日周日 21:59写道：
>
> A blk-iolatency rq-qos teardown can free struct blk_iolatency while a
> freshly queued enable_work callback still references it. The observed
> failure is:
>
> blkcg_iolatency_exit() flushes enable_work before deactivating the
> iolatency policy. However, blkcg_deactivate_policy() calls
> iolatency_pd_offline() for online policy data, and iolatency_pd_offline()
> clears min_lat_nsec through iolatency_set_min_lat_nsec(). If this clears
> the last nonzero latency target, enable_cnt reaches zero and schedules
> enable_work again after the flush has already returned.
>
> The buggy scenario involves two paths, with each column showing the order
> within that path:
>
> blkcg_iolatency_exit() path:          system_wq worker path:
> 1. Flush old enable_work.             1. enable_work is idle.
> 2. Deactivate the policy.             2. no worker owns it.
> 3. Offline queues new enable_work.    3. work item becomes pending.
> 4. Free blkiolat.                     4. worker later runs the item.
> 5. Owner storage is gone.             5. worker dereferences blkiolat.
>
> Flush enable_work again after blkcg_deactivate_policy() returns and before
> freeing blkiolat. Policy offline callbacks have completed at that point,
> so the second drain covers the late queueing path without changing the
> normal enable/disable accounting rules.
>
> Validation reproduced this kernel report:
> BUG: KASAN: slab-use-after-free in assign_work+0x2a/0x150
>
> Call Trace:
>  <TASK>
>  dump_stack_lvl+0x53/0x70
>  print_report+0xd0/0x630
>  ? __pfx__raw_spin_lock_irqsave+0x10/0x10
>  ? srso_alias_return_thunk+0x5/0xfbef5
>  ? __virt_addr_valid+0xea/0x1a0
>  ? assign_work+0x2a/0x150
>  kasan_report+0xce/0x100
>  ? assign_work+0x2a/0x150
>  assign_work+0x2a/0x150
>  worker_thread+0x1b7/0x500
>  ? __pfx_worker_thread+0x10/0x10
>  kthread+0x192/0x1d0
>  ? __pfx_kthread+0x10/0x10
>  ret_from_fork+0x2ac/0x3c0
>  ? __pfx_ret_from_fork+0x10/0x10
>  ? srso_alias_return_thunk+0x5/0xfbef5
>  ? __switch_to+0x2d5/0x6e0
>  ? __pfx_kthread+0x10/0x10
>  ret_from_fork_asm+0x1a/0x30
>  </TASK>
>
> Allocated by task 470:
>  kasan_save_stack+0x33/0x60
>  kasan_save_track+0x14/0x30
>  __kasan_kmalloc+0x8f/0xa0
>  iolatency_set_limit+0x301/0x450
>  cgroup_file_write+0x178/0x2e0
>  kernfs_fop_write_iter+0x1ef/0x290
>  vfs_write+0x446/0x6f0
>  ksys_write+0xc7/0x160
>  do_syscall_64+0xf9/0x540
>  entry_SYSCALL_64_after_hwframe+0x77/0x7f
>
> Freed by task 611:
>  kasan_save_stack+0x33/0x60
>  kasan_save_track+0x14/0x30
>  kasan_save_free_info+0x3b/0x60
>  __kasan_slab_free+0x43/0x70
>  kfree+0x131/0x390
>  rq_qos_exit+0x5d/0x90
>  __del_gendisk+0x394/0x490
>  del_gendisk+0xa1/0xe0
>  virtblk_remove+0x41/0xd0
>  virtio_dev_remove+0x63/0xe0
>  device_release_driver_internal+0x246/0x2e0
>  unbind_store+0xa9/0xb0
>  kernfs_fop_write_iter+0x1ef/0x290
>  vfs_write+0x446/0x6f0
>  ksys_write+0xc7/0x160
>  do_syscall_64+0xf9/0x540
>  entry_SYSCALL_64_after_hwframe+0x77/0x7f
>
> Last potentially related work creation:
>  kasan_save_stack+0x33/0x60
>  kasan_record_aux_stack+0x8c/0xa0
>  __queue_work+0x42a/0x800
>  queue_work_on+0x5d/0x70
>  iolatency_set_min_lat_nsec+0x196/0x230
>  iolatency_pd_offline+0x1f/0x40
>  blkcg_deactivate_policy+0x194/0x270
>  blkcg_iolatency_exit+0x33/0x40
>  rq_qos_exit+0x5d/0x90
>  __del_gendisk+0x394/0x490
>  del_gendisk+0xa1/0xe0
>  virtblk_remove+0x41/0xd0
>  virtio_dev_remove+0x63/0xe0
>  device_release_driver_internal+0x246/0x2e0
>  unbind_store+0xa9/0xb0
>  kernfs_fop_write_iter+0x1ef/0x290
>  vfs_write+0x446/0x6f0
>  ksys_write+0xc7/0x160
>  do_syscall_64+0xf9/0x540
>  entry_SYSCALL_64_after_hwframe+0x77/0x7f
>
> Second to last potentially related work creation:
>  kasan_save_stack+0x33/0x60
>  kasan_record_aux_stack+0x8c/0xa0
>  __queue_work+0x42a/0x800
>  queue_work_on+0x5d/0x70
>  iolatency_set_min_lat_nsec+0x196/0x230
>  iolatency_set_limit+0x3f1/0x450
>  cgroup_file_write+0x178/0x2e0
>  kernfs_fop_write_iter+0x1ef/0x290
>  vfs_write+0x446/0x6f0
>  ksys_write+0xc7/0x160
>  do_syscall_64+0xf9/0x540
>  entry_SYSCALL_64_after_hwframe+0x77/0x7f
>
> Fixes: 8a177a36da6c ("blk-iolatency: Fix inflight count imbalances and IO hangs on offline")
> Assisted-by: Codex:gpt-5.5
> Signed-off-by: Cen Zhang <zzzccc427@xxxxxxxxx>
> ---
>  block/blk-iolatency.c | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c
> index 1aaee6fb0f59..a0bdd8a5c94c 100644
> --- a/block/blk-iolatency.c
> +++ b/block/blk-iolatency.c
> @@ -639,6 +639,11 @@ static void blkcg_iolatency_exit(struct rq_qos *rqos)
>         timer_shutdown_sync(&blkiolat->timer);
>         flush_work(&blkiolat->enable_work);
>         blkcg_deactivate_policy(rqos->disk, &blkcg_policy_iolatency);
> +       /*
> +        * blkcg_deactivate_policy() invokes iolatency_pd_offline(), which may
> +        * queue enable_work again when it clears the last latency target.
> +        */
> +       flush_work(&blkiolat->enable_work);
>         kfree(blkiolat);
>  }
>
> --
> 2.43.0
>