Re: [RFC PATCH v4] ceph: prevent a client from exceeding the MDS maximum xattr size

[Xiubo Li]

On 6/3/22 7:52 PM, Luís Henriques wrote:
> Xiubo Li <xiubli@xxxxxxxxxx> writes:
>
> > On 6/2/22 10:36 PM, Luís Henriques wrote:
> > > The MDS tries to enforce a limit on the total key/values in extended
> > > attributes.  However, this limit is enforced only if doing a synchronous
> > > operation (MDS_OP_SETXATTR) -- if we're buffering the xattrs, the MDS
> > > doesn't have a chance to enforce these limits.
> > >
> > > This patch adds support for decoding the xattrs maximum size setting that is
> > > distributed in the mdsmap.  Then, when setting an xattr, the kernel client
> > > will revert to do a synchronous operation if that maximum size is exceeded.
> > >
> > > While there, fix a dout() that would trigger a printk warning:
> > >
> > > [   98.718078] ------------[ cut here ]------------
> > > [   98.719012] precision 65536 too large
> > > [   98.719039] WARNING: CPU: 1 PID: 3755 at lib/vsprintf.c:2703 vsnprintf+0x5e3/0x600
> > > ...
> > >
> > > URL: https://tracker.ceph.com/issues/55725
> > > Signed-off-by: Luís Henriques <lhenriques@xxxxxxx>
> > > ---
> > >    fs/ceph/mdsmap.c            | 28 ++++++++++++++++++++++++----
> > >    fs/ceph/xattr.c             | 12 ++++++++----
> > >    include/linux/ceph/mdsmap.h |  1 +
> > >    3 files changed, 33 insertions(+), 8 deletions(-)
> > >
> > > * Changes since v3
> > >
> > > As per Xiubo review:
> > >     - Always force a (sync) SETXATTR Op when connecting to an old cluster
> > >     - use '>' instead of '>='
> > > Also fixed the warning detected by 0day.
> > >
> > > * Changes since v2
> > >
> > > Well, a lot has changed since v2!  Now the xattr max value setting is
> > > obtained through the mdsmap, which needs to be decoded, and the feature
> > > that was used in the previous revision was dropped.  The drawback is that
> > > the MDS isn't unable to know in advance if a client is aware of this xattr
> > > max value.
> > >
> > > * Changes since v1
> > >
> > > Added support for new feature bit to get the MDS max_xattr_pairs_size
> > > setting.
> > >
> > > Also note that this patch relies on a patch that hasn't been merged yet
> > > ("ceph: use correct index when encoding client supported features"),
> > > otherwise the new feature bit won't be correctly encoded.
> > >
> > > diff --git a/fs/ceph/mdsmap.c b/fs/ceph/mdsmap.c
> > > index 30387733765d..c6ce83a48175 100644
> > > --- a/fs/ceph/mdsmap.c
> > > +++ b/fs/ceph/mdsmap.c
> > > @@ -13,6 +13,12 @@
> > >      #include "super.h"
> > >    +/*
> > > + * Maximum size of xattrs the MDS can handle per inode by default.  This
> > > + * includes the attribute name and 4+4 bytes for the key/value sizes.
> > > + */
> > > +#define MDS_MAX_XATTR_SIZE (1<<16) /* 64K */
> > This macro makes no sense anymore, right ?
> Ah, yeah.  That's still leftovers from the previous version.
>
> > I will merge this patch by removing this then.
> Thanks!  Also... maybe you could also fix the (embarrassing) issue
> detected by 0day that I _tried_ to fix but failed (see below).
>
> > -- Xiubo
> >
> >
> > > +
> > >    #define CEPH_MDS_IS_READY(i, ignore_laggy) \
> > >    	(m->m_info[i].state > 0 && ignore_laggy ? true : !m->m_info[i].laggy)
> > >    @@ -352,12 +358,10 @@ struct ceph_mdsmap *ceph_mdsmap_decode(void **p, void
> > > *end, bool msgr2)
> > >    		__decode_and_drop_type(p, end, u8, bad_ext);
> > >    	}
> > >    	if (mdsmap_ev >= 8) {
> > > -		u32 name_len;
> > >    		/* enabled */
> > >    		ceph_decode_8_safe(p, end, m->m_enabled, bad_ext);
> > > -		ceph_decode_32_safe(p, end, name_len, bad_ext);
> > > -		ceph_decode_need(p, end, name_len, bad_ext);
> > > -		*p += name_len;
> > > +		/* fs_name */
> > > +		ceph_decode_skip_string(p, end, bad_ext);
> > >    	}
> > >    	/* damaged */
> > >    	if (mdsmap_ev >= 9) {
> > > @@ -370,6 +374,22 @@ struct ceph_mdsmap *ceph_mdsmap_decode(void **p, void *end, bool msgr2)
> > >    	} else {
> > >    		m->m_damaged = false;
> > >    	}
> > > +	if (mdsmap_ev >= 17) {
> > > +		/* balancer */
> > > +		ceph_decode_skip_string(p, end, bad_ext);
> > > +		/* standby_count_wanted */
> > > +		ceph_decode_skip_32(p, end, bad_ext);
> > > +		/* old_max_mds */
> > > +		ceph_decode_skip_32(p, end, bad_ext);
> > > +		/* min_compat_client */
> > > +		ceph_decode_skip_8(p, end, bad_ext);
> > > +		/* required_client_features */
> > > +		ceph_decode_skip_set(p, end, 64, bad_ext);
> > > +		ceph_decode_64_safe(p, end, m->m_max_xattr_size, bad_ext);
> > > +	} else {
> > > +		/* This forces the usage of the (sync) SETXATTR Op */
> > > +		m->m_max_xattr_size = 0;
> > > +	}
> > >    bad_ext:
> > >    	dout("mdsmap_decode m_enabled: %d, m_damaged: %d, m_num_laggy: %d\n",
> > >    	     !!m->m_enabled, !!m->m_damaged, m->m_num_laggy);
> > > diff --git a/fs/ceph/xattr.c b/fs/ceph/xattr.c
> > > index 8c2dc2c762a4..1be415e9220b 100644
> > > --- a/fs/ceph/xattr.c
> > > +++ b/fs/ceph/xattr.c
> > > @@ -1086,7 +1086,7 @@ static int ceph_sync_setxattr(struct inode *inode, const char *name,
> > >    			flags |= CEPH_XATTR_REMOVE;
> > >    	}
> > >    -	dout("setxattr value=%.*s\n", (int)size, value);
> > > +	dout("setxattr value size: %lu\n", size);
> This should be "%zu" and not "%lu".  I'm OK sending v5 to fix these 2
> issues if you prefer.

Sure, please send a v5 for it.


> [ And then there's also the MDS-side which hasn't been merged yet.  But
>    this patch is probably worth in itself as it will change the client
>    behavior to do the sync operation. ]

It's okay, this will fix this issue when we are connecting to old cephs, 
and we could force to do the sync request always.

No need to wait the ceph PR.

-- Xiubo

> Cheers,