Re: Unsubscription Incident

[John 'Warthog9' Hawley]

On 10/27/21 17:27, Slade Watkins wrote:
> On Wed, Oct 27, 2021 at 4:42 PM John 'Warthog9' Hawley
> <warthog9@xxxxxxxxxxxxxx> wrote:
> > On 10/27/21 12:34 PM, Jakub Kicinski wrote:
> > > On Mon, 25 Oct 2021 11:34:28 -0700 Shannon Nelson wrote:
> > > > > > It happened to a bunch of people on gmail:
> > > > > > https://lore.kernel.org/netdev/1fd8d0ac-ba8a-4836-59ab-0ed3b0321775@xxxxxxxxxxxx/t/#u
> > > > > I can at least confirm that this didn't happen to me on my hosted
> > > > > Gmail through Google Workspace. Could be wrong, but it seems isolated
> > > > > to normal @gmail.com accounts.
> > > > >
> > > > > Best,
> > > >
> > > > Alternatively, I can confirm that my pensando.io address through gmail
> > > > was affected until I re-subscribed.
> > >
> > > Did it just work after re-subscribing again? Without cleaning the inbox?
> > > John indicated off list that Gmail started returning errors related to
> > > quota, no idea what that translates to in reality maybe they added some
> > > heuristic on too many emails from one source?
> >
> > At least for the users I've had anyone mention to me (which for the
> > record apparently this happened on the 11th, and people are only
> > reaching out now about), the reasons for the unsubscribe was that the
> > upstream servers were reporting that the users in question were over
> > quota permanently.  We take that hinting at face value, and since the
> > server is telling us (basically) that the user isn't going to be
> > accepting mail anytime soon, we go ahead and unsubscribe them and clear
> > the queue so that the users don't cause unnecessary back log.  Noting,
> > this is an automated process that runs and deals with this that runs
> > periodically.
> >
> > Also noting, that there's not a good way to notify individuals when this
> > happens because, unsurprisingly, their email providers aren't accepting
> > mail from us.
> >
> > If folks reach out to postmaster@ I'm more than happy to take a look at
> > the 'why' something happened, and I'm happy to re-subscribe folks in the
> > backend saving them the back and forth with majorodomo's command system.
>
> John,
> That's great.
>
> > If I had to speculate, something glitched at gmail, a subset of users
> > got an odd error code returned (which likely wasn't the correct error
> > code for the situation, and noting the number of affected users is
> > fairly small given the number of users from gmail that are subscribed).
> >   Likely similar to when gmail had that big outage and it reported
> > something way off base and as a result every gmail user got unsubscribed
> > (and subsequently resubscribed in the backend by me when the outage was
> > over).
>
> Is there any way to detect if something like that affects Google
> Workspace hosted inboxes too? Sounds like those in that group who were
> affected were very few though but I thought I'd ask anyway.

Looking back around then it looks like a few individual users might have 
been, but it's sporadic on the domains that aren't gmail.com, so it 
doesn't look like (to what I can see anyway) it was a full worksapce for 
instance.

The real problem is the error that was getting kicked back was a generic 
enough error, that's it's hard to tell what users may have actually been 
over quota vs. something else.

- John 'Warthog9' Hawley