The driver, recognizing that the .port_vlan_filtering callback was never
coming after the port left its parent bridge, decided to take that duty
in its own hands. DSA now takes care of this condition, so fix that.
Signed-off-by: Vladimir Oltean <olteanv@xxxxxxxxx>