Re: [PATCH 0/1] Fiji GPU audio register timeout when in BACO state

[Nicholas Johnson]

On Tue, Apr 28, 2020 at 09:57:24AM +0200, Takashi Iwai wrote:
> On Mon, 27 Apr 2020 20:43:54 +0200,
> Alex Deucher wrote:
> > 
> > On Mon, Apr 27, 2020 at 2:39 PM Takashi Iwai <tiwai@xxxxxxx> wrote:
> > >
> > > On Mon, 27 Apr 2020 20:28:12 +0200,
> > > Alex Deucher wrote:
> > > >
> > > > On Mon, Apr 27, 2020 at 2:07 PM Nicholas Johnson
> > > > <nicholas.johnson-opensource@xxxxxxxxxxxxxx> wrote:
> > > > >
> > > > > On Mon, Apr 27, 2020 at 05:15:55PM +0200, Takashi Iwai wrote:
> > > > > > On Mon, 27 Apr 2020 16:22:21 +0200,
> > > > > > Deucher, Alexander wrote:
> > > > > > >
> > > > > > > [AMD Public Use]
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Nicholas Johnson <nicholas.johnson-opensource@xxxxxxxxxxxxxx>
> > > > > > > > Sent: Sunday, April 26, 2020 12:02 PM
> > > > > > > > To: linux-kernel@xxxxxxxxxxxxxxx
> > > > > > > > Cc: Deucher, Alexander <Alexander.Deucher@xxxxxxx>; Koenig, Christian
> > > > > > > > <Christian.Koenig@xxxxxxx>; Zhou, David(ChunMing)
> > > > > > > > <David1.Zhou@xxxxxxx>; Nicholas Johnson <nicholas.johnson-
> > > > > > > > opensource@xxxxxxxxxxxxxx>
> > > > > > > > Subject: [PATCH 0/1] Fiji GPU audio register timeout when in BACO state
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Since Linux v5.7-rc1 / commit 4fdda2e66de0 ("drm/amdgpu/runpm: enable
> > > > > > > > runpm on baco capable VI+ asics"), my AMD R9 Nano has been using runpm /
> > > > > > > > BACO. You can tell visually when it sleeps, because the fan on the graphics
> > > > > > > > card is switched off to save power. It did not spin down the fan in v5.6.x.
> > > > > > > >
> > > > > > > > This is great (I love it), except that when it is sleeping, the PCIe audio function
> > > > > > > > of the GPU has issues if anything tries to access it. You get dmesg errors such
> > > > > > > > as these:
> > > > > > > >
> > > > > > > > snd_hda_intel 0000:08:00.1: spurious response 0x0:0x0, last cmd=0x170500
> > > > > > > > snd_hda_intel 0000:08:00.1: azx_get_response timeout, switching to polling
> > > > > > > > mode: last cmd=0x001f0500 snd_hda_intel 0000:08:00.1: No response from
> > > > > > > > codec, disabling MSI: last cmd=0x001f0500 snd_hda_intel 0000:08:00.1: No
> > > > > > > > response from codec, resetting bus: last cmd=0x001f0500
> > > > > > > > snd_hda_codec_hdmi hdaudioC1D0: Unable to sync register 0x2f0d00. -11
> > > > > > > >
> > > > > > > > The above is with the Fiji XT GPU at 0000:08:00.0 in a Thunderbolt enclosure
> > > > > > > > (not that Thunderbolt should affect it, but I feel I should mention it just in
> > > > > > > > case). I dropped a lot of duplicate dmesg lines, as some of them repeated a
> > > > > > > > lot of times before the driver gave up.
> > > > > > > >
> > > > > > > > I offer this patch to disable runpm for Fiji while a fix is found, if you decide
> > > > > > > > that is the best approach. Regardless, I will gladly test any patches you come
> > > > > > > > up with instead and confirm that the above issue has been fixed.
> > > > > > > >
> > > > > > > > I cannot tell if any other GPUs are affected. The only other cards to which I
> > > > > > > > have access are a couple of AMD R9 280X (Tahiti XT), which use radeon driver
> > > > > > > > instead of amdgpu driver.
> > > > > > >
> > > > > > > Adding a few more people.  Do you know what is accessing the audio?  The audio should have a dependency on the GPU device.  The GPU won't enter runtime pm until the audio has entered runtime pm and vice versa on resume. Please attach a copy of your dmesg output and lspci output.
> > > > >
> > > > > pci 0000:08:00.1: D0 power state depends on 0000:08:00.0
> > > > > The above must be the dependency of which you speak from dmesg.
> > > > >
> > > > > Accessing the audio? I did not have a single method for triggering it.
> > > > > Sometimes it happened on shutdown. Sometimes when restarting gdm.
> > > > > Sometimes when playing with audio settings in Cinnamon Desktop. But most
> > > > > often when changing displays. It might have something to do with the
> > > > > audio device associated with a monitor being created when the monitor is
> > > > > found. If an audio device is created, then pulseaudio might touch it.
> > > > > Sorry, this is a very verbose "not quite sure".
> > > > >
> > > > > To trigger the bug, this time I did the following:
> > > > >
> > > > > 1. Boot laptop without Fiji and log in
> > > > >
> > > > > 2. Attach Fiji via Thunderbolt (no displays attached to Fiji) and
> > > > > approve Thunderbolt device
> > > > >
> > > > > 3. Log in again because the session gets killed when GPU is hot-added
> > > > >
> > > > > 4. Wait for Fiji to fall asleep (fan stops)
> > > > >
> > > > > 5. Open "dmesg -w" on laptop display
> > > > >
> > > > > 6. Attach display to DisplayPort on Fiji (it should still stay asleep)
> > > > >
> > > > > 7. Do WindowsKey+P to activate external display. The error appears in
> > > > > dmesg window that instant.
> > > > >
> > > > > Could it be a race condition when waking the card up?
> > > > >
> > > > > I cannot get the graphics card fan to spin down if the Thunderbolt
> > > > > enclosure is attached at boot time. It only does it if hot-added.
> > > > >
> > > > > If you think it will help, I can take out the Fiji and put it in a test
> > > > > rig and try to replicate the issue without Thunderbolt, but it looks
> > > > > like it will not spin the fan down if Fiji is attached at boot time.
> > > > >
> > > > > Question, why would the fan not spin down if Fiji is attached at boot
> > > > > time, and how would one make the said fan turn off? Aside from being
> > > > > useful for pinning down the audio register issue, I would like to make
> > > > > sure the power savings are realised whenever the GPU is not being used.
> > > >
> > > > Presumably something is using the device.  Maybe a framebuffer console
> > > > or X?  Or maybe the something like tlp has disabled runtime pm on your
> > > > device?  You can see the current status by reading the files in
> > > > /sys/class/drm/cardX/device/power/ .  Replace cardX with card0, card1,
> > > > etc. depending on which device is the radeon card.
I had card1 = Fiji stuck awake and card2 = Fiji asleep (both in separate 
Thunderbolt enclosures).

The sysfs values in /sys/class/drm/card{0,1}/device/power/ were the 
same.

The powertop utility did not help in "tunables" tab.

I compiled kernel without fbcon and it still did it.

But moving from Arch to Ubuntu changed the behaviour. I am still 
investigating.

> > > >
> > > > FWIW, I have a fiji board in a desktop system and it worked fine when
> > > > this code was enabled.
> > >
> > > Is the new DC code used for Fiji boards?  IIRC, the audio component
> > > binding from amdgpu is enabled only for DC, and without the audio
> > > component binding the runtime PM won't be linked up, hence you can't
> > > power up GPU from the audio side access automatically.
> > >
> > 
> > Yes, DC is enabled by default for all cards with runtime pm enabled.
> 
> OK, thanks, I found that amdgpu got bound via component in the dmesg
> output, too:
> 
> [   21.294927] snd_hda_intel 0000:08:00.1: bound 0000:08:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
> 
> This is the place soon after amdgpu driver gets initialized.
> Then we see later another initialization phase:
> 
> [   26.904127] rfkill: input handler enabled
> [   37.264152] [drm] PCIE GART of 1024M enabled (table at 0x000000F400000000).
> 
> here shows 10 seconds between them.  Then, it complained something:
> 
> 
> [   37.363287] [drm] UVD initialized successfully.
> [   37.473340] [drm] VCE initialized successfully.
> [   37.477942] amdgpu 0000:08:00.0: [drm] Cannot find any crtc or sizes

The above would be me hitting WindowsKey+P to change screens, but with 
no DisplayPort attached to Fiji, hence it unable to find crtc.

> 
> ... and go further, and hitting HD-audio error:
> 
That would be me having attached the DisplayPort and done WindowsKey+P 
again.

> 
> [   38.936624] [drm] fb mappable at 0x4B0696000
> [   38.936626] [drm] vram apper at 0x4B0000000
> [   38.936626] [drm] size 33177600
> [   38.936627] [drm] fb depth is 24
> [   38.936627] [drm]    pitch is 15360
> [   38.936673] amdgpu 0000:08:00.0: fb1: amdgpudrmfb frame buffer device
> [   40.092223] snd_hda_intel 0000:08:00.1: azx_get_response timeout, switching to polling mode: last cmd=0x00170500
> 
> After this point, HD-audio communication was screwed up.
> 
> This lastcmd in the above message is AC_SET_POWER_STATE verb for the
> root node to D0, so the very first command to power up the codec. 
> The rest commands are also about the power up of each node, so the
> whole error indicate that the power up at runtime resume failed.
> 
> So, this looks to me as if the device gets runtime-resumed at the bad
> moment?
It does. However, this is not going to be easy to pin down.

I moved from Arch to Ubuntu, and it behaves differently. I cannot 
trigger the bug in Ubuntu. Plus, it puts the GPUs asleep, even if 
attached at boot, unlike Arch. I will continue to try to trigger it. But 
even if this is a problem with the Linux distribution, it should not be 
able to trigger a kernel mode bug, so we should persist with finding it.

Regards,
Nicholas

> 
> 
> thanks,
> 
> Takashi