Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
From: Yu Chen
Date: Wed Sep 06 2017 - 22:50:19 EST
On Wed, Sep 06, 2017 at 10:03:58AM +0200, Thomas Gleixner wrote:
> On Wed, 6 Sep 2017, Yu Chen wrote:
> > On Wed, Sep 06, 2017 at 12:57:41AM +0200, Thomas Gleixner wrote:
> > > I have a hard time to figure out how the 133 vectors on CPU31 are now
> > > magically fitting in the empty space on CPU0, which is 204 - 133 = 71. In
> > > my limited understanding of math 133 is greater than 71, but your patch
> > > might make that magically be wrong.
> > >
> > The problem is reproduced when the network cable is not plugged in,
> > because this driver looks like this:
> >
> > step 1. Reserved enough irq vectors and corresponding IRQs.
> > step 2. If the network is activated, invoke request_irq() to
> > register the handler.
> > step 3. Invoke set_affinity() to spread the IRQs onto different
> > CPUs, thus to spread the vectors too.
> >
> > Here's my understanding for why spreading vectors might help for this
> > special case:
> > As step 2 will not get invoked, the IRQs of this driver
> > has not been enabled, thus in migrate_one_irq() this IRQ
> > will not be considered because there is a check of
> > irqd_is_started(d), thus there should only be 8 vectors
> > allocated by this driver on CPU0, and 8 vectors left on
> > CPU31, and the 8 vectors on CPU31 will not be migrated
> > to CPU0 neither, so there is room for other 'valid' vectors
> > to be migrated to CPU0.
>
> Can you please spare me repeating your theories, as long as you don't have
> hard facts to back them up? The network cable is changing the symptoms,
> but the underlying root cause is definitely something different.
>
> > # cat /sys/kernel/debug/irq/domains/*
> > name: VECTOR
> > size: 0
> > mapped: 388
> > flags: 0x00000041
>
> So we have 388 vectors mapped in total. And those are just device vectors
> because system vectors are not accounted there.
>
> > name: IO-APIC-0
> > size: 24
> > mapped: 16
>
> That's the legacy space
>
> > name: IO-APIC-1
> > size: 8
> > mapped: 2
>
> > name: IO-APIC-2
> > size: 8
> > mapped: 0
>
> > name: IO-APIC-3
> > size: 8
> > mapped: 0
>
> > name: IO-APIC-4
> > size: 8
> > mapped: 5
>
> And a few GSIs: Total GSIs = 16 + 2 + 5 = 23
>
> > name: PCI-MSI-2
> > size: 0
> > mapped: 365
>
> Plus 365 PCI-MSI vectors allocated.
>
> > flags: 0x00000051
> > parent: VECTOR
> > name: VECTOR
> > size: 0
> > mapped: 388
>
> Which nicely sums up to 388
>
> > # ls /sys/kernel/debug/irq/irqs
> > ls /sys/kernel/debug/irq/irqs
> > 0 10 11 13 142 184 217 259 292 31 33 337 339
> > 340 342 344 346 348 350 352 354 356 358 360 362
> > 364 366 368 370 372 374 376 378 380 382 384 386
> > 388 390 392 394 4 6 7 9 1 109 12 14 15 2
> > 24 26 3 32 335 338 34 341 343 345 347 349
> > 351 353 355 357 359 361 363 365 367 369 371 373
> > 375 377 379 381 383 385 387 389 391 393 395 5
> > 67 8
>
> That are all interrupts which are active. That's a total of 89. Can you
> explain where the delta of 299 vectors comes from?
>
> 299 allocated, vector mapped, but unused interrupts?
>
> That's where your problem is, not in the vector spreading. You have a
> massive leak.
>
> > BTW, do we have sysfs to display how much vectors used on each CPUs?
>
> Not yet.
>
> Can you please apply the debug patch below, boot the machine and right
> after login provide the output of
>
> # cat /sys/kernel/debug/tracing/trace
>
Ok, I've tested on top of 4.13, here're the results:
# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
kworker/0:0-3 [000] .... 7.373461: msi_domain_alloc_irqs: dev: 0000:00:1c.0 nvec 1 virq 24
kworker/0:0-3 [000] .... 7.373666: msi_domain_alloc_irqs: dev: 0000:b2:00.0 nvec 1 virq 26
kworker/0:0-3 [000] .... 7.578980: msi_domain_alloc_irqs: dev: 0000:00:11.5 nvec 1 virq 31
kworker/0:0-3 [000] .... 7.676938: msi_domain_alloc_irqs: dev: 0000:00:17.0 nvec 1 virq 32
kworker/0:0-3 [000] .... 8.017664: msi_domain_alloc_irqs: dev: 0000:00:14.0 nvec 1 virq 33
kworker/0:2-303 [000] .... 9.135467: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 34
kworker/0:2-303 [000] .... 9.135476: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 35
kworker/0:2-303 [000] .... 9.135484: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 36
kworker/0:2-303 [000] .... 9.135492: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 37
kworker/0:2-303 [000] .... 9.135499: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 38
kworker/0:2-303 [000] .... 9.135508: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 39
kworker/0:2-303 [000] .... 9.135516: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 40
kworker/0:2-303 [000] .... 9.135526: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 41
kworker/0:2-303 [000] .... 9.135534: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 42
kworker/0:2-303 [000] .... 9.135542: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 43
kworker/0:2-303 [000] .... 9.135550: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 44
kworker/0:2-303 [000] .... 9.135557: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 45
kworker/0:2-303 [000] .... 9.135564: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 46
kworker/0:2-303 [000] .... 9.135572: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 47
kworker/0:2-303 [000] .... 9.135579: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 48
kworker/0:2-303 [000] .... 9.135587: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 49
kworker/0:2-303 [000] .... 9.135594: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 50
kworker/0:2-303 [000] .... 9.135601: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 51
kworker/0:2-303 [000] .... 9.135608: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 52
kworker/0:2-303 [000] .... 9.135616: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 53
kworker/0:2-303 [000] .... 9.135623: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 54
kworker/0:2-303 [000] .... 9.135631: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 55
kworker/0:2-303 [000] .... 9.135638: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 56
kworker/0:2-303 [000] .... 9.135645: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 57
kworker/0:2-303 [000] .... 9.135653: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 58
kworker/0:2-303 [000] .... 9.135660: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 59
kworker/0:2-303 [000] .... 9.135668: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 60
kworker/0:2-303 [000] .... 9.135676: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 61
kworker/0:2-303 [000] .... 9.135683: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 62
kworker/0:2-303 [000] .... 9.135691: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 63
kworker/0:2-303 [000] .... 9.135699: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 64
kworker/0:2-303 [000] .... 9.135707: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 65
kworker/0:2-303 [000] .... 9.135714: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 66
kworker/0:2-303 [000] .... 9.135721: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 67
kworker/0:2-303 [000] .... 9.135729: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 68
kworker/0:2-303 [000] .... 9.135736: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 69
kworker/0:2-303 [000] .... 9.135744: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 70
kworker/0:2-303 [000] .... 9.135751: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 71
kworker/0:2-303 [000] .... 9.135758: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 72
kworker/0:2-303 [000] .... 9.135766: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 73
kworker/0:2-303 [000] .... 9.135773: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 74
kworker/0:2-303 [000] .... 9.135780: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 75
kworker/0:2-303 [000] .... 9.135788: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 76
kworker/0:2-303 [000] .... 9.135795: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 77
kworker/0:2-303 [000] .... 9.135803: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 78
kworker/0:2-303 [000] .... 9.135810: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 79
kworker/0:2-303 [000] .... 9.135818: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 80
kworker/0:2-303 [000] .... 9.135826: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 81
kworker/0:2-303 [000] .... 9.135833: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 82
kworker/0:2-303 [000] .... 9.135841: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 83
kworker/0:2-303 [000] .... 9.135848: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 84
kworker/0:2-303 [000] .... 9.135856: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 85
kworker/0:2-303 [000] .... 9.135863: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 86
kworker/0:2-303 [000] .... 9.135871: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 87
kworker/0:2-303 [000] .... 9.135878: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 88
kworker/0:2-303 [000] .... 9.135885: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 89
kworker/0:2-303 [000] .... 9.135893: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 90
kworker/0:2-303 [000] .... 9.135900: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 91
kworker/0:2-303 [000] .... 9.135908: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 92
kworker/0:2-303 [000] .... 9.135915: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 93
kworker/0:2-303 [000] .... 9.135923: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 94
kworker/0:2-303 [000] .... 9.135930: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 95
kworker/0:2-303 [000] .... 9.135937: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 96
kworker/0:2-303 [000] .... 9.135944: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 97
kworker/0:2-303 [000] .... 9.135951: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 98
kworker/0:2-303 [000] .... 9.135958: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 99
kworker/0:2-303 [000] .... 9.135964: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 100
kworker/0:2-303 [000] .... 9.135971: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 101
kworker/0:2-303 [000] .... 9.135978: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 102
kworker/0:2-303 [000] .... 9.135985: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 103
kworker/0:2-303 [000] .... 9.135991: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 104
kworker/0:2-303 [000] .... 9.135998: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 105
kworker/0:2-303 [000] .... 9.136005: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 106
kworker/0:2-303 [000] .... 9.136012: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 107
kworker/0:2-303 [000] .... 9.136019: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 108
kworker/0:2-303 [000] .... 9.599173: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 109
kworker/0:2-303 [000] .... 9.599181: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 110
kworker/0:2-303 [000] .... 9.599189: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 111
kworker/0:2-303 [000] .... 9.599196: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 112
kworker/0:2-303 [000] .... 9.599203: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 113
kworker/0:2-303 [000] .... 9.599210: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 114
kworker/0:2-303 [000] .... 9.599217: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 115
kworker/0:2-303 [000] .... 9.599224: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 116
kworker/0:2-303 [000] .... 9.599231: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 117
kworker/0:2-303 [000] .... 9.599239: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 118
kworker/0:2-303 [000] .... 9.599246: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 119
kworker/0:2-303 [000] .... 9.599253: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 120
kworker/0:2-303 [000] .... 9.599259: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 121
kworker/0:2-303 [000] .... 9.599266: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 122
kworker/0:2-303 [000] .... 9.599273: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 123
kworker/0:2-303 [000] .... 9.599279: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 124
kworker/0:2-303 [000] .... 9.599286: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 125
kworker/0:2-303 [000] .... 9.599293: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 126
kworker/0:2-303 [000] .... 9.599300: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 127
kworker/0:2-303 [000] .... 9.599307: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 128
kworker/0:2-303 [000] .... 9.599314: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 129
kworker/0:2-303 [000] .... 9.599321: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 130
kworker/0:2-303 [000] .... 9.599327: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 131
kworker/0:2-303 [000] .... 9.599334: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 132
kworker/0:2-303 [000] .... 9.599341: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 133
kworker/0:2-303 [000] .... 9.599349: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 134
kworker/0:2-303 [000] .... 9.599356: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 135
kworker/0:2-303 [000] .... 9.599363: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 136
kworker/0:2-303 [000] .... 9.599369: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 137
kworker/0:2-303 [000] .... 9.599376: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 138
kworker/0:2-303 [000] .... 9.599382: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 139
kworker/0:2-303 [000] .... 9.599389: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 140
kworker/0:2-303 [000] .... 9.599395: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 141
kworker/0:2-303 [000] .... 9.599402: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 142
kworker/0:2-303 [000] .... 9.599409: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 143
kworker/0:2-303 [000] .... 9.599416: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 144
kworker/0:2-303 [000] .... 9.599422: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 145
kworker/0:2-303 [000] .... 9.599428: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 146
kworker/0:2-303 [000] .... 9.599437: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 147
kworker/0:2-303 [000] .... 9.599444: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 148
kworker/0:2-303 [000] .... 9.599450: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 149
kworker/0:2-303 [000] .... 9.599456: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 150
kworker/0:2-303 [000] .... 9.599463: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 151
kworker/0:2-303 [000] .... 9.599470: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 152
kworker/0:2-303 [000] .... 9.599476: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 153
kworker/0:2-303 [000] .... 9.599482: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 154
kworker/0:2-303 [000] .... 9.599488: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 155
kworker/0:2-303 [000] .... 9.599494: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 156
kworker/0:2-303 [000] .... 9.599501: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 157
kworker/0:2-303 [000] .... 9.599507: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 158
kworker/0:2-303 [000] .... 9.599513: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 159
kworker/0:2-303 [000] .... 9.599519: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 160
kworker/0:2-303 [000] .... 9.599527: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 161
kworker/0:2-303 [000] .... 9.599534: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 162
kworker/0:2-303 [000] .... 9.599541: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 163
kworker/0:2-303 [000] .... 9.599547: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 164
kworker/0:2-303 [000] .... 9.599553: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 165
kworker/0:2-303 [000] .... 9.599560: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 166
kworker/0:2-303 [000] .... 9.599567: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 167
kworker/0:2-303 [000] .... 9.599573: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 168
kworker/0:2-303 [000] .... 9.599579: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 169
kworker/0:2-303 [000] .... 9.599585: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 170
kworker/0:2-303 [000] .... 9.599591: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 171
kworker/0:2-303 [000] .... 9.599597: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 172
kworker/0:2-303 [000] .... 9.599604: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 173
kworker/0:2-303 [000] .... 9.599610: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 174
kworker/0:2-303 [000] .... 9.599616: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 175
kworker/0:2-303 [000] .... 9.599622: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 176
kworker/0:2-303 [000] .... 9.599628: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 177
kworker/0:2-303 [000] .... 9.599634: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 178
kworker/0:2-303 [000] .... 9.599640: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 179
kworker/0:2-303 [000] .... 9.599646: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 180
kworker/0:2-303 [000] .... 9.599653: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 181
kworker/0:2-303 [000] .... 9.599660: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 182
kworker/0:2-303 [000] .... 9.599666: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 183
kworker/0:2-303 [000] .... 9.699902: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 184
kworker/0:2-303 [000] .... 9.699910: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 185
kworker/0:2-303 [000] .... 9.699917: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 186
kworker/0:2-303 [000] .... 9.699924: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 187
kworker/0:2-303 [000] .... 9.699931: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 188
kworker/0:2-303 [000] .... 9.699939: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 189
kworker/0:2-303 [000] .... 9.699946: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 190
kworker/0:2-303 [000] .... 9.699953: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 191
kworker/0:2-303 [000] .... 9.699960: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 192
kworker/0:2-303 [000] .... 9.699967: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 193
kworker/0:2-303 [000] .... 9.699975: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 194
kworker/0:2-303 [000] .... 9.699983: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 195
kworker/0:2-303 [000] .... 9.699989: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 196
kworker/0:2-303 [000] .... 9.699996: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 197
kworker/0:2-303 [000] .... 9.700003: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 198
kworker/0:2-303 [000] .... 9.700012: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 199
kworker/0:2-303 [000] .... 9.700019: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 200
kworker/0:2-303 [000] .... 9.700026: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 201
kworker/0:2-303 [000] .... 9.700032: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 202
kworker/0:2-303 [000] .... 9.700039: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 203
kworker/0:2-303 [000] .... 9.700047: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 204
kworker/0:2-303 [000] .... 9.700053: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 205
kworker/0:2-303 [000] .... 9.700060: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 206
kworker/0:2-303 [000] .... 9.700067: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 207
kworker/0:2-303 [000] .... 9.700074: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 208
kworker/0:2-303 [000] .... 9.700081: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 209
kworker/0:2-303 [000] .... 9.700088: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 210
kworker/0:2-303 [000] .... 9.700095: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 211
kworker/0:2-303 [000] .... 9.700103: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 212
kworker/0:2-303 [000] .... 9.700112: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 213
kworker/0:2-303 [000] .... 9.700121: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 214
kworker/0:2-303 [000] .... 9.700129: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 215
kworker/0:2-303 [000] .... 9.700137: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 216
kworker/0:2-303 [000] .... 9.700145: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 217
kworker/0:2-303 [000] .... 9.700154: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 218
kworker/0:2-303 [000] .... 9.700163: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 219
kworker/0:2-303 [000] .... 9.700172: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 220
kworker/0:2-303 [000] .... 9.700180: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 221
kworker/0:2-303 [000] .... 9.700189: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 222
kworker/0:2-303 [000] .... 9.700200: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 223
kworker/0:2-303 [000] .... 9.700209: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 224
kworker/0:2-303 [000] .... 9.700217: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 225
kworker/0:2-303 [000] .... 9.700225: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 226
kworker/0:2-303 [000] .... 9.700235: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 227
kworker/0:2-303 [000] .... 9.700244: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 228
kworker/0:2-303 [000] .... 9.700253: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 229
kworker/0:2-303 [000] .... 9.700261: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 230
kworker/0:2-303 [000] .... 9.700269: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 231
kworker/0:2-303 [000] .... 9.700278: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 232
kworker/0:2-303 [000] .... 9.700287: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 233
kworker/0:2-303 [000] .... 9.700295: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 234
kworker/0:2-303 [000] .... 9.700304: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 235
kworker/0:2-303 [000] .... 9.700313: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 236
kworker/0:2-303 [000] .... 9.700321: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 237
kworker/0:2-303 [000] .... 9.700330: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 238
kworker/0:2-303 [000] .... 9.700339: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 239
kworker/0:2-303 [000] .... 9.700347: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 240
kworker/0:2-303 [000] .... 9.700355: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 241
kworker/0:2-303 [000] .... 9.700363: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 242
kworker/0:2-303 [000] .... 9.700372: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 243
kworker/0:2-303 [000] .... 9.700380: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 244
kworker/0:2-303 [000] .... 9.700390: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 245
kworker/0:2-303 [000] .... 9.700398: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 246
kworker/0:2-303 [000] .... 9.700406: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 247
kworker/0:2-303 [000] .... 9.700415: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 248
kworker/0:2-303 [000] .... 9.700424: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 249
kworker/0:2-303 [000] .... 9.700432: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 250
kworker/0:2-303 [000] .... 9.700440: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 251
kworker/0:2-303 [000] .... 9.700449: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 252
kworker/0:2-303 [000] .... 9.700458: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 253
kworker/0:2-303 [000] .... 9.700466: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 254
kworker/0:2-303 [000] .... 9.700474: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 255
kworker/0:2-303 [000] .... 9.700483: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 256
kworker/0:2-303 [000] .... 9.700492: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 257
kworker/0:2-303 [000] .... 9.700500: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 258
kworker/0:2-303 [000] .... 9.761556: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 259
kworker/0:2-303 [000] .... 9.761570: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 260
kworker/0:2-303 [000] .... 9.761583: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 261
kworker/0:2-303 [000] .... 9.761595: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 262
kworker/0:2-303 [000] .... 9.761608: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 263
kworker/0:2-303 [000] .... 9.761619: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 264
kworker/0:2-303 [000] .... 9.761629: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 265
kworker/0:2-303 [000] .... 9.761639: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 266
kworker/0:2-303 [000] .... 9.761649: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 267
kworker/0:2-303 [000] .... 9.761658: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 268
kworker/0:2-303 [000] .... 9.761669: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 269
kworker/0:2-303 [000] .... 9.761678: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 270
kworker/0:2-303 [000] .... 9.761688: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 271
kworker/0:2-303 [000] .... 9.761697: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 272
kworker/0:2-303 [000] .... 9.761707: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 273
kworker/0:2-303 [000] .... 9.761717: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 274
kworker/0:2-303 [000] .... 9.761726: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 275
kworker/0:2-303 [000] .... 9.761736: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 276
kworker/0:2-303 [000] .... 9.761745: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 277
kworker/0:2-303 [000] .... 9.761756: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 278
kworker/0:2-303 [000] .... 9.761765: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 279
kworker/0:2-303 [000] .... 9.761775: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 280
kworker/0:2-303 [000] .... 9.761784: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 281
kworker/0:2-303 [000] .... 9.761793: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 282
kworker/0:2-303 [000] .... 9.761803: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 283
kworker/0:2-303 [000] .... 9.761812: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 284
kworker/0:2-303 [000] .... 9.761822: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 285
kworker/0:2-303 [000] .... 9.761831: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 286
kworker/0:2-303 [000] .... 9.761840: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 287
kworker/0:2-303 [000] .... 9.761854: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 288
kworker/0:2-303 [000] .... 9.761863: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 289
kworker/0:2-303 [000] .... 9.761872: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 290
kworker/0:2-303 [000] .... 9.761881: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 291
kworker/0:2-303 [000] .... 9.761892: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 292
kworker/0:2-303 [000] .... 9.761903: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 293
kworker/0:2-303 [000] .... 9.761913: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 294
kworker/0:2-303 [000] .... 9.761923: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 295
kworker/0:2-303 [000] .... 9.761932: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 296
kworker/0:2-303 [000] .... 9.761941: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 297
kworker/0:2-303 [000] .... 9.761951: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 298
kworker/0:2-303 [000] .... 9.761961: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 299
kworker/0:2-303 [000] .... 9.761970: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 300
kworker/0:2-303 [000] .... 9.761979: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 301
kworker/0:2-303 [000] .... 9.761988: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 302
kworker/0:2-303 [000] .... 9.761998: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 303
kworker/0:2-303 [000] .... 9.762008: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 304
kworker/0:2-303 [000] .... 9.762017: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 305
kworker/0:2-303 [000] .... 9.762027: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 306
kworker/0:2-303 [000] .... 9.762037: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 307
kworker/0:2-303 [000] .... 9.762046: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 308
kworker/0:2-303 [000] .... 9.762055: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 309
kworker/0:2-303 [000] .... 9.762065: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 310
kworker/0:2-303 [000] .... 9.762074: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 311
kworker/0:2-303 [000] .... 9.762085: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 312
kworker/0:2-303 [000] .... 9.762094: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 313
kworker/0:2-303 [000] .... 9.762104: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 314
kworker/0:2-303 [000] .... 9.762113: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 315
kworker/0:2-303 [000] .... 9.762122: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 316
kworker/0:2-303 [000] .... 9.762133: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 317
kworker/0:2-303 [000] .... 9.762142: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 318
kworker/0:2-303 [000] .... 9.762151: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 319
kworker/0:2-303 [000] .... 9.762162: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 320
kworker/0:2-303 [000] .... 9.762171: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 321
kworker/0:2-303 [000] .... 9.762182: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 322
kworker/0:2-303 [000] .... 9.762191: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 323
kworker/0:2-303 [000] .... 9.762200: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 324
kworker/0:2-303 [000] .... 9.762209: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 325
kworker/0:2-303 [000] .... 9.762220: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 326
kworker/0:2-303 [000] .... 9.762231: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 327
kworker/0:2-303 [000] .... 9.762240: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 328
kworker/0:2-303 [000] .... 9.762249: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 329
kworker/0:2-303 [000] .... 9.762258: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 330
kworker/0:2-303 [000] .... 9.762268: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 331
kworker/0:2-303 [000] .... 9.762278: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 332
kworker/0:2-303 [000] .... 9.762288: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 333
kworker/0:0-3 [000] .... 12.026100: msi_domain_alloc_irqs: dev: 0000:00:04.0 nvec 1 virq 335
kworker/0:0-3 [000] .... 12.143303: msi_domain_alloc_irqs: dev: 0000:00:04.1 nvec 1 virq 337
kworker/0:0-3 [000] .... 12.158686: msi_domain_alloc_irqs: dev: 0000:00:04.2 nvec 1 virq 338
kworker/0:0-3 [000] .... 12.168628: msi_domain_alloc_irqs: dev: 0000:00:04.3 nvec 1 virq 339
kworker/0:0-3 [000] .... 12.176980: msi_domain_alloc_irqs: dev: 0000:00:04.4 nvec 1 virq 340
kworker/0:0-3 [000] .... 12.185280: msi_domain_alloc_irqs: dev: 0000:00:04.5 nvec 1 virq 341
kworker/0:0-3 [000] .... 12.204748: msi_domain_alloc_irqs: dev: 0000:00:04.6 nvec 1 virq 342
kworker/0:0-3 [000] .... 12.214549: msi_domain_alloc_irqs: dev: 0000:00:04.7 nvec 1 virq 343
kworker/0:1-169 [000] .... 12.440642: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 344
kworker/0:1-169 [000] .... 12.440664: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 345
kworker/0:1-169 [000] .... 12.440685: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 346
kworker/0:1-169 [000] .... 12.440704: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 347
kworker/0:1-169 [000] .... 12.440724: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 348
kworker/0:1-169 [000] .... 12.440742: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 349
kworker/0:1-169 [000] .... 12.440761: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 350
kworker/0:1-169 [000] .... 12.440780: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 351
kworker/0:1-169 [000] .... 12.440802: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 352
kworker/0:1-169 [000] .... 12.440822: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 353
kworker/0:1-169 [000] .... 12.440841: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 354
kworker/0:1-169 [000] .... 12.440860: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 355
kworker/0:1-169 [000] .... 12.440878: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 356
kworker/0:1-169 [000] .... 12.440897: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 357
kworker/0:1-169 [000] .... 12.440917: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 358
kworker/0:1-169 [000] .... 12.440936: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 359
kworker/0:1-169 [000] .... 12.440954: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 360
kworker/0:1-169 [000] .... 13.251009: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 361
kworker/0:1-169 [000] .... 13.251033: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 362
kworker/0:1-169 [000] .... 13.251055: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 363
kworker/0:1-169 [000] .... 13.251076: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 364
kworker/0:1-169 [000] .... 13.251100: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 365
kworker/0:1-169 [000] .... 13.251120: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 366
kworker/0:1-169 [000] .... 13.251142: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 367
kworker/0:1-169 [000] .... 13.251161: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 368
kworker/0:1-169 [000] .... 13.251182: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 369
kworker/0:1-169 [000] .... 13.251204: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 370
kworker/0:1-169 [000] .... 13.251224: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 371
kworker/0:1-169 [000] .... 13.251263: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 372
kworker/0:1-169 [000] .... 13.251288: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 373
kworker/0:1-169 [000] .... 13.251308: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 374
kworker/0:1-169 [000] .... 13.251330: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 375
kworker/0:1-169 [000] .... 13.251349: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 376
kworker/0:1-169 [000] .... 13.251374: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 377
kworker/0:1-169 [000] .... 14.043690: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 378
kworker/0:1-169 [000] .... 14.043721: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 379
kworker/0:1-169 [000] .... 14.043745: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 380
kworker/0:1-169 [000] .... 14.043767: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 381
kworker/0:1-169 [000] .... 14.043793: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 382
kworker/0:1-169 [000] .... 14.043817: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 383
kworker/0:1-169 [000] .... 14.043840: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 384
kworker/0:1-169 [000] .... 14.043862: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 385
kworker/0:1-169 [000] .... 14.043884: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 386
kworker/0:1-169 [000] .... 14.043909: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 387
kworker/0:1-169 [000] .... 14.043932: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 388
kworker/0:1-169 [000] .... 14.043953: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 389
kworker/0:1-169 [000] .... 14.043975: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 390
kworker/0:1-169 [000] .... 14.043997: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 391
kworker/0:1-169 [000] .... 14.044021: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 392
kworker/0:1-169 [000] .... 14.044043: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 393
kworker/0:1-169 [000] .... 14.044065: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 394
kworker/0:1-169 [000] .... 5181.743530: msi_domain_alloc_irqs: dev: 0000:00:1f.6 nvec 1 virq 395
ip-2037 [029] .... 5182.263982: msi_domain_alloc_irqs: dev: 0000:00:1f.6 nvec 1 virq 395
ip-2037 [029] .... 5182.367473: msi_domain_alloc_irqs: dev: 0000:00:1f.6 nvec 1 virq 395
bb:00.[0-3] Ethernet controller: Intel Corporation Device 37d0 (rev 03)
-+-[0000:b2]-+-00.0-[b3-bc]----00.0-[b4-bc]--+-00.0-[b5-b6]----00.0
| | +-01.0-[b7-b8]----00.0
| | +-02.0-[b9-ba]----00.0
| | \-03.0-[bb-bc]--+-00.0
| | +-00.1
| | +-00.2
| | \-00.3
and they are using i40e driver, the vectors should be reserved by:
i40e_probe() ->
i40e_init_interrupt_scheme() ->
i40e_init_msix() ->
i40e_reserve_msix_vectors() ->
pci_enable_msix_range()
# ls /sys/kernel/debug/irq/irqs
0 10 11 13 142 184 217 259 292 31 33
337 339 340 342 344 346 348 350 352 354 356
358 360 362 364 366 368 370 372 374 376 378
380 382 384 386 388 390 392 394 4 6 7 9
1 109 12 14 15 2 24 26 3 32 335
338 34 341 343 345 347 349 351 353 355 357
359 361 363 365 367 369 371 373 375 377 379
381 383 385 387 389 391 393 395 5 67 8
# cat /sys/kernel/debug/irq/domains/*
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-0
size: 24
mapped: 16
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-1
size: 8
mapped: 2
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-2
size: 8
mapped: 0
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-3
size: 8
mapped: 0
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-4
size: 8
mapped: 5
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: PCI-HT
size: 0
mapped: 0
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: PCI-MSI-2
size: 0
mapped: 365
flags: 0x00000051
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
Thanks,
Yu
> Thanks,
>
> tglx
>
> 8<-------------------
> --- a/kernel/irq/msi.c
> +++ b/kernel/irq/msi.c
> @@ -372,6 +372,9 @@ int msi_domain_alloc_irqs(struct irq_dom
> return ret;
> }
>
> + trace_printk("dev: %s nvec %d virq %d\n",
> + dev_name(dev), desc->nvec_used, virq);
> +
> for (i = 0; i < desc->nvec_used; i++)
> irq_set_msi_desc_off(virq, i, desc);
> }
> @@ -419,6 +422,8 @@ void msi_domain_free_irqs(struct irq_dom
> * entry. If that's the case, don't do anything.
> */
> if (desc->irq) {
> + trace_printk("dev: %s nvec %d virq %d\n",
> + dev_name(dev), desc->nvec_used, desc->irq);
> irq_domain_free_irqs(desc->irq, desc->nvec_used);
> desc->irq = 0;
> }