Re: [RFC]Add new mdev interface for QoS

[Gao, Ping A]

On 2017/8/4 5:11, Alex Williamson wrote:
> On Thu, 3 Aug 2017 20:26:14 +0800
> "Gao, Ping A" <ping.a.gao@xxxxxxxxx> wrote:
>
>> On 2017/8/3 0:58, Alex Williamson wrote:
>>> On Wed, 2 Aug 2017 21:16:28 +0530
>>> Kirti Wankhede <kwankhede@xxxxxxxxxx> wrote:
>>>  
>>>> On 8/2/2017 6:29 PM, Gao, Ping A wrote:  
>>>>> On 2017/8/2 18:19, Kirti Wankhede wrote:    
>>>>>> On 8/2/2017 3:56 AM, Alex Williamson wrote:    
>>>>>>> On Tue, 1 Aug 2017 13:54:27 +0800
>>>>>>> "Gao, Ping A" <ping.a.gao@xxxxxxxxx> wrote:
>>>>>>>    
>>>>>>>> On 2017/7/28 0:00, Gao, Ping A wrote:    
>>>>>>>>> On 2017/7/27 0:43, Alex Williamson wrote:      
>>>>>>>>>> [cc +libvir-list]
>>>>>>>>>>
>>>>>>>>>> On Wed, 26 Jul 2017 21:16:59 +0800
>>>>>>>>>> "Gao, Ping A" <ping.a.gao@xxxxxxxxx> wrote:
>>>>>>>>>>      
>>>>>>>>>>> The vfio-mdev provide the capability to let different guest share the
>>>>>>>>>>> same physical device through mediate sharing, as result it bring a
>>>>>>>>>>> requirement about how to control the device sharing, we need a QoS
>>>>>>>>>>> related interface for mdev to management virtual device resource.
>>>>>>>>>>>
>>>>>>>>>>> E.g. In practical use, vGPUs assigned to different quests almost has
>>>>>>>>>>> different performance requirements, some guests may need higher priority
>>>>>>>>>>> for real time usage, some other may need more portion of the GPU
>>>>>>>>>>> resource to get higher 3D performance, corresponding we can define some
>>>>>>>>>>> interfaces like weight/cap for overall budget control, priority for
>>>>>>>>>>> single submission control.
>>>>>>>>>>>
>>>>>>>>>>> So I suggest to add some common attributes which are vendor agnostic in
>>>>>>>>>>> mdev core sysfs for QoS purpose.      
>>>>>>>>>> I think what you're asking for is just some standardization of a QoS
>>>>>>>>>> attribute_group which a vendor can optionally include within the
>>>>>>>>>> existing mdev_parent_ops.mdev_attr_groups.  The mdev core will
>>>>>>>>>> transparently enable this, but it really only provides the standard,
>>>>>>>>>> all of the support code is left for the vendor.  I'm fine with that,
>>>>>>>>>> but of course the trouble with and sort of standardization is arriving
>>>>>>>>>> at an agreed upon standard.  Are there QoS knobs that are generic
>>>>>>>>>> across any mdev device type?  Are there others that are more specific
>>>>>>>>>> to vGPU?  Are there existing examples of this that we can steal their
>>>>>>>>>> specification?      
>>>>>>>>> Yes, you are right, standardization QoS knobs are exactly what I wanted.
>>>>>>>>> Only when it become a part of the mdev framework and libvirt, then QoS
>>>>>>>>> such critical feature can be leveraged by cloud usage. HW vendor only
>>>>>>>>> need to focus on the implementation of the corresponding QoS algorithm
>>>>>>>>> in their back-end driver.
>>>>>>>>>
>>>>>>>>> Vfio-mdev framework provide the capability to share the device that lack
>>>>>>>>> of HW virtualization support to guests, no matter the device type,
>>>>>>>>> mediated sharing actually is a time sharing multiplex method, from this
>>>>>>>>> point of view, QoS can be take as a generic way about how to control the
>>>>>>>>> time assignment for virtual mdev device that occupy HW. As result we can
>>>>>>>>> define QoS knob generic across any device type by this way. Even if HW
>>>>>>>>> has build in with some kind of QoS support, I think it's not a problem
>>>>>>>>> for back-end driver to convert mdev standard QoS definition to their
>>>>>>>>> specification to reach the same performance expectation. Seems there are
>>>>>>>>> no examples for us to follow, we need define it from scratch.
>>>>>>>>>
>>>>>>>>> I proposal universal QoS control interfaces like below:
>>>>>>>>>
>>>>>>>>> Cap: The cap limits the maximum percentage of time a mdev device can own
>>>>>>>>> physical device. e.g. cap=60, means mdev device cannot take over 60% of
>>>>>>>>> total physical resource.
>>>>>>>>>
>>>>>>>>> Weight: The weight define proportional control of the mdev device
>>>>>>>>> resource between guests, itâs orthogonal with Cap, to target load
>>>>>>>>> balancing. E.g. if guest 1 should take double mdev device resource
>>>>>>>>> compare with guest 2, need set weight ratio to 2:1.
>>>>>>>>>
>>>>>>>>> Priority: The guest who has higher priority will get execution first,
>>>>>>>>> target to some real time usage and speeding interactive response.
>>>>>>>>>
>>>>>>>>> Above QoS interfaces cover both overall budget control and single
>>>>>>>>> submission control. I will sent out detail design later once get aligned.      
>>>>>>>> Hi Alex,
>>>>>>>> Any comments about the interface mentioned above?    
>>>>>>> Not really.
>>>>>>>
>>>>>>> Kirti, are there any QoS knobs that would be interesting
>>>>>>> for NVIDIA devices?
>>>>>>>    
>>>>>> We have different types of vGPU for different QoS factors.
>>>>>>
>>>>>> When mdev devices are created, its resources are allocated irrespective
>>>>>> of which VM/userspace app is going to use that mdev device. Any
>>>>>> parameter we add here should be tied to particular mdev device and not
>>>>>> to the guest/app that are going to use it. 'Cap' and 'Priority' are
>>>>>> along that line. All mdev device might not need/use these parameters,
>>>>>> these can be made optional interfaces.    
>>>>> We also define some QoS parameters in Intel vGPU types, but it only
>>>>> provided a default fool-style way. We still need a flexible approach
>>>>> that give user the ability to change QoS parameters freely and
>>>>> dynamically according to their requirement , not restrict to the current
>>>>> limited and static vGPU types.
>>>>>     
>>>>>> In the above proposal, I'm not sure how 'Weight' would work for mdev
>>>>>> devices on same physical device.
>>>>>>
>>>>>> In the above example, "if guest 1 should take double mdev device
>>>>>> resource compare with guest 2" but what if guest 2 never booted, how
>>>>>> will you calculate resources?    
>>>>> Cap is try to limit the max physical GPU resource for vGPU, it's a
>>>>> vertical limitation, but weight is a horizontal limitation that define
>>>>> the GPU resource consumption ratio between vGPUs. Cap is easy to
>>>>> understand as it's just a percentage. For weight. for example, if we
>>>>> define the max weight is 16, the vGPU_1 who get weight 8 should been
>>>>> assigned double GPU resources compared to the vGPU_2 whose weight is 4,
>>>>> we can translate it to this formula:  resource_of_vGPU_1 = 8 / (8+4) *
>>>>> total_physical_GPU_resource.
>>>>>     
>>>> How will vendor driver provide max weight to userspace
>>>> application/libvirt? Max weight will be per physical device, right?
>>>>
>>>> How would such resource allocation reflect in 'available_instances'?
>>>> Suppose in above example, vGPU_1 is of 1G FB with weight 8, vGPU_2 with
>>>> 1G FB with weight 4 and vGPU_3 with 1G FB with weight 4. Now you have 1G
>>>> FB free but you have reached max weight, so will you make
>>>> available_instances = 0 for all types on that physical GPU?  
>>> No, per the algorithm above, the available scheduling for the remaining
>>> mdev device is N / (8 + 4 + 4 + N), where N is 1-16 (or maybe 0-16,
>>> we'd need to define or make the range discoverable, 16 seems rather
>>> arbitrary).  We can always add new scheduling participants.  AIUI,
>>> Intel uses round-robin scheduling now, where you could consider all
>>> mdev devices to have the same weight.  Whether we consider that to be a
>>> weight of 16 or zero or 8 doesn't really matter.  
>> QoS is to control the device's process capability like GPU
>> rendering/computing that can be time multiplexing, not used to control
>> the dedicated partition resources like FB, so there is no impact on
>> 'available_instances'.
>>
>> if vGPU_1 weight=8, vGPU_2 weight=4;
>> then vGPU_1_res = 8 / (8 + 4) * total,  vGPU_2_res = 4 / (8 + 4) * total;
>> if vGPU_3 created with weight 2;
>> then vGPU_1_res = 8 /(8 + 4 + 2) * total, vGPU_2_res = 4 / (8 + 4 + 2) *
>> total, vGPU_3_res = 2 / (8 + 4 + 2) * total.
>>
>> The resource allocation of vGPU_1 and vGPU_2 have been dynamically
>> changed after vGPU_3 creating, that's weight doing as it's to define the
>> relationship of all the vGPUs, the performance degradation is meet
>> expectation. The end-user should know about such behavior.
>>
>> However the argument on weight let me has some self-reflection, does the
>> end-user real need weight? does weight has actually application
>> requirement?  Maybe the cap and priority are enough?
> What sort of SLAs do you want to be able to offer?  For instance if I
> want to be able to offer a GPU in 1/4 increments, how does that work?
> I might sell customers A & B 1/4 increment each and customer C a 1/2
> increment.  If weight is removed, can we do better than capping A & B
> at 25% each and C at 50%?  That has the downside that nobody gets to
> use the unused capacity of the other clients.  The SLA is some sort of
> "up to X% (and no more)" model.  With weighting it's as simple as making
> sure customer C's vGPU has twice the weight of that given to A or B.
> Then you get an "at least X%" SLA model and any customer can use up to
> 100% if the others are idle.  Combining weight and cap, we can do "at
> least X%, but no more than Y%".
>
> All of this feels really similar to how cpusets must work since we're
> just dealing with QoS relative to scheduling and we should not try to
> reinvent scheduling QoS.  Thanks,
>

Yeah, that's also my original thoughts.
Since we get aligned about the QoS basic definition, I'm going to
prepare the code in kernel side. How about the corresponding part in
libvirt? Implemented separately after the kernel interface finalizing?

Thanks,
Ping