Re: About add an A64FX cache control function into resctrl

[Reinette Chatre]

Hi Tan Shaopeng,

On 7/21/2021 1:10 AM, tan.shaopeng@xxxxxxxxxxx wrote:
> Hi Reinette,
>
> > On 7/7/2021 4:26 AM, tan.shaopeng@xxxxxxxxxxx wrote:
> > > > > Sorry, I have not explained A64FX's sector cache function well yet.
> > > > > I think I need explain this function from different perspective.
> > > >
> > > > You have explained the A64FX's sector cache function well. I have
> > > > also read both specs to understand it better. It appears to me that
> > > > you are not considering the resctrl architecture as part of your
> > > > solution but instead just forcing your architecture onto the resctrl
> > > > filesystem. For example, in resctrl the resource groups are not just
> > > > a directory structure but has significance in what is being
> > > > represented within the directory (a class of service). The files
> > > > within a resource group's directory build on that. From your side I
> > > > have not seen any effort in aligning the sector cache function with the
> > resctrl architecture but instead you are just changing resctrl interface to match
> > the A64FX architecture.
> > > > Could you please take a moment to understand what resctrl is and how
> > > > it could be mapped to A64FX in a coherent way?
> > >
> > > Previously, my idea is based on how to make instructions use different
> > > sectors in one task. After I studied resctrl, to utilize resctrl
> > > architecture on A64FX, I think it’s better to assign one sector to one
> > > task. Thanks for your idea that "sectors" could be considered the same
> > > as the resctrl "classes of service".
> > >
> > > Based on your idea, I am considering the implementation details.
> > > In this email, I will explain the outline of new proposal, and then
> > > please allow me to confirm a few technologies about resctrl.
> > >
> > > The outline of my proposal is as follows.
> > > - Add a sector function equivalent to Intel's CAT function into resctrl.
> > >     (divide shared L2 cache into multiple partitions for multiple cores
> > > use)
> > > - Allocate one sector to one resource group (one CLOSID). Since one
> > >     core can only be assigned to one resource group, on A64FX each core
> > >     only uses one sector at a time.
> >
> > ok, so a sector is a portion of cache and matches with what can be represented
> > with a resource group.
> >
> > The second part of your comment is not clear to me. In the first part you
> > mention: "one core can only be assigned to one resource group" - this seems to
> > indicate some static assignment between cores and sectors and if this is the
>
> Sorry, does "static assignment between cores and sectors" mean
> each core always use a fixed sector id? For example, core 0 always
> use sector 0 at any case. It is not.
>
> > case this needs more thinking since the current implementation assumes that
> > any core that can access the cache can access all resource groups associated
> > with that cache. On the other hand, you mention "on A64FX each core only uses
> > one sector at a time" - this now sounds dynamic and is how resctrl works since
> > the CPU is assigned a single class of service to indicate all resources
> > accessible to it.
>
> It is correct. Each core can be assigned to any resource group, and
> each core only uses one sector at a time. Additionally, which sector
> each core uses depends on the resource group (class of service) ID.

Thank you for clarifying. From what I understand this could be supported  
by existing resctrl flows.

...

> > > In resctrl_sched_in(), there are comments as follow:
> > >     /*
> > >    * If this task has a closid/rmid assigned, use it.
> > >     * Else use the closid/rmid assigned to this cpu.
> > >     */
> > > I thought when we write PID to tasks file, this task (PID) will only
> > > run on the CPUs which are specified in cpus file in the same resource
> > > group. So, the task_struct's closid and cpu's closid is the same.
> > > When task's closid is different from cpu's closid?
> >
> > resctrl does not manage the affinity of tasks.
> >
> > Tony recently summarized the cpus file very well to me: The actual semantics of
> > the CPUs file is to associate a CLOSid for a task that is in the default resctrl
> > group ? while it is running on one of the listed CPUs.
> >
> > To answer your question the task's closid could be different from the CPU's
> > closid if the task's closid is 0 while it is running on a CPU that is in the cpus file
> > of a non-default resource group.
> >
> > You can see a summary of the decision flow in section "Resource allocation
> > rules" in Documentation/x86/resctrl.rst
> >
> > The "cpus" file was created in support of the real-time use cases. In these use
> > cases a group of CPUs can be designated as supporting the real-time work and
> > with their own resource group and assigned the needed resources to do the
> > real-time work. A real-time task can then be started with affinity to those CPUs
> > and dynamically any kernel threads (that will be started on the same CPU)
> > doing work on behalf of this task would be able to use the resources set aside
> > for the real-time work.
>
> Thanks for your explanation. I understood it.
>
> I will implement this sector function, and if I have other questions,
> please allow me to mail you.

I will help where I can. You may also be interested in the work James is  
busy with. See his latest series at
https://lore.kernel.org/lkml/20210715173043.14222-1-james.morse@xxxxxxx/

Reinette