Re: [PATCH v4 02/16] selftests/resctrl: Calculate resctrl FS derived mem bw over sleep(1) only

[Ilpo Järvinen]

On Tue, 28 May 2024, Reinette Chatre wrote:
> On 5/28/24 3:19 AM, Ilpo Järvinen wrote:
> > On Fri, 24 May 2024, Ilpo Järvinen wrote:
> > > On Fri, 24 May 2024, Reinette Chatre wrote:
> > > > On 5/24/24 12:57 AM, Ilpo Järvinen wrote:
> > > > > On Thu, 23 May 2024, Reinette Chatre wrote:
> > > > > > On 5/20/24 5:30 AM, Ilpo Järvinen wrote:
> > > > > > > For MBM/MBA tests, measure_vals() calls get_mem_bw_imc() that
> > > > > > > performs
> > > > > > > the measurement over a duration of sleep(1) call. The memory
> > > > > > > bandwidth
> > > > > > > numbers from IMC are derived over this duration. The resctrl FS
> > > > > > > derived
> > > > > > > memory bandwidth, however, is calculated inside measure_vals() and
> > > > > > > only
> > > > > > > takes delta between the previous value and the current one which
> > > > > > > besides the actual test, also samples inter-test noise.
> > > > > > > 
> > > > > > > Rework the logic in measure_vals() and get_mem_bw_imc() such that
> > > > > > > the
> > > > > > > resctrl FS memory bandwidth section covers much shorter duration
> > > > > > > closely matching that of the IMC perf counters to improve
> > > > > > > measurement
> > > > > > > accuracy. Open two the resctrl mem bw files twice to avoid opening
> > > > > > > after the test during measurement period (reading the same file
> > > > > > > twice
> > > > > > > returns the same value so two files are needed).
> > > > > > 
> > > > > > I think this is only because of how the current reading is done,
> > > > > > resctrl
> > > > > > surely supports keeping a file open and reading from it multiple
> > > > > > times.
> > > > > > 
> > > > > > There seems to be two things that prevent current code from doing
> > > > > > this
> > > > > > correctly:
> > > > > > (a) the fscanf() code does not take into account that resctrl also
> > > > > >       prints a "\n" ... (this seems to be the part that may cause
> > > > > > the same
> > > > > >       value to be returned).
> > > > > >       So:
> > > > > > 	if (fscanf(fp, "%lu", mbm_total) <= 0) {
> > > > > >       should be:
> > > > > > 	if (fscanf(fp, "%lu\n", mbm_total) <= 0) {
> > > > > > (b) the current reading does not reset the file position so a second
> > > > > >       read will attempt to read past the beginning. A "rewind(fp)"
> > > > > >       should help here.
> > > > > 
> > > > > (b) cannot be the cause for returning the same value again. It would
> > > > > not be able to reread the number at all if file position is not moved.
> > > > 
> > > > I know. This was not intended to explain the duplicate answer but
> > > > instead
> > > > describe another change required to use current code in a loop. I
> > > > specifically said in (a) that "(this seems to be the part that may cause
> > > > the same value to be returned)".
> > > > 
> > > > > I certainly tried with fseek() and it is when I got same value on the
> > > > > second read which is when I just went to two files solution.
> > > > > 
> > > > > > A small program like below worked for me by showing different values
> > > > > > on every read:
> > > > > > 
> > > > > > #include <stdio.h>
> > > > > > #include <stdlib.h>
> > > > > > #include <unistd.h>
> > > > > > 
> > > > > > const char *mbm_total_path =
> > > > > > "/sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes";
> > > > > > 
> > > > > > int main(void)
> > > > > > {
> > > > > > 	unsigned long mbm_total;
> > > > > > 	FILE *fp;
> > > > > > 	int count;
> > > > > > 
> > > > > > 	fp = fopen(mbm_total_path, "r");
> > > > > > 	if (!fp) {
> > > > > > 		perror("Opening data file\n");
> > > > > > 		exit(1);
> > > > > > 	}
> > > > > > 	for (count = 0; count < 100; count++) {
> > > > > > 		if (fscanf(fp, "%lu\n", &mbm_total) <= 0) {
> > > > > > 			perror("Unable to read from data file\n");
> > > > > > 			exit(1);
> > > > > > 		}
> > > > > > 		printf("Read %d: %lu\n",count ,mbm_total );
> > > > > > 		sleep(1);
> > > > > > 		rewind(fp);
> > > > > > 	}
> > > > > > 	fclose(fp);
> > > > > > 	return 0;
> > > > > > }
> > > > > 
> > > > > Okay, so perhaps it's your explanation (a) but can libc be trusted to
> > > > > not
> > > > > do buffering/caching for FILE *? So to be on the safe side, it would
> > > > 
> > > > Coding with expectation that libc cannot be trusted sounds strange to
> > > > me.
> > > > 
> > > > > need to use syscalls directly to guarantee it's read the file twice.
> > > > > 
> > > > > If I convert it into fds, fscanf() cannot be used which would
> > > > > complicate
> > > > > the string processing by adding extra steps.
> > > > > 
> > > > 
> > > > It is not clear to me why you think that fscanf() cannot be used.
> > > 
> > > This was related to fscanf() not being able to read from an fd which is
> > > different interface than what libc's FILE * is.
> 
> The part I am missing is why you believe syscalls are required. Could
> you please elaborate why FILE * cannot be used?
> 
> > > 
> > > > Could you please elaborate what the buffering issues are?
> > > 
> > > I'm pretty sure that by default libc does some buffering (even std*
> > > streams are line buffered and others streams even more). I'm not entirely
> > > sure about the extent of that buffering but here we need to always read
> > > the up to date value from the file itself, not from some buffer.
> > > 
> > > Maybe there never is any problem that the earlier read values are returned
> > > from some libc buffer when lseek/rewind is used, I just don't know that
> > > for sure. You seem to be more certain but I've not seen on what basis
> > > (other than the anecdotial test you provided).
> 
> I demonstrated that it works. I have not heard a clear reason why this
> conclusion
> is incorrect. The above remains vague to me and I cannot find a description of
> a clear problem that can be studied.

It's pointless to continue this discussion as I don't have anything 
concrete to prove you wrong.

So count it as my paranoia when it comes to putting buffering in between 
something that is read more than once and expecting to get different 
bits out of that buffered interface :-).

I'll just use a single FILE *.

> > > > It is not necessary to open and close the file every time a value needs
> > > > to be read from it.
> > 
> > I'm bit unsure where to go with this. While I could change the code to
> > match what you described, I realized with the two files approach there's
> > no need to do even review/lseek() call during the measurement. It might
> > not be very significant compared with the open that was there initially
> > but it's still extra.
> 
> We are discussing the resctrl selftests that will accompany the resctrl
> filesystem in the kernel. When in doubt on how to interact with resctrl users
> use the selftests as reference. Needing to open and close a resctrl file
> every time a value needs to be read from it is not the correct guidance.

That's actually a different goal from the earlier, but I've no problem 
adjusting to it.

Initially, this open/close() refactoring was made because of another goal 
which was to avoid doing extra syscalls during the test.

-- 
 i.