On Fri, Apr 03, 1998 at 11:11:15AM -0800, William J. Earl wrote:
> > The VCE bug is actually worse than I thought before. I was in the assumption
> > that we'd handle all cases were VC might hit us because the MIPS ABI takes
> > care of by it's restrictions of the virtual addresses for mmaping. Well,
> > I was wrong. Writing via write(2) to a file that is also mmap(2)ed may
> > result in virtual coherency problems.
> >
> > Another problem is that under Linux one cannot simply allocate a page of
> > a desired colour - which would of course be the prefered solution. Luckily
> > a vce exceptionhandler will not run into the problem under Linux.
In the meantime I've got a draft of the a handler available.
> > A small test program for the mmap/write problem attached. If may be
> > necessary to start it several times in order to make it print the ``Big
> > trouble, man ...'' message.
>
> As soon as I get a chance, I will look at the relevant linux
> code. Note that physical color allocation can also make a big
> performance difference on direct mapped secondary caches, as on all of
> the Indy processors with secondary caches.
Mark Hemment and David Miller had a page coloring scheme implemented. I
think he dumped it again for whatever reasons. Whatever, given the problems
with cache aliasing as result of not having proper cache coloring the code
needs to be resurrected.
> That is, you want to
> maximize the likelihood that the secondary cache indexes of the
> physical pages in a given application will be uniformly distributed
> across the secondary cache. Excessive hot spots will lead to
> dramatically lower performance. Allocation of a page where physical
> color matches intented virtual color matters only if you need
> to use a K0SEG address for the page to avoid TLB misses (as in
> the general exception handler, unless the K2SEG address is wired).
>
> For the mmap/write problem, what I did in IRIX was to first try
> to assign mmap() virtual colors and buffer cache virtual colors
> (colors of the K2SEG address for the page, not necessarily physical
> color of the page, although having the physical color match means that
> a cheaper K0SEG reference can be used) congruent to the virtual color
> of the file offset for that page. Then write() will see the same
> virtual color when accessing the page as will the user program when
> accessing the page using an address created using mmap(). When
> MAP_FIXED and MAP_SHARED are set, however, and the specified virtual
> color for the mapping is not congruent to the specified file offset,
> an extra mechanism is required, namely software ownership switching of
> the "current" virtual color. For the page frame, we remember the
> current virtual color, and arrange that the pg_vr bit is set only for
> mappings which match that virtual color. If we get a fault on a
> mapping of a different color, we writeback-invalidate the primary
> caches for the "current" color, invalidate the "current" mappings (by
> turning off pg_vr), record the new "current" color, and then validate
> the new "current" mappings (by turning on pg_vr). In IRIX 6.3 and
> later versions, I also allow the possibility of a "shared read
> multiple color" state, where all mappings were allowed to be valid,
> but with pg_m off. That is, the "current" color became a
> multiple-reader/single-writer lock on access to the page (where the
> "single-writer" was a color equivalence class, not a single mapping).
> In this case, the transition from "multiple-reader" mode to
> "single-writer" mode requires invalidating all colors of the primary
> cache for the given page. Note that for MAP_FIXED with MAP_PRIVATE,
> we can simply copy the page, even when it has not yet been modified,
> if the mapped virtual color is not congruent to the file offset
> virtual color.
>
> In IRIX, we handle the instruction cache specially, and do not
> attempt to keep it coherent on the processors without hardware VCE
> detection, so the above description is a little more restrictive than
> what actually happens. This approach is based on updates to instruction
> pages being relatively rare, compared to updates to other pages,
> so we wind up doing fewer I cache invalidates overall.
I *think* the VCEI case never happens under Linux, at least not until an
application does something _really_ stupid. What an apps would need to
do is something opening a mapping with PROT_EXEC|PROT_WRITE|PROT_READ
and attempt to modify it by write(2) to it. I don't think this ever
happens in real live.
Anyway, we have to deal with the problem and I'm going to hope the people
that actually have machines with SC/MC CPUs were the hardware detects the
problem for us will help me by running test kernels.
Ralf
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu