Hi,
recently upgrading one of my two CPUs, I found kernel-2.4.2 to be unable to
handle the situation with 2 different CPUs (AMP = Assymmetric
multiprocessing ;-) correctly.
Some details on my system:
Dual BX board (DFI P2XBL/D), iPII 350 (Deschutes) + iPIII 850 (Coppermine)
Note: The difference in features is the XMM (SSE) flag.
The problems are twofold
(a) Determination of the correct common features (=: COMCAP), i.e.
boot_cpu_data.x86_capaility[0] at the correct time
(b) TSC stuff
Ad (a):
There is code in identify_cpu() to make sure the common subset of features
is stored in the COMCAP, however this does not work at all, as the freshly
booted CPUs just overwrite it in head.S.
Fixed this, so the system basically worked, if the iPII was the boot CPU.
With iPIII as boot CPU, it rebooted after (captured via serial console):
Asserting INIT.
Waiting for send to finish...
+Deasserting INIT.
Waiting for send to finish...
+#startup loops: 2.
Sending STARTUP #1.
After apic_write.
The problem is that the C/FPU gets initialized to use XMM instructions,
before the other CPUs are even started and checked for capabilities.
Booting with nofxsr (or noxmm which I added) helps.
However, the kernel should be able to find out himself.
To find out about the COMCAP earlier, one has two choices
* boot the secondary CPUs earlier (before initialiying usage of FXSR/XMM)
* try to find out without booting the 2ndary CPUs
The only way I found to do the latter is to use the results of the MPtable
parsing. However, this information may not be reliable. I did implement
this, but on my system, a lot of features were not reported at all ...
So I chose to do smp_init() earlier. It's a good idea to have this info
available before we get to do RAID5 init anyway.
I chose to move smp_init()/smp_boot_cpus() [which now calls identify_cpu()
and check_config() for the boot_cpu and later identify_cpu() on the secondary
CPUs and sets COMCAP correctly], then check_bugs() [which does not do the
identify_cpu() on SMP anymore) before bdev_init().
Problem solved. It does certainly solve XMM/SSE and FXSR issues. PSE36
differences, e.g., may still cause problems on 64GB kernels ...
Ad (b):
The fast_gettimeoffset() does return nonsense, if it runs on the wrong CPU.
There are two reasons: On the one hand, the TSC speed is only calculated for
one CPU, which does not fit the other. On the other hand, xtime gets updated
and the low TSC bits are saved. If gettimeofday() is run on the other CPU,
you compare the TSC of this CPU with the saved one from the other, which
will return nonsense. I've even seen negative numbers.
This behaviour screws up timing information. xntp did not work any more and
the glx module crashed the AGP/PCI bus for my MGA G400.
notsc could work around this of course.
I decided to fix this problem as well.
Therefore, cpu_khz, fast_gettimeoffset_quotient, last_tsc_low have been
moved to, and xtime duplicated in the cpuinfo_x86 struct.
identify_cpu() now calls time_init_cpu() for the secondary CPUs, which
initializes cpu_khz and the quotient.
The timer irq now stores the last_tsc_low in the CPU specific struct.
It also copies the xtime to the current CPU's struct.
gettimeofday() just uses this CPU specific info.
It works.
Find here some evidence:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 8
model name : Pentium III (Coppermine)
stepping : 6
cpu MHz : 851.946
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1701.88
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 5
model name : Pentium II (Deschutes)
stepping : 2
cpu MHz : 350.800
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr
bogomips : 701.54
ntpq> pe
remote refid st t when poll reach delay offset jitter
==============================================================================
LOCAL(0) LOCAL(0) 10 l 15 64 377 0.000 0.000 0.000
*casa.casa-etp.n ntp2.ptb.de 2 u 15 64 377 0.365 1.453 5.000
boot_cpu_data (COMCAP) contains:
Intel Pentium III (Coppermine) caps: 0183fbff fpu vme de pse tsc msr pae mce
cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr stepping 02
Patch against 2.4.2 is attached.
Feedback is welcome. I think the patch is safe, but I could imagine that the
earlier SMP initialization might cause problems for some people or other archs.
I would like this patch to go into the mainstream kernel, if no problem can
be found.
Regards,
-- Kurt Garloff <kurt@garloff.de> [Eindhoven, NL] Physics: Plasma simulations <K.Garloff@Phys.TUE.NL> [TU Eindhoven, NL] Linux: SCSI, Security <garloff@suse.de> [SuSE Nuernberg, FRG] (See mail header or public key servers for PGP2 and GPG public keys.)
This archive was generated by hypermail 2b29 : Fri Mar 23 2001 - 21:00:15 EST