Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?)

[Martin Steigerwald]

Dear Joanne.

jdow - 28.06.18, 08:39:
> Anything done to RDBs for Linux must remain 100.000% compatible with
> existing Amiga equipment. Otherwise, what's the point of bothering to
> use RDBs?

Done to, in the sense of written to: Yes. I completely agree. But that 
is for amiga-fdisk and parted. And for partitioning tools on native OS.

[â]
> That brings to the fore an interesting question. Why bother with RDBs
> over 2TB unless you want a disk with one single partition? This Win10
> monster I am using has a modest BIOS driver partition for the OS and
> a giant data partition. That smaller partition would easily work with
> any RDB/Filesystem combination since 2.0. So there are some good
> workarounds that are probably "safer" and at least as flexible as
> RDBs, one Linux has used for a very long time, too.

Well, my use case was simple:

I had this 2 TB disk and I choose to share it as a backup disk for Linux 
*and* AmigaOS 4.x on that Sam440ep I still have next to me desk here.

As AmigaOS up to my knowledge does not have GPT support and MBR with 2 
TB disk is asking for even more trouble and is not supported in any 
sensible partitioning tool on AmigaOS and I choose to use Media Toolbox, 
I went with RDB as I thought it is nicely supported in Linux, which it 
is, apart from the overflowing issue.

So I did it this way.

> As I have said, for the RDB parser fix the famndool thing. Do fix it
> right in such a manner that if somebody compiles it against a version
> with no 64 bit device code it will throw a proper overflow error and
> protect the user. Users are dumb. We like to think of ourselves as

Sure, if the Linux kernel canÂt handle it due to missing CONFIG_LBDAF or 
soâ by all means *bail* out. Without even a kernel option to parse this 
anyway. No overflowing. Period. That is what I wrote from the beginning.

> smart. Let's try to be smart about this where we can so fingers can't
> point back at us rather than the fool that made some other error.

I completely agree.

> And do remember, I am merely (and vociferously) advising rather than
> dictating. You don't need my approval to proceed. I may want my name
> noted as an early contributor only. Meanwhile I spit out ideas as
> they come to me. One or more of them might be good. And offering
> alternatives is better than simply saying "No" most of the time.
> 
> If people are using RDBs for TB level disks I doubt they can remember
> which is the left shoe when they are getting dressed in the morning
> before going out in the yard to beat some dead horses. Or else maybe

Heh, I remembered my shoes back then. And I informed myself about what I 
was doing.

> they just want to see how far they can flog the m68k architecture as
> a mental challenge. In that case, taking it too seriously could hurt.

And well, yesâ I wanted to see how far I can get away with it :)

> Note that I am mostly ignoring m68k Linux. People using that are hard
> core. People using x86/x64 Linux aren't such hard core folks. And I
> bet most of them want to read the disks so they can copy stuff to
> Amiga Forever or WinUAE running on other architectures. So TB is not
> likely to be an issue for them, either.

Yes.

Print a warning and be done with it in the RDB parser code.

Put a big fat warning into amiga-fdisk and parted!

Thanks,
Martin

> {^_^}
> 
> On 20180627 22:43, Michael Schmitz wrote:
> > Joanne,
> > 
> > Linux on m68k has supported lseek64 (or llseek) for a long time
> > (from glibc version 2.1 according to what I found). About the only
> > area where we are limited by 32 bits is the virtual memory size.
> > 
> > I'm not proposing to modify the RDB format definition, though an
> > extension to store 64 bit offsets separate from the 32 bit ones
> > would be one way to make certain such disks are safe to use on 3.1
> > and earlier versions of AmigaOS. (Another one would be to modify
> > the disk drivers on older versions to do the offset calculation in
> > 64 bit, and check for overflow just as we do here. Not sure whether
> > that's feasible. And as you so eloquently describe, we can't rely
> > on users listening.)
> > 
> > Either way, we need the cooperation of partitioning tool writers to
> > ensure partition information that is prone to overflows is never
> > stored in the 32 bit, classic RDB. That appears to have failed
> > already, as Martin's experience illustrates.
> > 
> > I'm only concerned with fixing the (dangerous) but in the Linux
> > partition format parser for RDB. Refusing to use any partitions
> > that will cause havoc on old AmigaOS versions is all I can do to
> > try and get the users' attention.
> > 
> > Your warning makes me wonder whether the log message should just say
> > 'report this bug to linux-m68k@xxxxxxxxxxxxxxx' to at least try and
> > educate any that respond about the dangers of their partitioning
> > scheme before telling them about the override option. Problem with
> > that is, in about three years no one will remember any of this ...
> > 
> > Cheers,
> > 
> >      Michael
> > 
> > Am 28.06.2018 um 15:44 schrieb jdow:
> >> Michael, as long as m68k only supports int fseek( int ) 4 GB *
> >> block
> >> size is your HARD limit. Versions that support __int64 fseek64(
> >> __int64 ) can work with larger disks. RDBs could include RDSK and
> >> a new set of other blocks that replace the last two characters
> >> with "64" and use __int64 where needed in the various values. That
> >> way a clever disk partitioner could give allow normal (32 bit) RDB
> >> definitions where possible. Then at least SOME of the disk could
> >> be supported AND a very clever filesystem that abstracts very
> >> large disks properly could give access to the whole disk. (Read
> >> the RDBs first 32 bits. Then if a filesystem or driveinit was
> >> loaded re-read the RDBs to see if new 64 bit partitions are
> >> revealed.
> >> 
> >> I could be wrong but I do not think RDBs could be safely modified
> >> any
> >> other way to work. And, trust me as I bet this is still true, you
> >> will need a SERIOUSLY good bomb shelter on the Moon if you change
> >> RDBs. Suppose Joe Amigoid uses it, and then Joe Amigoid loads
> >> Amigados 2.4 because he wants to run a game that crashes on
> >> anything newer. Then Joe got far enough something writes to the
> >> disk and data is corrupted. Note further that Amigoids do NOT,
> >> repeat NOT, listen to facts in such cases. Hell, some of them
> >> never listened to facts about an incident at Jerry Pournelle's
> >> place when a 1.1 DPaint session with Kelly Freas hung and we lost
> >> a delightful drawing. Jerry reported it. Amigoids screamed. I
> >> tried to tell them I was there, it was my machine, and 1.1 was,
> >> indeed, crap.
> >> 
> >> {o.o}
> >> 
> >> On 20180627 02:00, Michael Schmitz wrote:
> >>> Joanne,
> >>> 
> >>> I'm not at all allergic to avoiding RDB at all cost for new disks.
> >>> If
> >>> AmigaOS 4.1 supports more recent partition formats, all the
> >>> better.
> >>> This is all about supporting use of legacy RDB disks on Linux
> >>> (though
> >>> 2 TB does stretch the definition of 'legacy' a little). My
> >>> interest in this is to ensure we can continue to use RDB format
> >>> disks on m68k Amiga computers which have no other way to boot
> >>> Linux from disk.
> >>> 
> >>> Not proposing to change the RDB format at all, either. Just trying
> >>> to
> >>> make sure we translate RDB info into Linux 512-byte block offset
> >>> and
> >>> size numbers correctly. The kernel won't modify the RDB at all
> >>> (intentionally, that is - with the correct choice of partition
> >>> sizes,
> >>> Martin might well have wiped out his RDB with the current version
> >>> of
> >>> the parser).
> >>> 
> >>> The choice of refusing to mount a disk (or mounting read-only)
> >>> rests
> >>> with the VFS drivers alone - AFFS in that case. Not touching any
> >>> of
> >>> that. At partition scan time, we only have the option of making
> >>> the
> >>> partition available (with a warning printed), or refusing to make
> >>> it
> >>> available to the kernel. Once it's made available, all bets are
> >>> off.
> >>> 
> >>>  From what Martin writes, his test case RDB was valid and worked
> >>> as
> >>> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
> >>> necessary extensions to handle the large offsets resulting from 2
> >>> TB
> >>> disks. Not sure what safeguards are in place when connecting such
> >>> a
> >>> disk to older versions of AmigaOS, but that is a different matter
> >>> entirely.
> >>> 
> >>> The overflows in partition offset and size are the only ones I can
> >>> see in the partition parser - there is no other overflow I've
> >>> identified. I just stated that in order to place a partition
> >>> towards the end of a 2 TB disk, the offset calculation will
> >>> overflow regardless of what combination of rdb->rdb_BlockBytes
> >>> and sector addresses stored in the RDB (in units of 512 byte
> >>> blocks) we use:
> >>> 
> >>>          blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
> >>> 
> >>> 
> >>>                  nr_sects = (be32_to_cpu(pb->pb_Environment[10]) +
> >>> 1 - be32_to_cpu(pb->pb_Environment[9])) *
> >>> be32_to_cpu(pb->pb_Environment[3]) *
> >>> be32_to_cpu(pb->pb_Environment[5]) * blksize;
> >>>                  if (!nr_sects)
> >>>                          continue;
> >>>                  start_sect = be32_to_cpu(pb->pb_Environment[9]) *
> >>>                               be32_to_cpu(pb->pb_Environment[3]) *
> >>>                               be32_to_cpu(pb->pb_Environment[5]) *
> >>>                               blksize;
> >>> 
> >>> But in the interest of avoiding any accidental use of a RDB
> >>> partition
> >>> where calculations currently overflow, I'll make the default
> >>> behaviour to bail out (instead of using wrong offset or size as
> >>> we currently do). Given the 'eat_my_RDB_disk=1' boot option, the
> >>> user may proceed at their own risk (though I still can't see what
> >>> harm should result from now translating a well formed v4.1 2 TB
> >>> disk RDB correctly for the first time).
> >>> 
> >>> Whether or not Linux correctly handles AFFS filesystems larger
> >>> than 1
> >>> TB is a matter for VFS experts. Bailing out on nr_sects
> >>> overflowing
> >>> ought to prevent accidental use of AFFS filesystems on RDB disks
> >>> which I suppose is the majority of use cases.
> >>> 
> >>> Bugs in partitioning tools on Linux are entirely out of scope -
> >>> the
> >>> partitioning tools bypass the partition structure discovered by
> >>> the
> >>> kernel, and work straight on the raw device. No protecting against
> >>> that.
> >>> 
> >>> If you can point out a way to cause data loss with these
> >>> precautions,
> >>> for a disk 2 TB or larger that was partitioned and used on a
> >>> recent
> >>> version or AmigaOS supporting such large disks, I'd consider
> >>> omitting
> >>> the 'eat_my_RDB_disk' boot option, and just bail out as the only
> >>> safe
> >>> option.
> >>> 
> >>> Cheers,
> >>> 
> >>>      Michael
> >>> 
> >>> Am 27.06.2018 um 18:24 schrieb jdow:
> >>>> You allergic to using a GPT solution? It will get away from some
> >>>> of the evils that RDB has inherent in it because they are also
> >>>> features? (Loading a filesystem or DriveInit code from RDBs is
> >>>> just asking for a nearly impossible to remove malware
> >>>> infection.) Furthermore, any 32 bit system that sees an RDSK
> >>>> block is going to try to translate it. If you add a new RDB
> >>>> format you are going to get bizarre and probably quite
> >>>> destructive results from the mistake. Fail safe is a rather good
> >>>> notion, methinks.
> >>>> 
> >>>> Personally I figure this is all rather surreal. 2TG of junk on an
> >>>> Amiga system seems utterly outlandish to me. You cited another
> >>>> overflow potential. There are at least three we've identified, I
> >>>> believe. Are you 100% sure there are no more? The specific one
> >>>> you mention of translating RDB to Linux has a proper solution in
> >>>> the RDB reader. It should recover such overflow errors in the
> >>>> RDB as it can with due care and polish. It should flag any other
> >>>> overflow error it detects within the RDBs and return an error
> >>>> such as to leave the disk unmounted or mounted read-only if you
> >>>> feel like messing up a poor sod's backups. The simple solution
> >>>> is to read each of the variables with the nominal RDB size and
> >>>> convert it to uint64_t before calculating byte indices.
> >>>> 
> >>>> However, consider my inputs as advice from an adult who has seen
> >>>> the
> >>>> Amiga Elephant so to speak. I am not trying to assert any
> >>>> control. Do as you wish; but, I would plead with you to avoid
> >>>> ANY chance you can for the user to make a bonehead stupid move
> >>>> and lose all his treasured disk archives. Doing otherwise is
> >>>> very poor form.
> >>>> 
> >>>> {o.o}   Joanne "Said enough, she has" Dow
> >>>> 
> >>>> On 20180626 18:07, Michael Schmitz wrote:
> >>>>> Joanne,
> >>>>> 
> >>>>> As far as I have been able to test, the change is backwards
> >>>>> compatible (RDB partitions from an old disk 80 GB disk are
> >>>>> still recognized OK). That"s only been done on an emulator
> >>>>> though.
> >>>>> 
> >>>>> Your advice about the dangers of using RDB disks that would have
> >>>>> failed the current Linux RDB parser on legacy 32 bit systems is
> >>>>> well
> >>>>> taken though. Maybe Martin can clarify that for me - was the 2
> >>>>> TB disk in question ever used on a 32 bit Amiga system?
> >>>>> 
> >>>>> RDB disk format is meant for legacy use only, so I think we can
> >>>>> get
> >>>>> away with printing a big fat warning during boot, advising the
> >>>>> user
> >>>>> that the oversize RDB partition(s) scanned are not compatible
> >>>>> with
> >>>>> legacy 32 bit AmigaOS. With the proposed fix they will work
> >>>>> under both AmigaOS 4.1 and Linux instead of truncating the
> >>>>> first overflowing partition at disk end and trashing valid
> >>>>> partitions that overlap, which is what Martin was after.
> >>>>> 
> >>>>> If that still seems too risky, we can make the default behaviour
> >>>>> to
> >>>>> bail out once a potential overflow is detected, and allow the
> >>>>> user to
> >>>>> override that through a boot parameter. I'd leave that decision
> >>>>> up for the code review on linux-block.
> >>>>> 
> >>>>> Two more comments: Linux uses 512 byte block sizes for the
> >>>>> partition
> >>>>> start and size calculations, so a change of the RDB blocksize to
> >>>>> reduce the block counts stored in the RDB would still result in
> >>>>> the
> >>>>> same overflow. And amiga-fdisk is indeed utterly broken and
> >>>>> needs
> >>>>> updating (along with probably most legacy m68k partitioners).
> >>>>> Adrian
> >>>>> has advertised parted as replacement for the old tools - maybe
> >>>>> this
> >>>>> would make a nice test case for parted?
> >>>>> 
> >>>>> Cheers,
> >>>>> 
> >>>>>    Michael
> >>>>> 
> >>>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@xxxxxxxxxxxxx> 
wrote:
> >>>>>> If it is not backwards compatible I for one would refuse to use
> >>>>>> it.
> >>>>>> And if
> >>>>>> it still mattered that much to me I'd also generate a
> >>>>>> reasonable
> >>>>>> alternative. Modifying RDBs nay not be even an approximation of
> >>>>>> a
> >>>>>> good idea.
> >>>>>> You'd discover that as soon as an RDB uint64_t disk is tasted
> >>>>>> by a
> >>>>>> uint32_t
> >>>>>> only system. If it is for your personal use then you're
> >>>>>> entirely
> >>>>>> free to
> >>>>>> reject my advice and are probably smart enough to keep it
> >>>>>> working for
> >>>>>> yourself.
> >>>>>> 
> >>>>>> GPT is probably the right way to go. Preserve the ability to
> >>>>>> read
> >>>>>> RDBs for
> >>>>>> legacy disks only.
> >>>>>> 
> >>>>>> {^_^}
> >>>>>> 
> >>>>>> On 20180626 01:31, Michael Schmitz wrote:
> >>>>>>> Joanne,
> >>>>>>> 
> >>>>>>> I think we all agree that doing 32 bit calculations on
> >>>>>>> 512-byte block
> >>>>>>> addresses that overflow on disks 2 TB and larger is a bug,
> >>>>>>> causing
> >>>>>>> the
> >>>>>>> issues Martin reported. Your patch addresses that by using the
> >>>>>>> correct
> >>>>>>> data type for the calculations (as do other partition parsers
> >>>>>>> that
> >>>>>>> may
> >>>>>>> have to deal with large disks) and fixes Martin's bug, so
> >>>>>>> appears
> >>>>>>> to be
> >>>>>>> the right thing to do.
> >>>>>>> 
> >>>>>>> Using 64 bit data types for disks smaller than 2 TB where
> >>>>>>> calculations
> >>>>>>> don't currently overflow is not expected to cause new issues,
> >>>>>>> other
> >>>>>>> than
> >>>>>>> enabling use of disk and partitions larger than 2 TB (which
> >>>>>>> may have
> >>>>>>> ramifications with filesystems on these partitions). So
> >>>>>>> comptibility is
> >>>>>>> preserved.
> >>>>>>> 
> >>>>>>> Forcing larger block sizes might be a good strategy to avoid
> >>>>>>> overflow
> >>>>>>> issues in filesystems as well, but I can't see how the block
> >>>>>>> size
> >>>>>>> stored
> >>>>>>> in the RDB would enforce use of the same block size in
> >>>>>>> filesystems.
> >>>>>>> We'll have to rely on the filesystem tools to get that right,
> >>>>>>> too.
> >>>>>>> Linux
> >>>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this
> >>>>>>> should
> >>>>>>> allow partitions larger than 2 TB to work already (but I
> >>>>>>> suspect Al
> >>>>>>> Viro
> >>>>>>> may have found a few issues when he looked at the AFFS code so
> >>>>>>> I
> >>>>>>> won't
> >>>>>>> say more). Anyway partitioning tools and filesystems are
> >>>>>>> unrelated to
> >>>>>>> the Linux partition parser code which is all we aim to fix in
> >>>>>>> this
> >>>>>>> patch.
> >>>>>>> 
> >>>>>>> If you feel strongly about unknown ramifications of any
> >>>>>>> filesystems on
> >>>>>>> partitions larger than 2 TB, say so and I'll have the kernel
> >>>>>>> print a
> >>>>>>> warning about these partitions.
> >>>>>>> 
> >>>>>>> I'll get this patch tested on Martin's test case image as well
> >>>>>>> as
> >>>>>>> on a
> >>>>>>> RDB image from a disk known to currently work under Linux
> >>>>>>> (thanks
> >>>>>>> Geert
> >>>>>>> for the losetup hint). Can't do much more without procuring a
> >>>>>>> working
> >>>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I
> >>>>>>> plan to
> >>>>>>> use
> >>>>>>> for tests is a long way away from my home indeed.
> >>>>>>> 
> >>>>>>> Cheers,
> >>>>>>> 
> >>>>>>>       Michael
> >>>>>>> 
> >>>>>>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>>>>>> As long as it preserves compatibility it should be OK, I
> >>>>>>>> suppose.
> >>>>>>>> Personally I'd make any partitioning tool front end gently
> >>>>>>>> force the
> >>>>>>>> block size towards 8k as the disk size gets larger. The file
> >>>>>>>> systems
> >>>>>>>> may also run into 2TB issues that are not obvious. An unused
> >>>>>>>> blocks
> >>>>>>>> list will have to go beyond a uint32_t size, for example. But
> >>>>>>>> a
> >>>>>>>> block
> >>>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a
> >>>>>>>> tad
> >>>>>>>> under
> >>>>>>>> 1% of the disk all by itself. A block bitmap is not quite so
> >>>>>>>> bad.
> >>>>>>>> {^_-}
> >>>>>>>> 
> >>>>>>>> Just be sure you are aware of all the ramifications when you
> >>>>>>>> make a
> >>>>>>>> change. I remember thinking about this for awhile and then
> >>>>>>>> determining
> >>>>>>>> I REALLY did not want to think about it as my brain was
> >>>>>>>> getting tied
> >>>>>>>> into a gordian knot.
> >>>>>>>> 
> >>>>>>>> {^_^}
> >>>>>>>> 
> >>>>>>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>>>>>> Joanne,
> >>>>>>>>> 
> >>>>>>>>> Martin's boot log (including your patch) says:
> >>>>>>>>> 
> >>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK
> >>>>>>>>> (512)
> >>>>>>>>> sdb1
> >>>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3
> >>>>>>>>> (DOS^C)(res
> >>>>>>>>> 2 spb
> >>>>>>>>> 4)
> >>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:
> >>>>>>>>> [sdb]
> >>>>>>>>> Attached SCSI disk
> >>>>>>>>> 
> >>>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512)
> >>>>>>>>> means
> >>>>>>>>> 512
> >>>>>>>>> byte blocks) and can be worked around by using a different
> >>>>>>>>> block
> >>>>>>>>> size.
> >>>>>>>>> 
> >>>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>>>>>> units.
> >>>>>>>>> I'll still submit a patch to Jens anyway as this may bite
> >>>>>>>>> others
> >>>>>>>>> yet.
> >>>>>>>>> 
> >>>>>>>>> Cheers,
> >>>>>>>>> 
> >>>>>>>>>      Michael
> >>>>>>>>> 
> >>>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@xxxxxxxxxxxxx> 
wrote:
> >>>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>>>>>> system is
> >>>>>>>>>> a famn
> >>>>>>>>>> dool.
> >>>>>>>>>> 
> >>>>>>>>>> If memory serves the RDBs think in blocks rather than bytes
> >>>>>>>>>> so it
> >>>>>>>>>> should
> >>>>>>>>>> work up to 2 gigablocks whatever your block size is. 512
> >>>>>>>>>> blocks is
> >>>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of
> >>>>>>>>>> disk in
> >>>>>>>>>> block maps.
> >>>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>>>>>> 
> >>>>>>>>>> {^_^}
> >>>>>>>>>> 
> >>>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>>>>>> Hi.
> >>>>>>>>>>> 
> >>>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>>>>>> test results at
> >>>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>>>>>> there,
> >>>>>>>>>>>> so if
> >>>>>>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>>>>>> 
> >>>>>>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>>>>>> 
> >>>>>>>>>>> I do not care enough about this, in order to motivate
> >>>>>>>>>>> myself
> >>>>>>>>>>> preparing
> >>>>>>>>>>> the a patch from Joanne DowÂs fix.
> >>>>>>>>>>> 
> >>>>>>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>>>>>> Sam440ep
> >>>>>>>>>>> which
> >>>>>>>>>>> I still have in my apartment.
> >>>>>>>>>>> 
> >>>>>>>>>>> So RDB support in Linux it remains broken for disks larger
> >>>>>>>>>>> 2 TB,
> >>>>>>>>>>> unless
> >>>>>>>>>>> someone else does.
> >>>>>>>>>>> 
> >>>>>>>>>>> Thanks.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k"
> in the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Martin