Re: moving affs + RDB partition support to staging?

[Martin Steigerwald]

Dear Joanne.

jdow - 27.06.18, 08:24:
> You allergic to using a GPT solution? It will get away from some of
> the evils that RDB has inherent in it because they are also features?
> (Loading a filesystem or DriveInit code from RDBs is just asking for
> a nearly impossible to remove malware infection.) Furthermore, any 32
> bit system that sees an RDSK block is going to try to translate it.
> If you add a new RDB format you are going to get bizarre and probably
> quite destructive results from the mistake. Fail safe is a rather
> good notion, methinks.
> 
> Personally I figure this is all rather surreal. 2TG of junk on an
> Amiga system seems utterly outlandish to me. You cited another
> overflow potential. There are at least three we've identified, I
> believe. Are you 100% sure there are no more? The specific one you
> mention of translating RDB to Linux has a proper solution in the RDB
> reader. It should recover such overflow errors in the RDB as it can
> with due care and polish. It should flag any other overflow error it
> detects within the RDBs and return an error such as to leave the disk
> unmounted or mounted read-only if you feel like messing up a poor
> sod's backups. The simple solution is to read each of the variables
> with the nominal RDB size and convert it to uint64_t before
> calculating byte indices.
> 
> However, consider my inputs as advice from an adult who has seen the
> Amiga Elephant so to speak. I am not trying to assert any control. Do
> as you wish; but, I would plead with you to avoid ANY chance you can
> for the user to make a bonehead stupid move and lose all his
> treasured disk archives. Doing otherwise is very poor form.

I am pretty confident that larger than 2 TiB disks are fully supported 
within AmigaOS 4, as I outlined in my other mail.

So with all due respect: I used a larger than 2 TiB disk in AmigaOS 4 in 
2012 already *just* fine. I even found I had the same questions back 
then, and researched it. Which lead to this official article back then:

http://wiki.amigaos.net/wiki/RDB

I am also pretty sure that AmigaOS still uses RDB as partitioning 
format. They support MBR. I don´t think AmigaOS 4.1 supports GPT. 
Whether to implement that of course is the decision of AmigaOS 4 
development team. I am no longer a member of it since some time.

Linux m68k should already be able to use disks in GPT format, but you 
likely won´t be able to read them in AmigaOS, unless there is some third 
party support for it meanwhile.

Thanks,
Martin

> 
> {o.o}   Joanne "Said enough, she has" Dow
> 
> On 20180626 18:07, Michael Schmitz wrote:
> > Joanne,
> > 
> > As far as I have been able to test, the change is backwards
> > compatible (RDB partitions from an old disk 80 GB disk are still
> > recognized OK). That"s only been done on an emulator though.
> > 
> > Your advice about the dangers of using RDB disks that would have
> > failed the current Linux RDB parser on legacy 32 bit systems is well
> > taken though. Maybe Martin can clarify that for me - was the 2 TB
> > disk in question ever used on a 32 bit Amiga system?
> > 
> > RDB disk format is meant for legacy use only, so I think we can get
> > away with printing a big fat warning during boot, advising the user
> > that the oversize RDB partition(s) scanned are not compatible with
> > legacy 32 bit AmigaOS. With the proposed fix they will work under
> > both AmigaOS 4.1 and Linux instead of truncating the first
> > overflowing partition at disk end and trashing valid partitions
> > that overlap, which is what Martin was after.
> > 
> > If that still seems too risky, we can make the default behaviour to
> > bail out once a potential overflow is detected, and allow the user
> > to
> > override that through a boot parameter. I'd leave that decision up
> > for the code review on linux-block.
> > 
> > Two more comments: Linux uses 512 byte block sizes for the partition
> > start and size calculations, so a change of the RDB blocksize to
> > reduce the block counts stored in the RDB would still result in the
> > same overflow. And amiga-fdisk is indeed utterly broken and needs
> > updating (along with probably most legacy m68k partitioners). Adrian
> > has advertised parted as replacement for the old tools - maybe this
> > would make a nice test case for parted?
> > 
> > Cheers,
> > 
> >    Michael
> > 
> > On Tue, Jun 26, 2018 at 9:45 PM, jdow <jdow@xxxxxxxxxxxxx> wrote:
> >> If it is not backwards compatible I for one would refuse to use it.
> >> And if it still mattered that much to me I'd also generate a
> >> reasonable alternative. Modifying RDBs nay not be even an
> >> approximation of a good idea. You'd discover that as soon as an
> >> RDB uint64_t disk is tasted by a uint32_t only system. If it is
> >> for your personal use then you're entirely free to reject my
> >> advice and are probably smart enough to keep it working for
> >> yourself.
> >> 
> >> GPT is probably the right way to go. Preserve the ability to read
> >> RDBs for legacy disks only.
> >> 
> >> {^_^}
> >> 
> >> On 20180626 01:31, Michael Schmitz wrote:
> >>> Joanne,
> >>> 
> >>> I think we all agree that doing 32 bit calculations on 512-byte
> >>> block
> >>> addresses that overflow on disks 2 TB and larger is a bug, causing
> >>> the issues Martin reported. Your patch addresses that by using
> >>> the correct data type for the calculations (as do other partition
> >>> parsers that may have to deal with large disks) and fixes
> >>> Martin's bug, so appears to be the right thing to do.
> >>> 
> >>> Using 64 bit data types for disks smaller than 2 TB where
> >>> calculations don't currently overflow is not expected to cause
> >>> new issues, other than enabling use of disk and partitions larger
> >>> than 2 TB (which may have ramifications with filesystems on these
> >>> partitions). So comptibility is preserved.
> >>> 
> >>> Forcing larger block sizes might be a good strategy to avoid
> >>> overflow
> >>> issues in filesystems as well, but I can't see how the block size
> >>> stored in the RDB would enforce use of the same block size in
> >>> filesystems. We'll have to rely on the filesystem tools to get
> >>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
> >>> limitation) so this should allow partitions larger than 2 TB to
> >>> work already (but I suspect Al Viro may have found a few issues
> >>> when he looked at the AFFS code so I won't say more). Anyway
> >>> partitioning tools and filesystems are unrelated to the Linux
> >>> partition parser code which is all we aim to fix in this patch.
> >>> 
> >>> If you feel strongly about unknown ramifications of any
> >>> filesystems on partitions larger than 2 TB, say so and I'll have
> >>> the kernel print a warning about these partitions.
> >>> 
> >>> I'll get this patch tested on Martin's test case image as well as
> >>> on a RDB image from a disk known to currently work under Linux
> >>> (thanks Geert for the losetup hint). Can't do much more without
> >>> procuring a working Amiga disk image to use with an emulator,
> >>> sorry. The Amiga I plan to use for tests is a long way away from
> >>> my home indeed.
> >>> 
> >>> Cheers,
> >>> 
> >>>       Michael
> >>> 
> >>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>> As long as it preserves compatibility it should be OK, I suppose.
> >>>> Personally I'd make any partitioning tool front end gently force
> >>>> the
> >>>> block size towards 8k as the disk size gets larger. The file
> >>>> systems
> >>>> may also run into 2TB issues that are not obvious. An unused
> >>>> blocks
> >>>> list will have to go beyond a uint32_t size, for example. But a
> >>>> block
> >>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
> >>>> under 1% of the disk all by itself. A block bitmap is not quite
> >>>> so bad. {^_-}
> >>>> 
> >>>> Just be sure you are aware of all the ramifications when you make
> >>>> a
> >>>> change. I remember thinking about this for awhile and then
> >>>> determining I REALLY did not want to think about it as my brain
> >>>> was getting tied into a gordian knot.
> >>>> 
> >>>> {^_^}
> >>>> 
> >>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>> Joanne,
> >>>>> 
> >>>>> Martin's boot log (including your patch) says:
> >>>>> 
> >>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512)
> >>>>> sdb1
> >>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
> >>>>> 2 spb
> >>>>> 4)
> >>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> >>>>> Attached SCSI disk
> >>>>> 
> >>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
> >>>>> 512
> >>>>> byte blocks) and can be worked around by using a different block
> >>>>> size.
> >>>>> 
> >>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>> units.
> >>>>> I'll still submit a patch to Jens anyway as this may bite others
> >>>>> yet.
> >>>>> 
> >>>>> Cheers,
> >>>>> 
> >>>>>      Michael
> >>>>> 
> >>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <jdow@xxxxxxxxxxxxx> 
wrote:
> >>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>> system is
> >>>>>> a famn
> >>>>>> dool.
> >>>>>> 
> >>>>>> If memory serves the RDBs think in blocks rather than bytes so
> >>>>>> it
> >>>>>> should
> >>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
> >>>>>> is
> >>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk
> >>>>>> in
> >>>>>> block maps.
> >>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>> 
> >>>>>> {^_^}
> >>>>>> 
> >>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>> Hi.
> >>>>>>> 
> >>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>> test results at
> >>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>> there, so if
> >>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>> 
> >>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>> 
> >>>>>>> I do not care enough about this, in order to motivate myself
> >>>>>>> preparing the a patch from Joanne Dow´s fix.
> >>>>>>> 
> >>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>> Sam440ep
> >>>>>>> which
> >>>>>>> I still have in my apartment.
> >>>>>>> 
> >>>>>>> So RDB support in Linux it remains broken for disks larger 2
> >>>>>>> TB,
> >>>>>>> unless
> >>>>>>> someone else does.
> >>>>>>> 
> >>>>>>> Thanks.
> >>>>>> 
> >>>>>> --
> >>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>> linux-m68k" in
> >>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
> >>>>>> More majordomo info at 
> >>>>>> http://vger.kernel.org/majordomo-info.html
> >>>> 
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe
> >>>> linux-m68k" in the body of a message to
> >>>> majordomo@xxxxxxxxxxxxxxx
> >>>> More majordomo info at 
> >>>> http://vger.kernel.org/majordomo-info.html
> >> 
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe
> >> linux-m68k" in the body of a message to majordomo@xxxxxxxxxxxxxxx
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Martin