Ideas for distributing multiple or larger ELKS disk images

[ghaerr]

From the recent and increasing flurry of activity, it seems that ELKS has, (finally!), reached the point where all the good stuff won't fit on one 1440k floppy disk. I'm using the 2880k floppy image for testing the native C86 compiler and have run out of space on that on occasion, having to delete some programs in order to proceed.

Here are some of the larger categories of stuff that have recently become available:

• Doom
• More and more games
• Nano-X graphical programs
• More man pages
• Unit test routines and C library tester (test_libc, test_float, etc)
• C86 native compiler toolchain with all development header files and libraries (this alone won't fit on a 1440k disk)
• Other user-contributed stuff that might not be in the main elkscmd/ Applications file/repo.

I thought to ask the community what might be best - separate "packages" of floppy images perhaps, or a hard disk image with everything on it? Either has its sets of problems: an issue with multiple floppy packages is whether the package disk is bootable, or whether it's just an add-on that must be mounted in order to be accessed, exactly which programs are on it, etc. An HD image with everything on it would be great for, say, a full C86 devkit and/or all games, but only immediately usable with an emulator. Users with hard drives would have to go through more of a hassle to get a system up and running.

Since I primarily run from QEMU, I kind of like the idea of producing an HD image that has everything on it - this would allow for a single image to be used for everything ELKS. Perhaps someone could write a script that would then extract from that into a series of floppy images for later installation onto real hardware - floppies or install sets for a real hard drive. For real hardware, updating a system like this can take a lot of work, and shouldn't be normally performed unless the distribution is stable.

Since Doom and C86 now require the Open Watcom C compiler, there is the additional issue of having that installed and/or built, in order to build a full ELKS system, should one be buildable from the ELKS repo. Given that the ELKS C library will have to be used for both OpenWatcom programs, as well as C86 toolchain, we're probably already well headed in the direction of requiring OWC to be installed to build ELKS "extras" or a full HD distribution.

Thoughts?

[floriangit]

For me personally, the only viable interface into my 286-PC is and remains the floppy image. Hence, I like the idea of multiple FD images, one bootable, and the other one with a pink EXTRA label ;). And I guess it would be a nice challenge to replace bin/sys finally with a proper installation program (and a proper name, I mean....why 'sys'?), that can handle an interactive "Please insert Disk 2" prompt like it was in good old times. With ASCII arts ... maybe. ;-)

I also think a HD image with "everything" is useful for many other cases, hence I don't see this as a "Either this way or that".

[ghaerr]

I like the idea of multiple FD images

I had the idea of producing a 'make_floppy' script along with a set of set of application lists (like elkscmd/Applications with tags) that would allow users to create floppy images, like:

make_floppy Applications :games   # copy all games from Applications
make_floppy Games :1440k          # make a games disk that fits on 1440k floppy
make_floppy Games-1440            # make a disk from the supplied description file

The idea needs more work, but would allow for "popular" lists to be kept in files named Games, Devkit, etc. It is a bit of a maintenance burden as more stuff is added, but could work.

and a proper name, I mean....why 'sys'?

LOL - I think the name was originally suggested by someone that had previously used the DOS "SYS" program to create bootable floppies. And so it stuck.

that can handle an interactive "Please insert Disk 2" prompt like it was in good old times. With ASCII arts ... maybe. ;-)

Calculating block sizes and handling files that span across multiple floppies would be a lot more work... looking for contributors on that one, and/or converting from shell scripts, which are used now for building ELKS images and the sys program, to a C program or something a lot more sophisticated. Was it you that pointed out the site that generated ascii art from any text? That was pretty cool :)

[tyama501]

Thank you for creating this thread.

This might be not comment for the disks but
how about using git submodule to include application other than ELKS repo?

I haven't use git submodule in github so I'm not sure it works well
but add submodule of released commit that is stable for large applications
like Doom and my Elksmoria.
(well, I'm not sure my application is stable but...)

[floriangit]

Calculating block sizes and handling files that span across multiple floppies would be a lot more work...

I was not thinking that complex to be honest. We can keep the current situation where the "base" is defined in (e.g.) the 1440 config file from which I guess menuconfig is using cross-configuration.

Then we could define:

• add make basediskconfig maybe same as menuconfig (or even keep menuconfig altogether instead)
• add make extradiskconfig to bake the image for a 2nd floppy.

One challenge could be, that we already want at the make basediskconfig time - when saving - to check whether the image size will fit into the configured, say, 1440 config. I'm afraid we will only know "after the fact" (after compilation) like it is now...?

In the kernel we could then, before the init process is spawned, check:

• "am I" booted from floppy? AND did I detect a HD during boot?
• if so, offer a selection to either continue boot (init) or installation (spawns our newly-made installation program)

Then everything could be done in user-space like /bin/sys does today, but more advanced of course. Could be written in C as well, surely. I found some similar logic/inspiration here, but seems for a different purpose, following these lines:

elks/elks/fs/super.c

Line 473 in 2f2c2e1

close_filp(d_inode, filp);

Anyhow, I could have a fundamental misunderstanding, so feel free to enlighten me 🥇

[tyama501]

Hello @ghaerr

Current ELKSmoria still can be built with ia16 gcc, so I updated the Makefile to build at elkscmd.
tyama501/ELKSmoria@2bc05fd

I noticed there is no memory.h anymore.
I used string.h instead.

I can copy elks releated files under elkscmd for now if it is easier to include in hard disk image.

It needs morifiles in current directory in the image now,
so it might be better to place those in /lib and read from there instead.

[floriangit]

In the kernel we could then, before the init process is spawned, check:

"am I" booted from floppy? AND did I detect a HD during boot?
if so, offer a selection to either continue boot (init) or installation (spawns our newly-made installation program)

master...floriangit:elks:install_option

This is hacky (some is too hacky I guess), but it works. It will currently either continue with the regular initprocess (no change) or start a shell directly (since we don't have an installer yet, this is just to prove it works). I tested on QEMU and on the 286-PC (where it will not run, since I boot from HDD and this fact does not make it an installation candidate ;) )

You can try the code path in QEMU by doing:
qemu-img create -f raw /tmp/hda.img 20M
and then add -hda /tmp/hda.img to you qemu command line in qemu.sh

[ghaerr]

Hello @floriangit,

While I can see this was probably fun to add into the kernel init code, my opinion is that we don't need special kernel handling in order to accomplish what you're suggesting. Using the /bootopts init= option or just adding the capability into /etc/rc.sys would work just as well and don't require special kernel handling. Both those approaches have the big advantage of not requiring a special kernel build for a distribution disk. There has been some work done to allow user programs/shell scripts to get a list of drives, but IIRC they may have to be mounted first, which won't work here. That should be somewhat easily extended though.

But before we get into all that, when someone boots from floppy and has a hard disk attached, what is wrong with just using the 'sys' command (script) to prepare the HD? I know you're not crazy about sys being a shell script, but aside from that, what exactly doesn't it do that should be handled? Sys should be able to copy everything (selectively perhaps) from the FD to the HD, which doesn't seem much different than running a special installer. If new disks need to be inserted, that should be able to be handled, although the kernel might get quite fussy and problematic about having the root-mounted floppy removed. (This would be the case even if handled internal to the kernel too, BTW).

if so, offer a selection to either continue boot (init) or installation

Then everything could be done in user-space like /bin/sys does today

I definitely like the idea of keeping it all in user space. And with an addition to /etc/rc.sys or /bootopts init=, yet another script could start running which could ask the installation question and and then do whatever is needed (as well as run a fancy new installation program, should someone write one) or continue with normal init.

IMO, a big advantage of keeping this sort of thing in shell script is maintainability, as well as allowing for ELKS build-time tools to easily specify/add/subtract programs to a distribution, without changing any binaries which would already have been built.

Thank you!

[floriangit]

Thanks for your feedback! Yes, it was more of a nice tour to learn, hence no PR ;-)
I like the idea though, since then ELKS would be smarter than some linux distros, that ask you during bootloader (grub2 etc) whether to install or "live" boot. At our point ELKS has a HDD mounted and we are on the potential installation medium, so we can exclusively jump to any user program (no need for thinking in disk blocks as you mentioned, but think in filesystems) like an installer handling multiple floppy installations. If that can all be done with init(tab) & rc scripts, that's also fine. I just did not look into it. ;)

sys is not my favorite, but I don't say it should be a C program if not really necessary. In my experience it has two downsides:

• cannot update a system. The error messages from mknodetc. that indicate that I already have /dev/zero and whatnot scroll down making me think that everything is wrong. Okay, we don't have any fancy update mechanism in ELKS anyhow so far. But thinking ahead of libc and more on the target and the usual question: what has been updated and what do I have to copy now? the kernel only or also libc, or also applications, the compiler? I think a smarter install/update tool is necessary.
• error handling in general needs improvement, e.g. some sort of interactivity if errors happen.

[ghaerr]

In my experience it has two downsides:

sys ... cannot update a system.
indicate that I already have /dev/zero and whatnot scroll down making me think that everything is wrong.

Ok - agreed. sys should allow one to sys a system that has already been sys'd (how do you like the sound of that line?!)
Fixing that is pretty easy - the script just needs a few if statements and those errors won't happen. I've made a note of it, unless you'd like to take a pass at it, and I agree it needs to be fixed.

the usual question: what has been updated and what do I have to copy now?

Also agreed. But the big problem here is that, given ELKS' stage of development, usually all the binaries and kernel should be updated together, to be safe. Even if we were not to do that, frankly, running cmp on the two files before copying using ELKS would be far slower than just copying in the first place. A solution involving host-based file manifests would be way too much work at this point in our dev cycle. So in general, I would say for a while into the future we're going to want to just update as much as possible (but not error like it does now).

the kernel only or also libc, or also applications, the compiler?

Yes, this would get longer-winded when apps + toolchain + header files need updating. In general though we're still limited to 1440k disks, one at a time. One idea here would be putting the sys program and kernel on any distribution, allowing it to boot and thus copy directly using sys or another script/program, but that uses up a lot of space. Other big problems are that this method won't work for Doom, nor likely a full dev-disk as they're way bigger than a 1440k floppy. So we're back into large files overlapping floppies, which would likely need an installation program, not script (as well as the host side too).

error handling in general needs improvement, e.g. some sort of interactivity if errors happen.

Agreed. I still think the easiest first step would be to get sys working without errors for the basic installation tasks you're talking about, then moving forward with the problem of larger distributions.

BTW, the kernel isn't set up for allowing a single floppy system to switch floppies (i.e. unmount root and put another disk in its place) for any installation program. There is currently no way to actually unmount root and have the kernel continue running safely. So a dual floppy system is potentially necessary, and/or having a bootable kernel on each floppy.

Other ideas we've talked about are distributing larger images, such as a 32MB (or smaller) HD distribution, which would basically just be a 32MB MINIX filesystem. That could potentially be copied using Linux tools or a network copy to an ELKS system with an HD. The advantage here is that a single HD distribution could be made with everything on it, very easily and automatically nightly.

Yet another idea would be to port one of the old UNIX utilities, such as cpio, which can already be used to partition files into fixed-size (floppy-sized) chunks, which could then be un-archived at runtime directly onto a mounted filesystem (HD). This would still require a two-floppy system though.

As I write this, I realize the whole problem is potentially quite messy.

[floriangit]

There is currently no way to actually unmount root and have the kernel continue running safely

That's what I wanted to learn ;-) Chicken & Egg, huh? How do DOS games do that? With that protected mode kernel mode thing which we don't (or do?) run?

Fixing that is pretty easy - the script just needs a few if statements and those errors won't happen. I've made a note of it, unless you'd like to take a pass at it, and I agree it needs to be fixed.

Agreed, and either way fine, I will not have time before the next weekend, but I can try if you want.

As I write this, I realize the whole problem is potentially quite messy.

Yes, still too many options and none seems golden. Multiplied by the usual: "Ask 3 persons to get 9 opinions" ;-)

[ghaerr]

Chicken & Egg, huh?

No, not really. ELKS or Linux as an OS is entirely different from DOS: ELKS/Linux are built as a multitasking kernel with an associated design that buffers I/O for speed. DOS on the other hand is single-tasking, and programs are allowed to hook interrupts, take over the system, etc. They are very different. Also, the FAT filesystem is extremely different than MINIX: FAT was designed for DOS which never buffers any I/O, whereas MINIX is run on multitasking OSes which, at any time, may write the filesystem with un-synced buffers. This is the reason why there is no easy way to umount the root filesystem, as there may be other running time-sliced processes requesting I/O at any time. There's lots of other technical issues, but that's it in a nutshell.

How do DOS games do that? With that protected mode kernel mode thing which we don't (or do?) run?

DOS games take over the system entirely - they reprogram the PIT and PIC, take over interrupt vector tables, and generally do whatever they want. Since nothing else is running, that works for DOS, and the program can restore it all on exit. They can request the user change a floppy and nothing else is running so nothing else cares. On ELKS/Linux, reprogramming any of the hardware used by the kernel is a strict no-no, and the kernel filesystem code assumes it can write to root at any time. There is no notion of "no root", even with the root filesystem mounted read-only. None of this really has anything to do with protected mode, just the nature of the difference of design of the OSes.

I can try if you want.

If you find time to enhance sys to update your system from a distribution floppy image, please go ahead, and submit a PR. And it sounds like you've got a system with which to test your changes with, which is great. If you don't, I've added it to my TODO list :)

[toncho11]

In my opinion it is also a question of folder organization.

For example:
/bin - all tools and system commands
/usr/bin - all programs with subfolders for the toolchain, doom (more than 1 file) , x-nano programs (sometimes more than 1 file)
/usr/src - all sources that can be compiled by the toolchain with subfolder for each application

Where will be the binaries of /usr/src deployed be default? It does not need to be /usr/bin. These are custom binaries that the user should decide if he wants to make them "common" binaries.
Once the folder organization is settled, we can start thinking of what extra distributions we can have.
Obviously having an HDD image with all programs and sources is needed at least per major ELKS release. It is simply a good showcase. It is a task I could do for each release, but the folder structure should already be there.

Also the PATH variable might grow a lot.

[floriangit]

I agree to the above from @toncho11 and also a side comment on a PR from @ghaerr. If a better installer/updater is appreciated, we should have some kind of requirements/design in which direction this should go. There are so many options as well as restrictions (don't oversize the image for real floppy PC users) I guess.
I wonder how qemu-only users deal with "reality"? Do you guys use -hda in your qemu.sh and have a pseudo-system to deal with? Or is it cared at all? I'm rather unsure what the project wants to achieve as a whole I guess ;-)

[toncho11]

The name sys was my idea as in DOS. In my opinion sys as a program/shell script simply works very well.

It could be extended to copy more stuff from a 1440 floppy for example. Maybe in this case we can have another script ... you already guessed it ... called "install".

[ghaerr]

The name sys was my idea as in DOS. In my opinion sys as a program/shell script simply works very well.
It could be extended to copy more stuff from a 1440 floppy for example. Maybe in this case we can have another script ... you already guessed it ... called "install".

It seems three separate things are brought up in this comment, and previously: 1) whether to call the program that creates a new, separate bootable HD or FD image should be called "sys", or "install", 2) whether that program should be a C program or a shell script, and 3) whether that same program should be extended to copy more files than is necessary for a bootable image, or whether a separate program/script (e.g. "install") should be used.

My take on these three items is:

• Either "sys" or "install" is OK with me. @toncho11 originally suggested "sys" as it was similar to DOS's "sys"; @floriangit has suggested that "install" is a better name for such a program.script. I suppose this may need more discussion between you two and possibly others.
• I think the sys/install program is much better suited as a shell script than as C program, as it is much easier to maintain and visually inspect by non-programming users more oriented towards distribution issues than programming.
• I think having a single script which handles copying from the booted volume to another volume, with an option for "full" versus bootable-system only (which is the current action and default) is much better than maintaining two scripts, which would be mostly duplicated code anyways. If an enhancement is made to "sys"/"install" to copy all binaries from the boot volume, I would say that "install" frankly is better name than sys for either option; "sys" could mean anything, whereas "install" is pretty explanatory (with the exception that ELKS has had a "sys" script for quite a while).

The only difference in copying all of binaries versus just the system binaries, especially now that the system installer has been enhanced to allow updating is about three lines of shell script: copy all of /bin rather than just named programs from /bin. The reason why we even built the installer to copy only a few files was/is to support building a 360k floppy from a 1440k floppy, as otherwise it would quickly run out of space. Of course, we could add special tags to indicate which files to copy for which destination sizes, but this complicates things quite a bit versus a simpler first approach of either copying a minimal bootable system versus everything.

I wonder how qemu-only users deal with "reality"? Do you guys use -hda in your qemu.sh and have a pseudo-system to deal with?

I'm not sure the question here? I myself use QEMU all the time, and uncomment various IMAGE= lines to support running whichever image I want to run. However, my use case is built around always running the latest build, rather than building a hard-drive image once with the intention to be updated occasionally. So yes - I run -hda or various combinations of floppy/hard drives in order to test and implement various ELKS facilities - but not to create distributions.

I'm rather unsure what the project wants to achieve as a whole I guess

I would say that, for me, my role as maintainer is different than a typical actual user (as described above). Thus, "distribution" oriented issues regarding creating images of programs built outside of the ELKS tree (e.g. Doom, 8086-toolchain, etc) might be better supported outside of the main ELKS repo, as the source code to those binaries aren't in the ELKS repo. I don't have a problem bringing those binaries into the repo, or possibly as @tyama501 suggested using git sub-repos, but that's another whole ballgame, at least for now, especially for those external repos that I do not maintain. Individual users's goals might be quite different - some want games, others want dev kits, etc, so more discussion will be needed on distribution issues versus ELKS "system updates".