[kernel] Added raw driver for BIOSHD, other fixes

[Mellvik]

Adding exceptions for BIOS IO in all kinds of utilities became too much of a pain. Thus a raw driver for BIOS io has been added. Summary of changes:

bioshd.c:

• Added raw support in the block driver and a new rbioshd.c char driver.
• Removed limitation that prevented floppy IO from spanning tracks.
• Initiated changes that will (soon) allow the driver to use the new floppy cache and the fdcache= bootopts setting.
• Added the capability to truncate IO ops if they partly exceed end of medium.

block_dev.c (raw_blk_rw):

• Added missing code for handling end-of-medium situations, which created IOERR instead of EOF or ENOSPC.

[ghaerr]

Removed limitation that prevented floppy IO from spanning tracks.

Nice easy enhancement there. Have you run into any issues with this? I'm trying to remember from our FDC discussions last August, but seem to remember that this was originally written assuming that the BIOS may not handle multi-track reads, even the the FDC chip always can. Do you believe any BIOSes may suffer from this limitation?

Added raw support in the block driver and a new rbioshd.c char driver.

It seems that with the open/close/ioctl entry points calling the same block driver function along with requiring major char/block device numbering to be the same, and only the struct file operations read/write entry points being different, that perhaps there isn't really a need for special "char" disk drivers separate from block device drivers. The block_rd/block_wr switch out could be handled in fs/block_dev.c or slightly above that level, and ultimately the driver wouldn't have to know whether its a char or block device driver. This was the design I was originally moving towards for char devices in ELKS, but I realize TLVC has taken a different approach - which now seems to be becoming a bit complicated and possibly restrictive with major numbers having to match and specialized char driver code increasingly added in a number of areas. Not to mention that every block device now has to have special extra code added to operate as a char device.

It seems to me that if the disk device drivers were only passed a sector number and number of sectors (which is how I rewrote ELKS block drivers in anticipation of this), as well as having either a buffer header or far address in the request struct, then the driver(s) could perform I/O without having to know anything about the upper level's use of the data. The difference between a char and block driver would be almost limited to using the buffer header and operating as a block device, vs far address and character device behavior. After I/O is complete, the same inspection of buffer header vs far address would be used to signal I/O complete, etc. I rewrote ll_rw_blk.c a long ago to pass only a rq_sector and rq_nr_sectors to the block device driver - so the ELKS drivers don't know anything about blocks, only sectors.

[ghaerr]

Adding exceptions for BIOS IO in all kinds of utilities became too much of a pain.

Definitely a good move getting #include <linuxmt/config.h> back out of as many applications as possible (for other reasons than why you're doing it) - as otherwise those applications must be recompiled for each version of the configured kernel, which them means that an entirely new set of applications must be distributed for every kernel config, as well as hardware platform variation (IBM PC, PC-98, etc). The remaining applications that must be recompiled are handled specially in the buildimages.sh script cleanup() function.

[Mellvik]

Removed limitation that prevented floppy IO from spanning tracks.

Nice easy enhancement there. Have you run into any issues with this? I'm trying to remember from our FDC discussions last August, but seem to remember that this was originally written assuming that the BIOS may not handle multi-track reads, even the the FDC chip always can. Do you believe any BIOSes may suffer from this limitation?

I don't, and I'm running tests on the original PC/XT as we speak (delayed by - incidentally - some floppy problems, physical type). My thinking is that if the XT BIOS can do it, there is no reason to think other BIOSes can not. Why would they be less powerful, this being an essentially no cost 'feature'? I'll be back on that.

Added raw support in the block driver and a new rbioshd.c char driver.

It seems that with the open/close/ioctl entry points calling the same block driver function along with requiring major char/block device numbering to be the same, and only the struct file operations read/write entry points being different, that perhaps there isn't really a need for special "char" disk drivers separate from block device drivers. The block_rd/block_wr switch out could be handled in fs/block_dev.c or slightly above that level, and ultimately the driver wouldn't have to know whether its a char or block device driver. This was the design I was originally moving towards for char devices in ELKS, but I realize TLVC has taken a different approach - which now seems to be becoming a bit complicated and possibly restrictive with major numbers having to match and specialized char driver code increasingly added in a number of areas. Not to mention that every block device now has to have special extra code added to operate as a char device.

It seems to me that if the disk device drivers were only passed a sector number and number of sectors (which is how I rewrote ELKS block drivers in anticipation of this), as well as having either a buffer header or far address in the request struct, then the driver(s) could perform I/O without having to know anything about the upper level's use of the data. The difference between a char and block driver would be almost limited to using the buffer header and operating as a block device, vs far address and character device behavior. After I/O is complete, the same inspection of buffer header vs far address would be used to signal I/O complete, etc. I rewrote ll_rw_blk.c a long ago to pass only a rq_sector and rq_nr_sectors to the block device driver - so the ELKS drivers don't know anything about blocks, only sectors.

I agree all the way. This is where I'm coming from (the original unix way) and this is where TLVC is heading. Unix'en had the aptly named physio lowlev routines doing exactly that, and TLVC (and, I suppose, ELKS) is very close to that point as you observe. Removing the skeleton char drivers is a minimal issue when the block_dev routines have been straightened out and merged when appropriate. The major device issue is minor. The 'equal major #' requirement was a shortcut to make life easy during development. I was in process of removing it when something more pressing caught my attention a few weeks ago. Simply a matter of a 6 or 8 byte table matching up the block and raw device numbers.

Whether to keep the buffer header as a mechanism to communicate between the layers, is debatable. It's really convenient - in particular given the fact that raw access requires a sector buffer to handle small transfers and odd byte counts.

Thanks.

[ghaerr]

Whether to keep the buffer header as a mechanism to communicate between the layers

Definitely not!!! Use the request queue, that's the communications mechanism to and from disk devices! Dragging in and redefining the kernel buffer system's buffer structures when only temp 512-byte character buffer is required is total kluge.

raw access requires a sector buffer to handle small transfers and odd byte counts.

Use heap_alloc in the upper level raw read/write routine, fill in a proper request header using a new standardized function, then wait for I/O to complete and free the buffer. Now that you more fully understand how a lot more of the kernel works, this shouldn't be a big deal.

and, I suppose, ELKS

When you first started working on raw driver support over a year ago, I immediately redesigned all ELKS block drivers to be ready to accept character-style requests from the request queue (only), in anticipation. A few details remain to be worked out, such as signaling I/O complete differently, etc. But I stopped working on it after seeing the buffer header kluge code and the creation of separate "character" drivers for each physical disk device. All are unnecessary with a better design, and I didn't want to add that all to ELKS, only to remove it all again.

I agree all the way. This is where I'm coming from (the original unix way) and this is where TLVC is heading.

I am glad you've seen the light! Happy to give you any ideas but I think given your increasing knowledge and speed of writing code with the system, it will be straightforward. What I hope to gain from your work and this discussion is a shared TLVC/ELKS driver compatibility, allowing for better integration of our work to be used by other people. The world already has far too many proprietary "drivers" (think network in this case) that are unique per system, and that code has to be rewritten, time after time again (by both of us now), trying to take code from a dead system but instead having to rewrite it. We're still rewriting NIC driver code from 30 years ago (boy I wish we could use the packet driver interface in ELKS or TLVC but I stopped going down that path when I realized the internals of which are littered with INT 21h DOS sys calls for arbitrary reasons).

A "driver" should depend on a single incoming data structure (e.g. the request queue), and not specialized (re)handling of the buffer system data structure (e.g. buffer header), especially when the "driver" is tasked with skipping buffer system handling altogether - the very purpose of raw devices. The purpose of the kernel buffer system is to reuse data without having to reread it; the raw device system's purpose is to force reading directly from the device and never reuse data. If a temp "buffer" is needed, heap_alloc is there for that, just as it is used in areas of the kernel.

[Mellvik]

FWIW - track spanning is ok on xt too, but my fix is the bioshd driver was wrong, fixed now. IOW, it's a keeper.
The other discussion - more tomorrow. Good to see we're aligned on this.

[Mellvik]

When you first started working on raw driver support over a year ago, I immediately redesigned all ELKS block drivers to be ready to accept character-style requests from the request queue (only), in anticipation. A few details remain to be worked out, such as signaling I/O complete differently, etc.

That's encouraging, I didn't know that. Over the years (while working on ELKS) my repeated attempts to make the case for raw access to storage devices were rejected, at times even ridiculed. One of many reasons TLVC got started in the first place.

But I stopped working on it after seeing the buffer header kluge code and the creation of separate "character" drivers for each physical disk device. All are unnecessary with a better design, and I didn't want to add that all to ELKS, only to remove it all again.

It's all a process - as we both know from experience. And as you point out, during that process I've gathered more understanding of parts of the system I didn't know much about before. New knowledge opens new paths.

I agree all the way. This is where I'm coming from (the original unix way) and this is where TLVC is heading.

I am glad you've seen the light! Happy to give you any ideas but I think given your increasing knowledge and speed of writing code with the system, it will be straightforward.

I don't know what light you're referring to :-), but I'm glad you like it! Moving towards the simplicity of the original Unix structure has always been a goal. Familiar territory. The skeleton raw drivers were always a hack, and like the buffer header 'abuse', they've served their purpose well. It's always (and I believe you've reiterated this a number of times too) easier to move ahead, improve, revise when something is working. It allows prioritizing too - and there is an endless list of improvements to be made to both systems.

What I hope to gain from your work and this discussion is a shared TLVC/ELKS driver compatibility, allowing for better integration of our work to be used by other people. The world already has far too many proprietary "drivers" (think network in this case) that are unique per system, and that code has to be rewritten, time after time again (by both of us now), trying to take code from a dead system but instead having to rewrite it. We're still rewriting NIC driver code from 30 years ago (boy I wish we could use the packet driver interface in ELKS or TLVC but I stopped going down that path when I realized the internals of which are littered with INT 21h DOS sys calls for arbitrary reasons).

Concur on that - including the strive for cross-compatibility and about the packet driver. The performance delivered by MTCP on DOS - on just about any thinkable net device is truly amazing, even if we deduct the benefit of a single tasking system.
That said and to the defence of writing or rewriting drivers, it forces complete understanding of the underlying hardware and key functions of the OS itself, which has been a strong motivator in itself (in my case). And maybe the reason drivers is my thing - enjoyable even.

A "driver" should depend on a single incoming data structure (e.g. the request queue), and not specialized (re)handling of the buffer system data structure (e.g. buffer header)...

Agreed, and that's the way it is in TLVC, most likely in ELKS too - with one exception: The drivers do buffer syncing and flushing. They should not.
Reworking the functions in block_dev.c is an interesting challenge. Removing the skeleton raw drivers (and disconnecting the major numbers) is just some work to be done. Right now there are other issues higher up on the list - like ktcp and directhd interrupts, but we're getting there.

[Mellvik]

FWIW - track spanning is ok on xt too, but my fix is the bioshd driver was wrong, fixed now. IOW, it's a keeper. The other discussion - more tomorrow. Good to see we're aligned on this.

I take that back. It turns out that a COMPAQ BIOS (compact portable - the original), fails in weird ways if a floppy read spans a track boundary. So much for that. I don't think it's worth it to put more effort into this (like, there is no reason the HD IO should restrict the number of sectors per IO either, and I haven't had any problems with setting the sector limit to 255, even with MFM drives, but again it depends on the BIOS and it's too time consuming to do extensive testing.

[Mellvik]

The final commit fixes the floppy multitrack issue by reverting to the old code if CONFIG_HW_PCXT is defined. This will prevent multitrack HD too, which is neither necessary nor beneficial but kept for simplicity. For hard disks with odd sector count (SPT) - like 17 or 63, common values on old drives - this means that a 1k block some times requires 2 transactions.

The max sectors per IO operation - 255 - for non-XT architectures is big enough to never put a practical limit on anything and cannot break anything. Even a XT class MFM drive, which would be extremely rare on an AT+ system, can transfer 256 sectors in a single operation, but no buffer on TLVC can ever exceed 64k bytes so anything above 128 is fine in this context.

Also in this commit, the separate raw driver code and file has been eliminated, the file operation structure integrated into the 'real' driver. An overdue cleanup - thanks for the push @ghaerr.