Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

makedir() reorders files, moving system files, making MS-DOS 5 non-bootable #24

Closed
dcoshea opened this issue Apr 9, 2022 · 4 comments
Closed
Assignees
Labels
bug Something isn't working

Comments

@dcoshea
Copy link

dcoshea commented Apr 9, 2022

With pyfatfs 1.0.3, if I open a bootable MS-DOS 5 floppy as fs and then invoke:

fs.makedir("KILLER")
fs.close()

it is no longer possible to boot from the floppy. This is what Central Point PC Tools 9's Disk Editor shows after the above:

NAME     EXT     SIZE     DATE      TIME     CLU#    ARC R/O SYS HID DIR VOL
KILLER  .             0  4/09/22   08:43p    1398                    Dir
CONFIG  .SYS         63  9/15/13   04:25p    1397    Arc
FDISK   .EXE      57224  4/09/91   05:00a     996    Arc
EGA     .CPI      58873  4/09/91   05:00a     390    Arc
MEM     .EXE      39818  4/09/91   05:00a    1108    Arc
HIMEM   .SYS      11552  4/09/91   05:00a     505    Arc
IO      .SYS      33430  4/09/91   05:00a       2            Sys Hid
MIRROR  .COM      18169  4/09/91   05:00a    1186    Arc
KEYB    .COM      14986  4/09/91   05:00a     528    Arc
MSDOS   .SYS      37394  4/09/91   05:00a      68            Sys Hid
RAMDRIVE.SYS       5873  4/09/91   05:00a    1222    Arc
KEYBOARD.SYS      34697  4/09/91   05:00a     558    Arc
SHARE   .EXE      10912  4/09/91   05:00a    1234    Arc
MODE    .COM      23537  4/09/91   05:00a     626    Arc
SMARTDRV.SYS       8335  4/09/91   05:00a    1256    Arc
SETVER  .EXE      12007  4/09/91   05:00a     672    Arc
COMMAND .COM      47845  4/09/91   05:00a     142    Arc
SYS     .COM      13440  4/09/91   05:00a    1273    Arc
ANSI    .SYS       9029  4/09/91   05:00a     696    Arc
EGA     .SYS       4885  4/09/91   05:00a     236    Arc
UNDELETE.EXE      13924  4/09/91   05:00a    1300    Arc
DEBUG   .EXE      20634  4/09/91   05:00a     714    Arc
FORMAT  .COM      32911  4/09/91   05:00a     246    Arc
UNFORMAT.COM      18576  4/09/91   05:00a    1328    Arc
DOSKEY  .COM       5883  4/09/91   05:00a     755    Arc
NLSFUNC .EXE       7052  4/09/91   05:00a     311    Arc
XCOPY   .EXE      15804  4/09/91   05:00a    1365    Arc
EDLIN   .EXE      12642  4/09/91   05:00a     767    Arc
COUNTRY .SYS      17069  4/09/91   05:00a     325    Arc
AUTOEXEC.BAT         24  9/15/13   04:25p    1396    Arc
EMM386  .EXE      91742  4/09/91   05:00a     792    Arc
DISPLAY .SYS      15792  4/09/91   05:00a     359    Arc
FASTOPEN.EXE      12050  4/09/91   05:00a     972    Arc
STARTUP .             0  9/15/13   04:25p       0                        Vol
             Unused
               Unused

Note that the new directory appears first, and the files are listed in some unknown order.

This is what it looked like prior to adding the directory:

NAME     EXT     SIZE     DATE      TIME     CLU#    ARC R/O SYS HID DIR VOL
IO      .SYS      33430  4/09/91   05:00a       2            Sys Hid
MSDOS   .SYS      37394  4/09/91   05:00a      68            Sys Hid
STARTUP .             0  9/15/13   04:25p       0                        Vol
COMMAND .COM      47845  4/09/91   05:00a     142    Arc
EGA     .SYS       4885  4/09/91   05:00a     236    Arc
FORMAT  .COM      32911  4/09/91   05:00a     246    Arc
NLSFUNC .EXE       7052  4/09/91   05:00a     311    Arc
COUNTRY .SYS      17069  4/09/91   05:00a     325    Arc
DISPLAY .SYS      15792  4/09/91   05:00a     359    Arc
EGA     .CPI      58873  4/09/91   05:00a     390    Arc
HIMEM   .SYS      11552  4/09/91   05:00a     505    Arc
KEYB    .COM      14986  4/09/91   05:00a     528    Arc
KEYBOARD.SYS      34697  4/09/91   05:00a     558    Arc
MODE    .COM      23537  4/09/91   05:00a     626    Arc
SETVER  .EXE      12007  4/09/91   05:00a     672    Arc
ANSI    .SYS       9029  4/09/91   05:00a     696    Arc
DEBUG   .EXE      20634  4/09/91   05:00a     714    Arc
DOSKEY  .COM       5883  4/09/91   05:00a     755    Arc
EDLIN   .EXE      12642  4/09/91   05:00a     767    Arc
EMM386  .EXE      91742  4/09/91   05:00a     792    Arc
FASTOPEN.EXE      12050  4/09/91   05:00a     972    Arc
FDISK   .EXE      57224  4/09/91   05:00a     996    Arc
MEM     .EXE      39818  4/09/91   05:00a    1108    Arc
MIRROR  .COM      18169  4/09/91   05:00a    1186    Arc
RAMDRIVE.SYS       5873  4/09/91   05:00a    1222    Arc
SHARE   .EXE      10912  4/09/91   05:00a    1234    Arc
SMARTDRV.SYS       8335  4/09/91   05:00a    1256    Arc
SYS     .COM      13440  4/09/91   05:00a    1273    Arc
UNDELETE.EXE      13924  4/09/91   05:00a    1300    Arc
UNFORMAT.COM      18576  4/09/91   05:00a    1328    Arc
XCOPY   .EXE      15804  4/09/91   05:00a    1365    Arc
AUTOEXEC.BAT         24  9/15/13   04:25p    1396    Arc
CONFIG  .SYS         63  9/15/13   04:25p    1397    Arc
           Unused
             Unused

Note that the files are listed in some different, unknown order, but that IO.SYS and MSDOS.SYS are the first two entries. https://en.wikipedia.org/wiki/IO.SYS notes that:

The two first entries of the root directory must be allocated by IO.SYS and MSDOS.SYS, in that order.
I'm not sure if that restriction still holds in Windows 95 but it certainly seems to affect MS-DOS 5.

It would be safer to mimic what I think DOS does and just use the first available directory entry for the entry being added rather than rewriting the entire directory. I noticed that adding files one by one to a directory seems a bit slow compared to shelling out to mtools to copy all of the files in one batch, and I wonder if that is because the entire directory is being rewritten each time?

@dcoshea
Copy link
Author

dcoshea commented Apr 9, 2022

setinfo() on an existing file seems to trigger this issue too, despite not actually changing the given file's timestamp.

nathanhi added a commit that referenced this issue Apr 10, 2022
When flushing directory entries back to disk, the
files have been reordered due to a usage of `set()`
and a `walk()`-like mechanism. This had the advantage
of automatically getting rid of duplicate directory entries
on flush to disk but also had the effect that it essentially
always re-ordered files due to the nature of `set()`.

In some cases like with IO.SYS on MS-DOS, a strict ordering
is required, otherwise rendering the system unbootable.

Instead, a list of dentries is now directly passed to
the flush method, leaving ordering as-is.
@nathanhi
Copy link
Owner

Thanks for your bug report! I wasn't aware that some system-files are required in a certain order. I've made a WIP fix that seems to work well (for what I was able to test), feel free to check it out (54640fb).

image

Regarding the performance issues: Yes, rewriting everything completely back to disk all the time certainly slows PyFatFS down a fair amount and there's currently no elegant way to improve performance but it is something I have on my roadmap but performance isn't something I aimed for in the 1.0 release.

@nathanhi nathanhi self-assigned this Apr 10, 2022
@nathanhi nathanhi added the bug Something isn't working label Apr 10, 2022
@dcoshea
Copy link
Author

dcoshea commented Apr 15, 2022

I've made a WIP fix that seems to work well (for what I was able to test), feel free to check it out (54640fb).

Thanks for the quick fix! I tested with 96ae6bc (whose parent is the commit referenced above) and it works well!

performance isn't something I aimed for in the 1.0 release.

That's definitely fair.

nathanhi added a commit that referenced this issue Apr 15, 2022
When flushing directory entries back to disk, the
files have been reordered due to a usage of `set()`
and a `walk()`-like mechanism. This had the advantage
of automatically getting rid of duplicate directory entries
on flush to disk but also had the effect that it essentially
always re-ordered files due to the nature of `set()`.

In some cases like with IO.SYS on MS-DOS, a strict ordering
is required, otherwise rendering the system unbootable.

Instead, a list of dentries is now directly passed to
the flush method, leaving ordering as-is.
@nathanhi
Copy link
Owner

Fixed on master, will be in the upcoming release

nathanhi added a commit that referenced this issue Apr 15, 2022
When flushing directory entries back to disk, the
files have been reordered due to a usage of `set()`
and a `walk()`-like mechanism. This had the advantage
of automatically getting rid of duplicate directory entries
on flush to disk but also had the effect that it essentially
always re-ordered files due to the nature of `set()`.

In some cases like with IO.SYS on MS-DOS, a strict ordering
is required, otherwise rendering the system unbootable.

Instead, a list of dentries is now directly passed to
the flush method, leaving ordering as-is.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants