Make SD cards available over web workflow #8699

tannewt · 2023-12-06T00:28:01Z

This changes storage.mount() to require that a mount point exist on the parent file system.

A bug in background tasks is also fixed where the function parameter is cleared on pending callbacks during "reset".

Disk usage is shown on the directory listing and changes based on the mounted file system. Writable is also loaded per-directory.

Fixes #8108. Fixes #8690. Fixes #8107.

tannewt · 2023-12-06T00:28:36Z

The server side USB check isn't quite right. Will need to look at that when I have a chance next. Out of time for today.

RetiredWizard · 2023-12-06T04:54:40Z

I ran the test code.py from #8690 and didn't see any delays/hangs in the timer printouts and was able to copy multiple files via web workflow without the webpage crashing. Thanks!!! 😁

I successfully copied a 4Mb file which did hang the code.py file while the transfer was happening but once the transfer finished everything behaved normally.

I was able to hang the code.py running on the serial terminal by doing a page refresh on the web workflow directory listing, at which point the serial terminal was completely unresponsive, however either performing a file transfer via web workfow, deleting a file or simply changing the directory that was being displayed released the serial terminal and it once again functioned normally.

The warning about having the CIRCUITPY drive mounted on the host computer didn't seem to be displayed on the web workflow page, When I tried to transfer a file with the CIRCUITPY drive mounted the web workflow file browser didn't display an error messages but the file wasn't copied.

I don't often use the web workflow "Full Code Editor" so I'm not sure this has anything to do with this PR but when I tried to launch it the following message was displayed:
Unable to connect. The device 10.0.0.218 was not found. Be sure it is plugged in and set up properly.

I mounted/unmounted a SD card multiple times and the web workflow displayed the directory contents that you would expect. I even tried mounting a second SD card on top of an already mounted SD card's mount point. That resulted in an error but didn't cause any problem with the already mounted directory.

The total used and total size of the flash was displayed on the web workflow directory listing:
Disk usage:5.8 MB out of 5.9 MB
versus the total free/available and total size on the Linux directory screen:
Free space: 93.0 KiB (Total: 5.7 MiB)
and a DF -h:
/dev/sde1 5.7M 5.6M 93K 99% /media/retiredwizard/CIRCUITPY

When I copied another 50K the web workflow disk usage indicated that the used space was equal to the total space of 5.9MB so the available space (total - used) seems accurate to a precision of 100KB.

tannewt · 2023-12-06T18:34:25Z

I ran the test code.py from #8690 and didn't see any delays/hangs in the timer printouts and was able to copy multiple files via web workflow without the webpage crashing. Thanks!!! 😁

Yay! Thanks for testing!

I successfully copied a 4Mb file which did hang the code.py file while the transfer was happening but once the transfer finished everything behaved normally.

This is expected. The web workflow code waits for the whole file before returning to python code. I don't think this is an issue because we'll reload soon after anyway.

I was able to hang the code.py running on the serial terminal by doing a page refresh on the web workflow directory listing, at which point the serial terminal was completely unresponsive, however either performing a file transfer via web workfow, deleting a file or simply changing the directory that was being displayed released the serial terminal and it once again functioned normally.

Could you explain this in more detail please? I'm not sure if you mean the REPL or not.

The warning about having the CIRCUITPY drive mounted on the host computer didn't seem to be displayed on the web workflow page, When I tried to transfer a file with the CIRCUITPY drive mounted the web workflow file browser didn't display an error messages but the file wasn't copied.

I'll be tweaking this soon. No error exists for failed upload yet. I'll need to add that too.

I don't often use the web workflow "Full Code Editor" so I'm not sure this has anything to do with this PR but when I tried to launch it the following message was displayed: Unable to connect. The device 10.0.0.218 was not found. Be sure it is plugged in and set up properly.

The full code isn't updated for the new directory listing yet. The developer tools in your browser may have more info.

The total used and total size of the flash was displayed on the web workflow directory listing: Disk usage:5.8 MB out of 5.9 MB versus the total free/available and total size on the Linux directory screen: Free space: 93.0 KiB (Total: 5.7 MiB) and a DF -h: /dev/sde1 5.7M 5.6M 93K 99% /media/retiredwizard/CIRCUITPY

When I copied another 50K the web workflow disk usage indicated that the used space was equal to the total space of 5.9MB so the available space (total - used) seems accurate to a precision of 100KB.

I have it setup to do / 1000 so it matches SD card size better. It was a couple GB for my 32GB card.

tannewt · 2023-12-06T19:23:37Z

BLE workflow hasn't been updated so the CI failed. I have ideas on doing clearer per-mount locking so I'm going to run with that. I'm thinking about the S3 future where we have both web and ble workflows going concurrently.

RetiredWizard · 2023-12-06T19:24:06Z

Could you explain this in more detail please? I'm not sure if you mean the REPL or not.

I have a tio terminal session connected to the micro controller, doesn't matter if it's in the REPL or running a python program.

I also have the web workflow file browser open in a web browser window (http://10.0.0.218/fs/#/) .

If I hit the refresh page button on the web browser the web workflow web page refreshes but the tio serial terminal hangs and will not respond to any input including Ctrl-C, Ctrl-D

Pressing the refresh button a second time on the web browser refreshes the workflow page normally and the tio serial terminal starts processing any keys that were pressed while it was hung, but after a few seconds hangs again. (While typing this in, the serial terminal screen started responding again after maybe a minute of being frozen, I hadn't seen it release by itself before but perhaps I never let it sit long enough).

If while the USB serial terminal is frozen, you peform any file activity (delete, rename, upload) or change the folder being displayed on the web workflow page, the serial terminal is released and functions normally again.

** I waited again with the terminal screen frozen and after 1-2 minutes it released again, so there does seem to be a timeout going on.

RetiredWizard · 2023-12-06T19:57:51Z

The full code isn't updated for the new directory listing yet. The developer tools in your browser may have more info.

I'm not sure what to look for but the developer tools console simply displayed a couple status messages:

Load different workflow               index.js:37
Initializing File Transfer Client...  index.js:37

tannewt · 2023-12-07T20:39:58Z

The full code isn't updated for the new directory listing yet. The developer tools in your browser may have more info.

I'm not sure what to look for but the developer tools console simply displayed a couple status messages:

The network tab can be helpful because it shows the calls being done to the device and if they failed.

RetiredWizard · 2023-12-07T20:47:08Z

I don't see any failed calls... The 9 second delay was me entering the CircuitPython password

tannewt · 2023-12-08T19:25:23Z

This may be how the unsupported version is handled. @makermelissa would know.

makermelissa · 2023-12-08T19:36:45Z

I can't really remember because it's been about a year, but I would have thought it would at least attempt to contact it.

This changes storage.mount() to require that a mount point exist on the parent file system. A bug in background tasks is also fixed where the function parameter is cleared on pending callbacks during "reset". Disk usage is shown on the directory listing and changes based on the mounted file system. Writable is also loaded per-directory. Fixes micropython#8108. Fixes micropython#8690. Fixes micropython#8107.

tannewt · 2024-01-16T22:58:55Z

I've reworked how locking is managed. IIRC I tested the web workflow side with the rework but not BLE. Marking ready for review because the code should be close.

jepler

I have some small comments & changes requested. The new functionality is exciting.

extmod/vfs_blockdev.c

jepler · 2024-01-18T17:35:42Z

shared-module/storage/__init__.c

-            mp_raise_OSError(MP_EEXIST);
+        } else {
+            // Something with the same name doesn't exists.
+            mp_raise_RuntimeError(MP_ERROR_TEXT("Mount point missing. Create first (maybe via USB)"));


What's the consequence of making this a warning in 9 and an error in 10? Maybe something like this but less horribly verbose.

warnings_warn(&mp_type_FutureWarning, MP_ERROR_TEXT( "Mount point missing. " "This will be an error in CircuitPython 10. " "This prevents file from being available via web & bluetooth workflow. " "Create an empty directory (maybe via USB) first");

Also is it important that the placeholder be a directory, which is a requirement on Unix-style systems I have used?

If it is a warning, then the parent directory won't list the mount point. So navigating into the mount wouldn't work.

supervisor/shared/background_callback.c

supervisor/shared/bluetooth/file_transfer.c

Makes sure mount point is a directory. Fixes micropython#8110 by making it explicit that progress is per-file.

tannewt · 2024-01-18T22:57:23Z

Ok, I've made some fixes and tested over WiFi and BLE. Please look again.

jepler · 2024-01-19T01:11:36Z

build is failing :(

tannewt · 2024-01-23T01:14:28Z

Hopefully this will pass. Please push back if I'm being too aggressive with these errors. I'm not sure how else to make more space.

tannewt · 2024-01-23T17:55:39Z

@jepler Please take another look. All builds fit now.

RetiredWizard · 2024-01-23T19:14:38Z

Perhaps #8690 should be separated from this PR so it doesn't hold it up?

tannewt · 2024-01-24T18:38:55Z

Perhaps #8690 should be separated from this PR so it doesn't hold it up?

I don't want to mess with this PR anymore. I'd like to get it merged since it is ready.

RetiredWizard · 2024-01-24T19:00:26Z

Perhaps #8690 should be separated from this PR so it doesn't hold it up?

I don't want to mess with this PR anymore. I'd like to get it merged since it is ready.

That's what I was thinking too 😁, merge this but remove the link to #8690 or re-open it after this is merged.

jepler · 2024-01-25T17:45:56Z

extmod/vfs_blockdev.c

+        // CIRCUITPY-CHANGE: Pass the blockdev object into native readblocks so
+        // it has the corresponding state.
+        mp_uint_t (*f)(mp_obj_t self, uint8_t *, uint32_t, uint32_t) = (void *)(uintptr_t)self->readblocks[2];


This cast would be an undetected error if we had any other block device implementations (micropython has several) but we control them all it's probably OK.

Would you like me to add native ioctl implementations for others with native read/write? Or what?

jepler

I found one item for possible future improvement, but if it fits let's put it in and get some testing!

shared-bindings/pwmio/PWMOut.c

We already consolidated the resulting message

jepler

thanks for implementing that change

tannewt added this to the 9.0.0 milestone Dec 6, 2023

tannewt requested a review from jepler December 6, 2023 00:28

tannewt force-pushed the ww_sd_card branch from dd2cb5f to 501aac6 Compare December 6, 2023 00:29

tannewt mentioned this pull request Dec 6, 2023

Web workflow interrupts running code.py and hangs after uploading a file #8690

Closed

tannewt removed the request for review from jepler December 6, 2023 19:22

tannewt marked this pull request as draft December 6, 2023 19:22

tannewt force-pushed the ww_sd_card branch from 501aac6 to 25e862d Compare January 16, 2024 22:16

tannewt marked this pull request as ready for review January 16, 2024 22:57

tannewt requested a review from jepler January 16, 2024 22:57

Disable terminalio on thunderpack for some translations

24359f9

jepler requested changes Jan 18, 2024

View reviewed changes

Fix move, mkdir and tweak dir listing

234ce7c

Makes sure mount point is a directory. Fixes micropython#8110 by making it explicit that progress is per-file.

tannewt requested a review from jepler January 18, 2024 22:57

tannewt added 3 commits January 19, 2024 21:31

Optimize error messages

61ec328

Run make translate

6311f13

Standardize more error messages to save space

8426773

Merge remote-tracking branch 'adafruit/main' into ww_sd_card

9538e00

jepler reviewed Jan 25, 2024

View reviewed changes

jepler previously approved these changes Jan 25, 2024

View reviewed changes

shared-bindings/pwmio/PWMOut.c Outdated Show resolved Hide resolved

Consolidate PWMOUT error enum

1b25e64

We already consolidated the resulting message

tannewt dismissed jepler’s stale review via 1b25e64 January 25, 2024 18:13

tannewt requested a review from jepler January 25, 2024 18:37

jepler approved these changes Jan 25, 2024

View reviewed changes

tannewt merged commit 474dae8 into adafruit:main Jan 25, 2024
465 checks passed

tannewt deleted the ww_sd_card branch January 25, 2024 18:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make SD cards available over web workflow #8699

Make SD cards available over web workflow #8699

tannewt commented Dec 6, 2023

tannewt commented Dec 6, 2023

RetiredWizard commented Dec 6, 2023 •

edited

Loading

tannewt commented Dec 6, 2023

tannewt commented Dec 6, 2023

RetiredWizard commented Dec 6, 2023 •

edited

Loading

RetiredWizard commented Dec 6, 2023

tannewt commented Dec 7, 2023

RetiredWizard commented Dec 7, 2023

tannewt commented Dec 8, 2023

makermelissa commented Dec 8, 2023

tannewt commented Jan 16, 2024

jepler left a comment

jepler Jan 18, 2024

tannewt Jan 18, 2024

tannewt commented Jan 18, 2024

jepler commented Jan 19, 2024

tannewt commented Jan 23, 2024

tannewt commented Jan 23, 2024

RetiredWizard commented Jan 23, 2024

tannewt commented Jan 24, 2024

RetiredWizard commented Jan 24, 2024

jepler Jan 25, 2024

tannewt Jan 25, 2024

jepler left a comment

jepler left a comment

Make SD cards available over web workflow #8699

Make SD cards available over web workflow #8699

Conversation

tannewt commented Dec 6, 2023

tannewt commented Dec 6, 2023

RetiredWizard commented Dec 6, 2023 • edited Loading

tannewt commented Dec 6, 2023

tannewt commented Dec 6, 2023

RetiredWizard commented Dec 6, 2023 • edited Loading

RetiredWizard commented Dec 6, 2023

tannewt commented Dec 7, 2023

RetiredWizard commented Dec 7, 2023

tannewt commented Dec 8, 2023

makermelissa commented Dec 8, 2023

tannewt commented Jan 16, 2024

jepler left a comment

Choose a reason for hiding this comment

jepler Jan 18, 2024

Choose a reason for hiding this comment

tannewt Jan 18, 2024

Choose a reason for hiding this comment

tannewt commented Jan 18, 2024

jepler commented Jan 19, 2024

tannewt commented Jan 23, 2024

tannewt commented Jan 23, 2024

RetiredWizard commented Jan 23, 2024

tannewt commented Jan 24, 2024

RetiredWizard commented Jan 24, 2024

jepler Jan 25, 2024

Choose a reason for hiding this comment

tannewt Jan 25, 2024

Choose a reason for hiding this comment

jepler left a comment

Choose a reason for hiding this comment

jepler left a comment

Choose a reason for hiding this comment

RetiredWizard commented Dec 6, 2023 •

edited

Loading

RetiredWizard commented Dec 6, 2023 •

edited

Loading