Scanning for audio files on external shares fills up local temporary file storage

[learal]

Expected behavior

Scanning audio files on external shares should have no significant impact on the local file system of the owncloud/nextcloud server.

Actual behavior

The temporary storage location, i.e. /tmp, fills up with oc_tmp_XXXXXXXX.mp3 files.

In audioplayer/controller/scannercontroller.php the scanForAudios() method makes the following call:

$ThisFileInfo = $getID3->analyze($userView->getLocalFile($audio['path']));

From what I can gather, where you are using external storage (e.g. SMB mount, FTP mount, etc.), the getLocalFile() method will download the file from the external storage into a temporary file and return the path to that file.

There doesn't appear to be any kind of cleanup done of any temporary files created, either during or on completion of the scan. All audio files copied from external storage would be left there permanently and, I'm guessing, may actually be the files used when the user goes to play the audio file.

For me, this meant that every audio file in my collection was being copied into /tmp resulting in gigabytes of files filling up the entire partition. I had to shutdown NextCloud and manually remove about 12GB of these files from /tmp before restarting and disabling the audioplayer app.

Note regarding my setup:
I run NextCloud in a Docker container which has limited local storage as it is just supposed to hold the system files, etc. necessary to run NextCloud with all user data stored in locations mapped into the container from outside.

All my audio files are stored in a SMB share that is configured under "External Storage" in my NextCloud account.

Steps to reproduce the behavior

1. Configure an installation with a small /tmp partition and audio files on a SMB share
2. Start a scan for audio files
3. Monitor the /tmp partition during and after completion of the scan.

Possible solution (optional)

The scanning method should check whether each file returned from getLocalFile() is a temporary file, and where this is the case, once the relevant information has been extracted and added the the audio database, the file should be removed from the local file system.

Any other uses of getLocalFile() outside of the scan method should also do likewise.

Server configuration

Operating system:
Unraid 6.2.1 (Linux 4.4.23)
Docker container - lsiodev - nextcloud

Web server:
nginx

Database:
MariaDB

PHP version:
PHP 5.6.26

ownCloud/Nextcloud version: (see /status.php)
NextCloud 10.0 (stable) (9.1.0.16)

Updated from an older ownCloud/Nextcloud or fresh install:
fresh install

Where did you install Audio Player from:
NextCloud built-in apps manager

Are you using external storage, if yes which one: local/smb/sftp/...
Yes, SMB

Are you using encryption: yes/no
No

Client configuration

Operating system:

Browser:

Log

ownCloud/Nextcloud log (/data/[owncloud|nextcloud].log)

Insert your ownCloud/Nextcloud log here

[Rello]

Hello,
Thank you for the information.
I will try to replacate the issue and let you know

[Rello]

analysis reference:

/lib/private/Files/View.php => getLocalFile
/lib/private/Files/Storage/Common.php => getLocalFile
lib/private/Files/Storage/LocalTempFileTrait.php => getCachedFile
$this->toTmpFile
$tmpFile = \OC::$server->getTempManager()->getTemporaryFile($extension);

[Rello]

@learal thank you for the very valuable hint.
this was actually causing other issues which I did not recognize to come from this.

I implemented the cleanup-handling and I am testing it locally.
I will push this to a bugfix as soon as possible

[Rello]

Hello,
I did make the necessary modification to the scanner.
same logic as in lib/private/Preview/Image.php was applied
do you have a possibility to download the new /controller/scannercontroller.php and give it a try?

[learal]

Hi @Rello,
I downloaded scannercontroller.php to my system to try it out. I applied the change to the deployed app, and restarted NextCloud.

I reset the music library and then started the scan, but it doesn't seem to get past the "Start scanning ..." message. After 45mins I stopped it, restarted everything and tried again and it's doing the same.

I've been looking for logs to determine what might be happening, but haven't seen anything relevant.

Where would that normally log to? Is there any additional debug logging I can switch on?

Thanks,
Alan.

[Rello]

Hello Alan,

please check owncloud.log in your nextcloud data folder. it should show what the problem is

[learal]

Hi @Rello,
Sorry for not getting back to you sooner.

I'm still having the same problem, i.e. every time I click the "Scan for new audio files" (circular arrows icon) the scan starting dialog displays and then nothing else happens.

I've checked the /data/nextcloud.log and there's nothing being logged there from audioplayer at all.

The main config.php has 'loglevel' => 0, also I added 'audioplayer_debug' => true, but there's still nothing being logged from audioplayer at all.

I tried adding some log lines into the scanForAudios() method, but those aren't being logged either. It's like the method is not being called for some reason.

Actually, just digging around in the app.js file and I think that may be commented out! See around line 1855. Is the initial import triggered elsewhere, maybe when first accessed and then the icon only runs the update, which is commented out?

Thanks,
Alan.

[learal]

Hi @Rello,

Please disregard my last message. After looking around the code a bit more I noticed that the dialog that pops ups, i.e. "Scan for audio files" actually has a button on it that needs clicking to start scanning.

With the default colour scheme on my monitor, combined with whatever angle I normally look at the screen, the box around the button is completely invisible. If I drop my head right down to look straight at that part of the screen it becomes visible !!! (I assumed clicking the update icon actually started the process.)

Apologies for that.

The scan is underway and there appears to be only a single mp3 file written to /tmp at any given time.
However, there seems to be an additional problem with the temporary files, for each mp3 file being imported, two files are generated:

-rw------- 1 abc abc   0 Oct 11 15:24 oc_tmp_khhjNm
-rw------- 1 abc abc 11M Oct 11 15:24 oc_tmp_khhjNm-.mp3

The mp3 one is being removed correctly but the other files are remaining, i.e. the zero-byte ones. I've a couple of thousand there so far.

Alan.

[Rello]

Hello,
thank you for the testing. good news.
I also recognized the 0 byte files. But as they should not be an issue, i focused on the others first (nevertheless will fix this also)

did your scan finish?
because when it finishes, it will cleanup all temp-files of that php sessions and also the 0s will be gone.
can you confirm?

[learal]

Hi @Rello,
Yes, the scan finished fine and all the zero byte temp files are gone.
Thanks,
Alan.

[Rello]

=> next release