Skip to content

FileNotFoundError when updating mtime of files in file cache #1675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
NeoLegends opened this issue Jan 16, 2025 · 8 comments
Closed

FileNotFoundError when updating mtime of files in file cache #1675

NeoLegends opened this issue Jan 16, 2025 · 8 comments
Assignees
@NeoLegends
Labels
bug

Comments

@NeoLegends
Copy link
Member

NeoLegends commented Jan 16, 2025 •
edited by albertz
Loading

Sadly only a part of the error message remains, and the rest of the log is lost:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/nas/models/asr/[...]/work/i6_core/tools/git/CloneGitRepositoryJob.LD5f1wKK7LPo/output/returnn/returnn/util/file_cache.py", line 479, in run
    os.utime(filename, None)
FileNotFoundError: [Errno 2] No such file or directory

First we should find out which file it is failing to touch.
@NeoLegends NeoLegends added the bug label Jan 16, 2025
@NeoLegends NeoLegends self-assigned this Jan 16, 2025
@NeoLegends
Copy link
Member Author

@Icemole If this occurs again, kindly post the full log here. :)

NeoLegends added a commit that referenced this issue Jan 16, 2025
@NeoLegends
FileCache: log file name on mtime update failure
dcae190 
Part of #1675
@albertz
Copy link
Member

albertz commented Jan 16, 2025

First we should find out which file it is failing to touch.

I'm not sure that this really helps us here. But certainly having the full log would be helpful.

Btw, why is filename not printed? It should via #1674, as now it should use better_exchook in the thread, or not?

@NeoLegends
Copy link
Member Author

I think the log came from befire #1674.

@hannah220
Copy link

I encountered the same issue:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()FileCache: Copy file /nas/data/speech/IT_IT/16kHz/AMARA-crawl/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-5/AMARA-crawl-batch.1.v2.hdf.4 to cache

  File "/nas/models/asr/am/IT_IT/16kHz/2025-03-04-hyoshimochi-conformer-mbw-ffnn-alignments/work/i6_core/tools/git/CloneGitRepositoryJob.LD5f1wKK7LPo/output/returnn/returnn/util/file_cache.py", line 479, in run
    os.utime(filename, None)
FileNotFoundError: [Errno 2] No such file or directory

Also, in the same log file, cached training HDF files are being deleted by returnn because there is not enough free disk space, late causing error:

FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/AMARA-crawl/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-5/AMARA-crawl-batch.1.v2.hdf.1, 
size 9.2MB. Still need more space, file is 0.8 hours old. After deletion, have 1.0GB free space.

FileNotFoundError: [Errno 2] Unable to synchronously open file (unable to open file: 
name = '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/AMARA-crawl/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-5/AMARA-crawl-batch.1.v2.hdf.1', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

@hannah220
Copy link

Full log

log.run.1.txt

@NeoLegends
Copy link
Member Author

NeoLegends commented Apr 11, 2025 •
edited
Loading

Hm, so this error keeps occuring. Looking at the following code, one thing stood out to me:

returnn/returnn/util/file_cache.py

Lines 100 to 134 in cf50800

def get_file(self, src_filename: str) -> str:
"""
Get cached file.
This will copy the file to the cache directory if it is not already there.
This will also make sure that the file is not removed from the cache directory
via the _touch_files_thread
until you call :func:`release_file`.
:param src_filename: source file to copy (if it is not already in the cache).
:return: cached file path (in the cache directory)
"""
dst_filename = self._get_dst_filename(src_filename)
last_error = None
for try_nr in range(self._num_tries):
if try_nr > 0:
print(
f"FileCache: Ignoring error while copying {dst_filename}: {type(last_error).__name__}: {last_error}"
)
time.sleep(1)
try:
self._copy_file_if_needed(src_filename, dst_filename)
break
except OSError as e:
if e.errno == errno.ENOSPC:
last_error = e
else:
raise e
if last_error is not None:
raise last_error
info_file = self._get_info_filename(dst_filename)
os.utime(dst_filename, None)
os.utime(info_file, None)
# protect info file from tempreaper, which looks at the mtime
self._touch_files_thread.files_extend([dst_filename, info_file])
return dst_filename

We only add the files to the mtime update thread after we think we have copied them over to the local disk. In the situation where error reoccured the file was 24h old, so its mtime was stale. One process found the file, thought it would still have access to the file, proceeded to update its mtime and then tried to read it. Meanwhile, a different process deleted it, because it thought the file was stale, and it did not see the mtime update.

I think the problem occurs because we no longer use a cache-dir wide lock for performance reasons (which is still the right thing to do). Maybe it's enough to lock globally around the cleanup operation though. In that case, if one process was currently doing cleanup, the other processes would wait, and if their file was deleted, they would know about it.

Latest log excerpt:

ep 21 train, step 36, ce_0 1.042, fer_0 0.299, ce_1 0.827, fer_1 0.250, ce 0.729, fer 0.223, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.718 sec/step, elapsed 0:00:31, exp. remaining 0:50:10, complete 1.03%
ep 21 train, step 36, ce_0 0.932, fer_0 0.272, ce_1 0.698, fer_1 0.215, ce 0.604, fer 0.192, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.718 sec/step, elapsed 0:00:31, exp. remaining 0:44:03, complete 1.17%
ep 21 train, step 36, ce_0 1.136, fer_0 0.303, ce_1 0.965, fer_1 0.277, ce 0.894, fer 0.258, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.727 sec/step, elapsed 0:00:31, exp. remaining 0:48:08, complete 1.07%
ep 21 train, step 37, ce_0 0.659, fer_0 0.207, ce_1 0.570, fer_1 0.185, ce 0.503, fer 0.165, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.715 sec/step, elapsed 0:00:31, exp. remaining 0:51:07, complete 1.03%
ep 21 train, step 37, ce_0 0.768, fer_0 0.228, ce_1 0.652, fer_1 0.195, ce 0.575, fer 0.185, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.717 sec/step, elapsed 0:00:31, exp. remaining 0:50:49, complete 1.04%
ep 21 train, step 37, ce_0 1.030, fer_0 0.267, ce_1 0.884, fer_1 0.244, ce 0.790, fer 0.220, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.717 sec/step, elapsed 0:00:31, exp. remaining 0:44:36, complete 1.18%
ep 21 train, step 37, ce_0 1.070, fer_0 0.285, ce_1 0.889, fer_1 0.248, ce 0.774, fer 0.227, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.722 sec/step, elapsed 0:00:32, exp. remaining 0:48:56, complete 1.08%
ep 21 train, step 38, ce_0 0.843, fer_0 0.253, ce_1 0.707, fer_1 0.215, ce 0.638, fer 0.197, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.715 sec/step, elapsed 0:00:32, exp. remaining 0:51:32, complete 1.04%
ep 21 train, step 38, ce_0 1.245, fer_0 0.326, ce_1 1.068, fer_1 0.285, ce 1.012, fer 0.270, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.719 sec/step, elapsed 0:00:32, exp. remaining 0:44:59, complete 1.19%
ep 21 train, step 38, ce_0 1.399, fer_0 0.323, ce_1 1.281, fer_1 0.294, ce 1.201, fer 0.278, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.729 sec/step, elapsed 0:00:32, exp. remaining 0:51:31, complete 1.05%
ep 21 train, step 38, ce_0 1.240, fer_0 0.315, ce_1 1.087, fer_1 0.278, ce 1.001, fer 0.264, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.725 sec/step, elapsed 0:00:32, exp. remaining 0:49:41, complete 1.09%
ep 21 train, step 39, ce_0 1.180, fer_0 0.317, ce_1 0.972, fer_1 0.285, ce 0.873, fer 0.261, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.712 sec/step, elapsed 0:00:33, exp. remaining 0:52:20, complete 1.05%
ep 21 train, step 39, ce_0 1.308, fer_0 0.320, ce_1 1.147, fer_1 0.289, ce 1.043, fer 0.270, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.716 sec/step, elapsed 0:00:33, exp. remaining 0:45:30, complete 1.21%
ep 21 train, step 39, ce_0 0.788, fer_0 0.246, ce_1 0.666, fer_1 0.217, ce 0.616, fer 0.199, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.717 sec/step, elapsed 0:00:33, exp. remaining 0:52:12, complete 1.06%
ep 21 train, step 39, ce_0 0.954, fer_0 0.269, ce_1 0.784, fer_1 0.236, ce 0.729, fer 0.219, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.729 sec/step, elapsed 0:00:33, exp. remaining 0:50:15, complete 1.10%
ep 21 train, step 40, ce_0 0.949, fer_0 0.277, ce_1 0.811, fer_1 0.254, ce 0.675, fer 0.220, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.724 sec/step, elapsed 0:00:34, exp. remaining 0:53:01, complete 1.06%
ep 21 train, step 40, ce_0 1.248, fer_0 0.334, ce_1 0.977, fer_1 0.269, ce 0.844, fer 0.244, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.721 sec/step, elapsed 0:00:34, exp. remaining 0:46:01, complete 1.22%
ep 21 train, step 40, ce_0 1.093, fer_0 0.300, ce_1 0.921, fer_1 0.272, ce 0.816, fer 0.244, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.723 sec/step, elapsed 0:00:34, exp. remaining 0:52:49, complete 1.07%
ep 21 train, step 40, ce_0 1.185, fer_0 0.307, ce_1 1.031, fer_1 0.273, ce 0.916, fer 0.246, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.730 sec/step, elapsed 0:00:34, exp. remaining 0:50:58, complete 1.11%
ep 21 train, step 41, ce_0 1.188, fer_0 0.318, ce_1 1.027, fer_1 0.284, ce 0.922, fer 0.261, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.717 sec/step, elapsed 0:00:34, exp. remaining 0:53:44, complete 1.07%
ep 21 train, step 41, ce_0 1.068, fer_0 0.290, ce_1 0.843, fer_1 0.244, ce 0.735, fer 0.214, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.729 sec/step, elapsed 0:00:34, exp. remaining 0:46:41, complete 1.23%
ep 21 train, step 41, ce_0 1.067, fer_0 0.315, ce_1 0.861, fer_1 0.264, ce 0.812, fer 0.254, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.722 sec/step, elapsed 0:00:34, exp. remaining 0:53:28, complete 1.08%
ep 21 train, step 41, ce_0 0.993, fer_0 0.285, ce_1 0.841, fer_1 0.256, ce 0.760, fer 0.231, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.735 sec/step, elapsed 0:00:35, exp. remaining 0:51:35, complete 1.12%
ep 21 train, step 42, ce_0 0.836, fer_0 0.260, ce_1 0.724, fer_1 0.219, ce 0.668, fer 0.205, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.728 sec/step, elapsed 0:00:35, exp. remaining 0:54:14, complete 1.08%
ep 21 train, step 42, ce_0 1.510, fer_0 0.362, ce_1 1.278, fer_1 0.311, ce 1.181, fer 0.299, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.713 sec/step, elapsed 0:00:35, exp. remaining 0:47:24, complete 1.23%
ep 21 train, step 42, ce_0 1.158, fer_0 0.305, ce_1 0.949, fer_1 0.263, ce 0.841, fer 0.239, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.726 sec/step, elapsed 0:00:35, exp. remaining 0:53:58, complete 1.09%
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-27/commonvoice-validated.v2.hdf.7
FileCache: Copy file /nas/data/speech/IT_IT/16kHz/raiplay/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-71/raiplay-batch.1.v2.hdf.48 to cache
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.140, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.140: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.140.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.7, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.7: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.7'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.7: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.7.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130, size 9.3MB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.140, size 8.1MB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.140: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.140'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.140: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.140.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.7, size 7.4MB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.7: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.7'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.7: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.7.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.16, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.16: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.16'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.16: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.16.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.18, size 1.0GB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.18: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.18'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.18: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.18.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.16, size 8.8MB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.16: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.16'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.16: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.16.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.18, size 7.3MB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.18: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.18'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.18: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.18.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.128, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.128: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.128.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.74, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 30.3GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.74: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.74'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.74: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.74.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.30, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 30.3GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.30: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.30'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.30: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.30.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149, size 8.5MB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.128, size 8.2MB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.128: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.128'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.128: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.128.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.74, size 7.8MB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.74: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.74'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.74: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.74.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.30, size 7.4MB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.30: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.30'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.30: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.30.returnn-info'
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.52
FileCache: Copy file /nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.100 to cache
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.17
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.86
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-27/commonvoice-validated.v2.hdf.19
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.29
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-27/commonvoice-validated.v2.hdf.27
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.85
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-raw_wav.16kHz.split-27/commonvoice-validated.v2.hdf.7
FileCache: Copy file /nas/data/speech/IT_IT/16kHz/raiplay/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-71/raiplay-batch.1.v2.hdf.48 to cache
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.52
FileCache: Copy file /nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.100 to cache
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.17
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.86
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-raw_wav.16kHz.split-27/commonvoice-validated.v2.hdf.19
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.29
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-raw_wav.16kHz.split-27/commonvoice-validated.v2.hdf.27
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.85
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-raw_wav.16kHz.split-27/commonvoice-validated.v2.hdf.7
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/raiplay/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-71/raiplay-batch.1.v2.hdf.48
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.52
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.100
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.17
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.86
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-raw_wav.16kHz.split-27/commonvoice-validated.v2.hdf.19
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.29
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-raw_wav.16kHz.split-27/commonvoice-validated.v2.hdf.27
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.85
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-27/commonvoice-validated.v2.hdf.7
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/raiplay/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-71/raiplay-batch.1.v2.hdf.48
ep 21 train, step 42, ce_0 1.154, fer_0 0.297, ce_1 0.997, fer_1 0.269, ce 0.951, fer 0.264, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.735 sec/step, elapsed 0:00:35, exp. remaining 0:52:15, complete 1.13%
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.52
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.100
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.17
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.86
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-27/commonvoice-validated.v2.hdf.19
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.29
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-27/commonvoice-validated.v2.hdf.27
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.85
Reading sequence list for MetaDataset 'dataset_id139694414567584' from sub-dataset 'dataset_id139694414567584_alignments'
Reinitialize dataset seq order for epoch 1.
ep 21 train, step 43, ce_0 1.202, fer_0 0.308, ce_1 1.031, fer_1 0.281, ce 0.967, fer 0.269, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.736 sec/step, elapsed 0:00:36, exp. remaining 0:54:43, complete 1.09%
ep 21 train, step 43, ce_0 1.197, fer_0 0.294, ce_1 1.064, fer_1 0.266, ce 0.975, fer 0.260, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.708 sec/step, elapsed 0:00:36, exp. remaining 0:48:04, complete 1.24%
ep 21 train, step 43, ce_0 1.599, fer_0 0.374, ce_1 1.428, fer_1 0.348, ce 1.330, fer 0.339, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.726 sec/step, elapsed 0:00:36, exp. remaining 0:54:27, complete 1.10%
ep 21 train, step 43, ce_0 1.000, fer_0 0.279, ce_1 0.830, fer_1 0.235, ce 0.737, fer 0.212, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.738 sec/step, elapsed 0:00:36, exp. remaining 0:52:57, complete 1.14%
ep 21 train, step 44, ce_0 1.198, fer_0 0.333, ce_1 1.054, fer_1 0.301, ce 0.988, fer 0.292, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.715 sec/step, elapsed 0:00:36, exp. remaining 0:48:49, complete 1.25%
ep 21 train, step 44, ce_0 1.022, fer_0 0.278, ce_1 0.882, fer_1 0.246, ce 0.820, fer 0.235, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.742 sec/step, elapsed 0:00:37, exp. remaining 0:55:23, complete 1.10%
ep 21 train, step 44, ce_0 1.029, fer_0 0.274, ce_1 0.898, fer_1 0.255, ce 0.810, fer 0.244, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.725 sec/step, elapsed 0:00:37, exp. remaining 0:55:08, complete 1.11%
ep 21 train, step 44, ce_0 0.824, fer_0 0.247, ce_1 0.694, fer_1 0.218, ce 0.649, fer 0.210, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.737 sec/step, elapsed 0:00:37, exp. remaining 0:53:32, complete 1.15%
ep 21 train, step 45, ce_0 0.984, fer_0 0.256, ce_1 0.853, fer_1 0.233, ce 0.778, fer 0.218, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.744 sec/step, elapsed 0:00:37, exp. remaining 0:49:28, complete 1.25%
ep 21 train, step 45, ce_0 1.321, fer_0 0.332, ce_1 1.116, fer_1 0.290, ce 0.971, fer 0.274, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.743 sec/step, elapsed 0:00:37, exp. remaining 0:56:00, complete 1.11%
ep 21 train, step 45, ce_0 1.125, fer_0 0.279, ce_1 1.001, fer_1 0.253, ce 0.922, fer 0.245, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.731 sec/step, elapsed 0:00:37, exp. remaining 0:55:39, complete 1.12%
ep 21 train, step 45, ce_0 0.822, fer_0 0.242, ce_1 0.689, fer_1 0.217, ce 0.672, fer 0.211, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.739 sec/step, elapsed 0:00:37, exp. remaining 0:53:59, complete 1.16%
MEMORY: sub proc CFD worker 22(3410798) increased RSS: rss=577.4MB pss=387.0MB uss=357.7MB shared=219.6MB
MEMORY: total (main 3362628, 2025-04-09, 19:29:42, 8 procs): pss=4.8GB uss=4.4GB
ep 21 train, step 46, ce_0 0.819, fer_0 0.257, ce_1 0.714, fer_1 0.227, ce 0.645, fer 0.214, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.736 sec/step, elapsed 0:00:38, exp. remaining 0:50:08, complete 1.26%
ep 21 train, step 46, ce_0 0.813, fer_0 0.243, ce_1 0.693, fer_1 0.212, ce 0.610, fer 0.201, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.734 sec/step, elapsed 0:00:38, exp. remaining 0:56:41, complete 1.12%
ep 21 train, step 46, ce_0 0.986, fer_0 0.265, ce_1 0.806, fer_1 0.226, ce 0.755, fer 0.221, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.725 sec/step, elapsed 0:00:38, exp. remaining 0:56:22, complete 1.13%
ep 21 train, step 46, ce_0 1.108, fer_0 0.296, ce_1 0.927, fer_1 0.268, ce 0.808, fer 0.243, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.732 sec/step, elapsed 0:00:38, exp. remaining 0:54:39, complete 1.17%
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.6
FileCache: Copy file /nas/data/speech/IT_IT/16kHz/raiplay/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-71/raiplay-batch.1.v2.hdf.71 to cache
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.140, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.140: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.140'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.7, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.7: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.7.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130, size 9.3MB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.140, size 8.1MB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.140: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.140'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.140: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.140.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.7, size 7.4MB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.7: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.7'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.7: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.7.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.16, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.16: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.16'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.16: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.16.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.18, size 1.0GB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.18: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.18'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.16, size 8.8MB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.18, size 7.3MB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.128, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 30.3GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.128: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.128'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.128: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.128.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.74, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 30.3GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.74: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.74'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.74: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.74.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.30, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 30.3GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.30: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.30'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149, size 8.5MB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.128, size 8.2MB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.74, size 7.8MB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.30, size 7.4MB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.20
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.96
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.92
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.107
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.32
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.15
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.94
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/raiplay/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-71/raiplay-batch.1.v2.hdf.33
FileCache: Copy file /nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6 to cache
FileCache: failed updating mtime of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.6: [Errno 2] No such file or directory
FileCache: Copy file /nas/data/speech/IT_IT/16kHz/raiplay/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-71/raiplay-batch.1.v2.hdf.71 to cache
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.20
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.96
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.92
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.107
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.32
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.15
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.94
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/raiplay/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-71/raiplay-batch.1.v2.hdf.33
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/raiplay/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-71/raiplay-batch.1.v2.hdf.71
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.20
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.96
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.92
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.107
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.32
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.15
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.94
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/raiplay/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-71/raiplay-batch.1.v2.hdf.33
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.6
Process DistributeFilesDataset train ep 21 worker ep 22:
ep 21 train, step 47, ce_0 1.655, fer_0 0.361, ce_1 1.437, fer_1 0.320, ce 1.321, fer 0.307, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.726 sec/step, elapsed 0:00:39, exp. remaining 0:50:44, complete 1.27%
ep 21 train, step 47, ce_0 1.081, fer_0 0.291, ce_1 0.946, fer_1 0.256, ce 0.869, fer 0.239, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.725 sec/step, elapsed 0:00:39, exp. remaining 0:57:04, complete 1.13%
ep 21 train, step 47, ce_0 0.914, fer_0 0.273, ce_1 0.769, fer_1 0.237, ce 0.680, fer 0.225, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.736 sec/step, elapsed 0:00:39, exp. remaining 0:57:12, complete 1.13%
EXCEPTION
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in BaseProcess._bootstrap
    line: self.run()
    locals:
      self = <local> <NonDaemonicSpawnProcess name='DistributeFilesDataset train ep 21 worker ep 22' parent=3363027 started>
      self.run = <local> <bound method BaseProcess.run of <NonDaemonicSpawnProcess name='DistributeFilesDataset train ep 21 worker ep 22' parent=3363027 started>>
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in BaseProcess.run
    line: self._target(*self._args, **self._kwargs)
    locals:
      self = <local> <NonDaemonicSpawnProcess name='DistributeFilesDataset train ep 21 worker ep 22' parent=3363027 started>
      self._target = <local> <function _worker_proc_loop at 0x7fbfc059eb90>
      self._args = <local> (22, 100, {'class': 'PostprocessingDataset', 'dataset': {'class': 'MetaDataset', 'datasets': {'features': {'class': 'HDFDataset', 'files': [CachedFile(filename='/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6'), ...
      self._kwargs = <local> {}
  File "/nas/models/asr/am/IT_IT/16kHz/2025-03-04-hyoshimochi-conformer-mbw-ffnn-alignments/work/i6_core/tools/git/CloneGitRepositoryJob.8WnBjcjotoot/output/returnn/returnn/datasets/distrib_files.py", line 588, in _worker_proc_loop
    line: dataset = init_dataset(dataset_dict)
    locals:
      dataset = <not found>
      init_dataset = <global> <function init_dataset at 0x7fc0daf15360>
      dataset_dict = <local> {'class': 'PostprocessingDataset', 'dataset': {'class': 'MetaDataset', 'datasets': {'features': {'class': 'HDFDataset', 'files': [CachedFile(filename='/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6'), CachedFile..., len = 6
  File "/nas/models/asr/am/IT_IT/16kHz/2025-03-04-hyoshimochi-conformer-mbw-ffnn-alignments/work/i6_core/tools/git/CloneGitRepositoryJob.8WnBjcjotoot/output/returnn/returnn/datasets/basic.py", line 1571, in init_dataset
    line: obj = clazz(**kwargs)
    locals:
      obj = <not found>
      clazz = <local> <class 'returnn.datasets.postprocessing.PostprocessingDataset'>
      kwargs = <local> {'dataset': {'class': 'MetaDataset', 'datasets': {'features': {'class': 'HDFDataset', 'files': ['/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6', '/ssd/hyoshimochi/returnn/file...
  File "/nas/models/asr/am/IT_IT/16kHz/2025-03-04-hyoshimochi-conformer-mbw-ffnn-alignments/work/i6_core/tools/git/CloneGitRepositoryJob.8WnBjcjotoot/output/returnn/returnn/datasets/postprocessing.py", line 149, in PostprocessingDataset.__init__
    line: self._dataset = init_dataset(self._dataset_def, parent_dataset=self)
    locals:
      self = <local> <PostprocessingDataset 'dataset_id140466284117392' epoch=None>
      self._dataset = <local> !AttributeError: 'PostprocessingDataset' object has no attribute '_dataset'
      init_dataset = <global> <function init_dataset at 0x7fc0daf15360>
      self._dataset_def = <local> {'class': 'MetaDataset', 'datasets': {'features': {'class': 'HDFDataset', 'files': ['/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6', '/ssd/hyoshimochi/returnn/file_cache/nas/d...
      parent_dataset = <not found>
  File "/nas/models/asr/am/IT_IT/16kHz/2025-03-04-hyoshimochi-conformer-mbw-ffnn-alignments/work/i6_core/tools/git/CloneGitRepositoryJob.8WnBjcjotoot/output/returnn/returnn/datasets/basic.py", line 1571, in init_dataset
    line: obj = clazz(**kwargs)
    locals:
      obj = <not found>
      clazz = <local> <class 'returnn.datasets.meta.MetaDataset'>
      kwargs = <local> {'datasets': {'features': {'class': 'HDFDataset', 'files': ['/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6', '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/r..., len = 6
  File "/nas/models/asr/am/IT_IT/16kHz/2025-03-04-hyoshimochi-conformer-mbw-ffnn-alignments/work/i6_core/tools/git/CloneGitRepositoryJob.8WnBjcjotoot/output/returnn/returnn/datasets/meta.py", line 250, in MetaDataset.__init__
    line: self.datasets = {
              key: init_dataset(datasets[key], extra_kwargs={"name": "%s_%s" % (self.name, key)}, parent_dataset=self)
              for key in self.dataset_keys
          }  # type: typing.Dict[str,Dataset]
    locals:
      self = <local> <MetaDataset 'dataset_id140461541005328' epoch=None>
      self.datasets = <local> !AttributeError: 'MetaDataset' object has no attribute 'datasets'
      key = <not found>
      init_dataset = <global> <function init_dataset at 0x7fc0daf15360>
      datasets = <local> {'features': {'class': 'HDFDataset', 'files': ['/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6', '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/raiplay/corpus...
      extra_kwargs = <not found>
      self.name = <local> 'dataset_id140461541005328', len = 25
      parent_dataset = <not found>
      self.dataset_keys = <local> {'features', 'alignments'}, len = 2
  File "/nas/models/asr/am/IT_IT/16kHz/2025-03-04-hyoshimochi-conformer-mbw-ffnn-alignments/work/i6_core/tools/git/CloneGitRepositoryJob.8WnBjcjotoot/output/returnn/returnn/datasets/meta.py", line 251, in <dictcomp>
    line: key: init_dataset(datasets[key], extra_kwargs={"name": "%s_%s" % (self.name, key)}, parent_dataset=self)
    locals:
      key = <local> 'alignments', len = 10
      init_dataset = <global> <function init_dataset at 0x7fc0daf15360>
      datasets = <local> {'features': {'class': 'HDFDataset', 'files': ['/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6', '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/raiplay/corpus...
      extra_kwargs = <not found>
      self = <local> <MetaDataset 'dataset_id140461541005328' epoch=None>
      self.name = <local> 'dataset_id140461541005328', len = 25
      parent_dataset = <not found>
  File "/nas/models/asr/am/IT_IT/16kHz/2025-03-04-hyoshimochi-conformer-mbw-ffnn-alignments/work/i6_core/tools/git/CloneGitRepositoryJob.8WnBjcjotoot/output/returnn/returnn/datasets/basic.py", line 1571, in init_dataset
    line: obj = clazz(**kwargs)
    locals:
      obj = <not found>
      clazz = <local> <class 'returnn.datasets.hdf.HDFDataset'>
      kwargs = <local> {'files': ['/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.6', '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/raiplay/corpus/batch..., len = 6
  File "/nas/models/asr/am/IT_IT/16kHz/2025-03-04-hyoshimochi-conformer-mbw-ffnn-alignments/work/i6_core/tools/git/CloneGitRepositoryJob.8WnBjcjotoot/output/returnn/returnn/datasets/hdf.py", line 60, in HDFDataset.__init__
    line: self.add_file(fn)
    locals:
      self = <local> <HDFDataset 'dataset_id140461541005328_alignments' epoch=None>
      self.add_file = <local> <bound method HDFDataset.add_file of <HDFDataset 'dataset_id140461541005328_alignments' epoch=None>>
      fn = <local> '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.6', len = 197
  File "/nas/models/asr/am/IT_IT/16kHz/2025-03-04-hyoshimochi-conformer-mbw-ffnn-alignments/work/i6_core/tools/git/CloneGitRepositoryJob.8WnBjcjotoot/output/returnn/returnn/datasets/hdf.py", line 94, in HDFDataset.add_file
    line: fin = h5py.File(filename, "r")
    locals:
      fin = <not found>
      h5py = <local> <module 'h5py' from '/usr/local/lib/python3.10/dist-packages/h5py/__init__.py'>
      h5py.File = <local> <class 'h5py._hl.files.File'>
      filename = <local> '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.6', len = 197
  File "/usr/local/lib/python3.10/dist-packages/h5py/_hl/files.py", line 561, in File.__init__
    line: fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
    locals:
      fid = <not found>
      make_fid = <global> <function make_fid at 0x7fbfc0802ef0>
      name = <local> b'/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.6', len = 197
      mode = <local> 'r'
      userblock_size = <local> None
      fapl = <local> <h5py.h5p.PropFAID object at 0x7fbfc0264090>
      fcpl = <local> None
      swmr = <local> False
  File "/usr/local/lib/python3.10/dist-packages/h5py/_hl/files.py", line 235, in make_fid
    line: fid = h5f.open(name, flags, fapl=fapl)
    locals:
      fid = <not found>
      h5f = <global> <module 'h5py.h5f' from '/usr/local/lib/python3.10/dist-packages/h5py/h5f.cpython-310-x86_64-linux-gnu.so'>
      h5f.open = <global> <cyfunction open at 0x7fbfc07a5080>
      name = <local> b'/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.6', len = 197
      flags = <local> 0
      fapl = <local> <h5py.h5p.PropFAID object at 0x7fbfc0264090>
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
    -- code not available --
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
    -- code not available --
  File "h5py/h5f.pyx", line 102, in h5py.h5f.open
    -- code not available --
FileNotFoundError: [Errno 2] Unable to synchronously open file (unable to open file: name = '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.6', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
MEMORY: sub proc CFD worker 22(3410797) increased RSS: rss=493.8MB pss=301.2MB uss=272.4MB shared=221.4MB
MEMORY: total (main 3362629, 2025-04-09, 19:29:43, 8 procs): pss=4.5GB uss=4.1GB
ep 21 train, step 47, ce_0 0.886, fer_0 0.243, ce_1 0.752, fer_1 0.205, ce 0.682, fer 0.195, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.731 sec/step, elapsed 0:00:39, exp. remaining 0:55:22, complete 1.17%
Reinitialize dataset seq order for epoch 1.
ep 21 train, step 48, ce_0 0.881, fer_0 0.252, ce_1 0.720, fer_1 0.222, ce 0.627, fer 0.196, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.722 sec/step, elapsed 0:00:39, exp. remaining 0:57:51, complete 1.14%
ep 21 train, step 48, ce_0 1.268, fer_0 0.333, ce_1 1.040, fer_1 0.292, ce 0.954, fer 0.268, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.733 sec/step, elapsed 0:00:39, exp. remaining 0:51:24, complete 1.28%
ep 21 train, step 48, ce_0 0.934, fer_0 0.265, ce_1 0.852, fer_1 0.246, ce 0.763, fer 0.231, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.730 sec/step, elapsed 0:00:39, exp. remaining 0:57:55, complete 1.14%
ep 21 train, step 48, ce_0 1.077, fer_0 0.289, ce_1 0.911, fer_1 0.257, ce 0.820, fer 0.243, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.727 sec/step, elapsed 0:00:40, exp. remaining 0:56:00, complete 1.18%
ep 21 train, step 49, ce_0 1.067, fer_0 0.288, ce_1 0.929, fer_1 0.261, ce 0.870, fer 0.242, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.726 sec/step, elapsed 0:00:40, exp. remaining 0:51:53, complete 1.29%
ep 21 train, step 49, ce_0 1.185, fer_0 0.333, ce_1 1.014, fer_1 0.291, ce 0.964, fer 0.284, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.731 sec/step, elapsed 0:00:40, exp. remaining 0:58:29, complete 1.15%
ep 21 train, step 49, ce_0 0.872, fer_0 0.236, ce_1 0.749, fer_1 0.207, ce 0.703, fer 0.203, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.724 sec/step, elapsed 0:00:40, exp. remaining 0:58:27, complete 1.15%
ep 21 train, step 49, ce_0 1.181, fer_0 0.310, ce_1 1.032, fer_1 0.287, ce 0.956, fer 0.260, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.725 sec/step, elapsed 0:00:40, exp. remaining 0:56:34, complete 1.19%
ep 21 train, step 50, ce_0 1.361, fer_0 0.347, ce_1 1.135, fer_1 0.298, ce 1.001, fer 0.276, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.727 sec/step, elapsed 0:00:41, exp. remaining 0:52:21, complete 1.30%
ep 21 train, step 50, ce_0 0.957, fer_0 0.264, ce_1 0.780, fer_1 0.233, ce 0.732, fer 0.225, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.725 sec/step, elapsed 0:00:41, exp. remaining 0:59:03, complete 1.16%
ep 21 train, step 50, ce_0 1.082, fer_0 0.274, ce_1 0.901, fer_1 0.241, ce 0.840, fer 0.231, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.723 sec/step, elapsed 0:00:41, exp. remaining 0:58:47, complete 1.16%
ep 21 train, step 50, ce_0 1.007, fer_0 0.279, ce_1 0.842, fer_1 0.244, ce 0.768, fer 0.233, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.730 sec/step, elapsed 0:00:41, exp. remaining 0:57:05, complete 1.20%
ep 21 train, step 51, ce_0 1.710, fer_0 0.397, ce_1 1.507, fer_1 0.366, ce 1.411, fer 0.352, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.727 sec/step, elapsed 0:00:42, exp. remaining 0:52:50, complete 1.31%
ep 21 train, step 51, ce_0 1.027, fer_0 0.295, ce_1 0.847, fer_1 0.258, ce 0.721, fer 0.236, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.724 sec/step, elapsed 0:00:42, exp. remaining 0:59:25, complete 1.17%
ep 21 train, step 51, ce_0 1.166, fer_0 0.315, ce_1 1.023, fer_1 0.279, ce 0.977, fer 0.273, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.730 sec/step, elapsed 0:00:42, exp. remaining 0:59:44, complete 1.16%
ep 21 train, step 51, ce_0 1.098, fer_0 0.312, ce_1 0.889, fer_1 0.265, ce 0.787, fer 0.243, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.737 sec/step, elapsed 0:00:42, exp. remaining 0:57:42, complete 1.21%
ep 21 train, step 52, ce_0 1.196, fer_0 0.302, ce_1 1.045, fer_1 0.262, ce 0.957, fer 0.247, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.738 sec/step, elapsed 0:00:42, exp. remaining 0:53:30, complete 1.32%
ep 21 train, step 52, ce_0 1.046, fer_0 0.283, ce_1 0.905, fer_1 0.242, ce 0.840, fer 0.230, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.737 sec/step, elapsed 0:00:42, exp. remaining 1:00:03, complete 1.18%
ep 21 train, step 52, ce_0 0.897, fer_0 0.262, ce_1 0.738, fer_1 0.216, ce 0.661, fer 0.199, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.739 sec/step, elapsed 0:00:42, exp. remaining 1:00:34, complete 1.17%
ep 21 train, step 52, ce_0 1.059, fer_0 0.302, ce_1 0.842, fer_1 0.254, ce 0.775, fer 0.236, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.735 sec/step, elapsed 0:00:43, exp. remaining 0:58:26, complete 1.21%
ep 21 train, step 53, ce_0 1.125, fer_0 0.271, ce_1 1.014, fer_1 0.243, ce 0.915, fer 0.230, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.731 sec/step, elapsed 0:00:43, exp. remaining 1:00:51, complete 1.18%
ep 21 train, step 53, ce_0 1.607, fer_0 0.363, ce_1 1.398, fer_1 0.332, ce 1.337, fer 0.320, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.742 sec/step, elapsed 0:00:43, exp. remaining 0:53:58, complete 1.33%
ep 21 train, step 53, ce_0 1.069, fer_0 0.307, ce_1 0.905, fer_1 0.273, ce 0.859, fer 0.265, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.742 sec/step, elapsed 0:00:43, exp. remaining 1:01:21, complete 1.17%
MEMORY: sub proc CFD worker 22(3410798) increased RSS: rss=0.9GB pss=699.8MB uss=666.6MB shared=219.8MB
MEMORY: total (main 3362628, 2025-04-09, 19:29:47, 8 procs): pss=5.1GB uss=4.7GB
ep 21 train, step 53, ce_0 0.711, fer_0 0.219, ce_1 0.586, fer_1 0.185, ce 0.537, fer 0.183, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.733 sec/step, elapsed 0:00:43, exp. remaining 0:58:59, complete 1.22%
ep 21 train, step 54, ce_0 1.348, fer_0 0.353, ce_1 1.123, fer_1 0.304, ce 0.970, fer 0.273, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.738 sec/step, elapsed 0:00:44, exp. remaining 1:01:28, complete 1.19%
ep 21 train, step 54, ce_0 1.037, fer_0 0.288, ce_1 0.905, fer_1 0.259, ce 0.837, fer 0.239, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.730 sec/step, elapsed 0:00:44, exp. remaining 1:02:03, complete 1.18%
ep 21 train, step 54, ce_0 0.900, fer_0 0.254, ce_1 0.763, fer_1 0.218, ce 0.694, fer 0.209, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.737 sec/step, elapsed 0:00:44, exp. remaining 0:54:38, complete 1.33%
ep 21 train, step 54, ce_0 1.030, fer_0 0.282, ce_1 0.902, fer_1 0.253, ce 0.848, fer 0.244, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.732 sec/step, elapsed 0:00:44, exp. remaining 0:59:34, complete 1.23%
MEMORY: proc CFD worker 22(3410797) exited, old: rss=493.8MB pss=301.2MB uss=272.4MB shared=221.4MB
MEMORY: total (main 3362629, 2025-04-09, 19:29:48, 7 procs): pss=4.2GB uss=3.8GB
ep 21 train, step 55, ce_0 1.186, fer_0 0.308, ce_1 0.990, fer_1 0.269, ce 0.848, fer 0.241, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.728 sec/step, elapsed 0:00:45, exp. remaining 0:55:11, complete 1.34%
ep 21 train, step 55, ce_0 1.135, fer_0 0.285, ce_1 0.949, fer_1 0.252, ce 0.834, fer 0.223, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.738 sec/step, elapsed 0:00:45, exp. remaining 1:02:12, complete 1.19%
ep 21 train, step 55, ce_0 0.952, fer_0 0.268, ce_1 0.794, fer_1 0.237, ce 0.708, fer 0.216, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.739 sec/step, elapsed 0:00:45, exp. remaining 1:02:45, complete 1.18%
ep 21 train, step 55, ce_0 1.103, fer_0 0.258, ce_1 0.957, fer_1 0.232, ce 0.914, fer 0.223, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.737 sec/step, elapsed 0:00:45, exp. remaining 1:00:13, complete 1.24%
ep 21 train, step 56, ce_0 1.290, fer_0 0.331, ce_1 1.095, fer_1 0.299, ce 0.992, fer 0.273, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.737 sec/step, elapsed 0:00:45, exp. remaining 1:02:48, complete 1.20%
ep 21 train, step 56, ce_0 1.558, fer_0 0.372, ce_1 1.442, fer_1 0.341, ce 1.346, fer 0.324, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.742 sec/step, elapsed 0:00:45, exp. remaining 0:55:47, complete 1.35%
ep 21 train, step 56, ce_0 1.063, fer_0 0.306, ce_1 0.903, fer_1 0.255, ce 0.816, fer 0.238, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.736 sec/step, elapsed 0:00:45, exp. remaining 1:03:27, complete 1.19%
ep 21 train, step 56, ce_0 0.781, fer_0 0.227, ce_1 0.644, fer_1 0.200, ce 0.596, fer 0.196, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.738 sec/step, elapsed 0:00:46, exp. remaining 1:00:45, complete 1.25%
ep 21 train, step 57, ce_0 1.089, fer_0 0.300, ce_1 0.922, fer_1 0.272, ce 0.787, fer 0.249, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.736 sec/step, elapsed 0:00:46, exp. remaining 1:03:20, complete 1.21%
ep 21 train, step 57, ce_0 1.645, fer_0 0.372, ce_1 1.547, fer_1 0.335, ce 1.506, fer 0.322, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.737 sec/step, elapsed 0:00:46, exp. remaining 0:56:19, complete 1.36%
ep 21 train, step 57, ce_0 1.008, fer_0 0.284, ce_1 0.878, fer_1 0.254, ce 0.807, fer 0.247, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.745 sec/step, elapsed 0:00:46, exp. remaining 1:04:03, complete 1.20%
ep 21 train, step 57, ce_0 1.151, fer_0 0.310, ce_1 0.940, fer_1 0.264, ce 0.834, fer 0.251, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:1 14.6GB, 0.736 sec/step, elapsed 0:00:46, exp. remaining 1:01:16, complete 1.26%
FileCache: Copy file /nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.3 to cache
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.130, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 28.1GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.140, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.140: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.140'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.140: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.140.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.7, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 29.2GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.7: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.7'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130, size 9.3MB. Still want more space, file is 24.3 hours old. After deletion, have 28.2GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.140, size 8.1MB. Still want more space, file is 24.3 hours old. After deletion, have 28.2GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.7, size 7.4MB. Still want more space, file is 24.3 hours old. After deletion, have 28.2GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.6, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 29.3GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.16, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 30.3GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.18, size 1.0GB. Still want more space, file is 24.3 hours old. After deletion, have 31.4GB free space.
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.18: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.18.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.16, size 8.8MB. Still want more space, file is 24.3 hours old. After deletion, have 31.4GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.16: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.16'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.16: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.16.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.6, size 8.2MB. Still want more space, file is 24.3 hours old. After deletion, have 31.4GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.18, size 7.3MB. Still want more space, file is 24.3 hours old. After deletion, have 31.4GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.18: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.18'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.18: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.18.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 32.5GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.128, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 32.5GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.128: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.128'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.74, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 32.5GB free space.
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.30, size 1.1GB. Still want more space, file is 24.3 hours old. After deletion, have 33.5GB free space.
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.30: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.30.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149, size 8.5MB. Still want more space, file is 24.3 hours old. After deletion, have 33.5GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.128, size 8.2MB. Still want more space, file is 24.3 hours old. After deletion, have 33.5GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.128: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.128'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.128: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.128.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.74, size 7.8MB. Still want more space, file is 24.3 hours old. After deletion, have 33.5GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.74: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.74'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.74: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.74.returnn-info'
FileCache: Delete file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.30, size 7.4MB. Still want more space, file is 24.3 hours old. After deletion, have 33.5GB free space.
FileCache: Error while removing /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.30: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.30'
FileCache: Ignoring error file removing info file of /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.30: FileNotFoundError: [Errno 2] No such file or directory: '/ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.30.returnn-info'
FileCache: Copy file /nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.15 to cache
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.55
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedConversations/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-11/SimulatedConversations-batch.2.v2.hdf.4
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-27/commonvoice-validated.v2.hdf.11
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.22
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.62
FileCache: Copy file /nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130 to cache
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.95
FileCache: Copy file /nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149 to cache
FileCache: Copy file /nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.3 to cache
FileCache: Copy file /nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.15 to cache
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.55
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedConversations/corpus/batch.2.v2/hdf-raw_wav-v2.16kHz.split-11/SimulatedConversations-batch.2.v2.hdf.4
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-raw_wav.16kHz.split-27/commonvoice-validated.v2.hdf.11
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.22
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.62
FileCache: Copy file /nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.130 to cache
FileCache: using existing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.95
FileCache: Copy file /nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149 to cache
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-25/PublicMedia-batch.1.v2.hdf.3
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-raw_wav-v2.16kHz.split-26/SimulatedCallcenter-batch.1.v2.hdf.15
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.55
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedConversations/corpus/batch.2.v2/hdf-raw_wav-v2.16kHz.split-11/SimulatedConversations-batch.2.v2.hdf.4
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-raw_wav.16kHz.split-27/commonvoice-validated.v2.hdf.11
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-raw_wav.16kHz.split-32/PublicMedia-batch.2.v2.hdf.22
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.62
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.130
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.95
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-raw_wav.16kHz.split-165/IT.parliament-batch.1.v2.hdf.149
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-25/PublicMedia-batch.1.v2.hdf.3
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedCallcenter/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-26/SimulatedCallcenter-batch.1.v2.hdf.15
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.55
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/8kHz/SimulatedConversations/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218-v2.alignment.split-11/SimulatedConversations-batch.2.v2.hdf.4
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/commonvoice/corpus/validated.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-27/commonvoice-validated.v2.hdf.11
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/PublicMedia/corpus/batch.2.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-32/PublicMedia-batch.2.v2.hdf.22
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.62
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.130
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.95
parsing file /ssd/hyoshimochi/returnn/file_cache/nas/data/speech/IT_IT/16kHz/IT.parliament/corpus/batch.1.v2/hdf-ffnn_monophone_dense_eow_20250218.alignment.split-165/IT.parliament-batch.1.v2.hdf.149
Reading sequence list for MetaDataset 'dataset_id139758134565216' from sub-dataset 'dataset_id139758134565216_alignments'
ep 21 train, step 58, ce_0 1.200, fer_0 0.297, ce_1 1.086, fer_1 0.270, ce 1.013, fer 0.248, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:0 14.3GB, 0.738 sec/step, elapsed 0:00:47, exp. remaining 0:56:57, complete 1.37%
ep 21 train, step 58, ce_0 1.063, fer_0 0.286, ce_1 0.936, fer_1 0.258, ce 0.868, fer 0.243, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:2 14.5GB, 0.745 sec/step, elapsed 0:00:47, exp. remaining 1:03:38, complete 1.22%
ep 21 train, step 58, ce_0 1.237, fer_0 0.314, ce_1 1.050, fer_1 0.282, ce 0.961, fer 0.260, num_seqs 19, max_size:time:var-unk:classes 500, max_size:time:var-unk:data 80240, mem_usage:cuda:3 14.5GB, 0.737 sec/step, elapsed 0:00:47, exp. remaining 1:04:44, complete 1.21%
Reinitialize dataset seq order for epoch 1.

@NeoLegends NeoLegends mentioned this issue Apr 11, 2025
Merged
@NeoLegends
Copy link
Member Author

NeoLegends commented Apr 11, 2025 •
edited
Loading

When only locking the cleanup, we need to make sure all processes try to acquire the cleanup lock every time they try to copy a file, to wait for potential other cleanups happening in the meantime. I think as a necessary evil this is fine, as the cleanup is not going to happen very often (and then not take very long), so the locks are going to be held only for very short amounts of time every time. Certainly this will be less under contention than locking the full cache directory every time you try to access a file (cf. #1548).

See #1709.

@albertz
Copy link
Member

albertz commented Apr 22, 2025

So, the PR #1709 was merged now. As I understand, this should have fixed this issue here.

Please reopen if you think this issue still persists.

In case you get similar but different problems, or other problems with the file cache (maybe with DistributeFilesDataset), or you are unsure, please open a new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@NeoLegends NeoLegends

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@albertz @NeoLegends @hannah220