How to access all file names in hydra config

[abhijeetdhakane]

I have a directory contains a bunch of txt files:
dir/train/[train1.txt, train2.txt, train3.txt]

I'm able to read a single file, if I define following in a config.yaml
file_name: ${paths.data_dir}/train/train1.txt

So I get the str and I used np.loadtxt(self.hparams.file_name)

I tried
file_name: ${paths.data_dir}/train/*

So I have List[str], I then loop over file_name

dat = []
for file in self.hparams.file_name:
   dat.append(np.loadtxt(file))

but it didn't work out.
Sorry for cross-posting

[abhijeetdhakane]

It works if I do the following:

file_name: ['${paths.data_dir}/train/train1.txt', '${paths.data_dir}/train/train2.txt','${paths.data_dir}/train/train3.txt']

But is there any smart way to avoid repetitive long paths names, and just change last file names?

[Jasha10]

You could define an OmegaConf custom resolver for this:

# my_app.py
import pathlib
from pathlib import Path
from typing import List

from omegaconf import OmegaConf

yaml_data = """
paths:
  data_dir: dir
file_names: ${pathlib_glob:${paths.data_dir}, 'train/*'}
"""


def pathlib_glob(data_dir: str, glob_pattern: str) -> List[str]:
    """Use Pathlib glob to get a list of filenames"""

    data_dir_path = pathlib.Path(data_dir)
    file_paths: List[Path] = [p for p in data_dir_path.glob(glob_pattern)]
    filenames: List[str] = [str(p) for p in file_paths]
    return filenames

OmegaConf.register_new_resolver("pathlib_glob", pathlib_glob)

cfg = OmegaConf.create(yaml_data)
assert cfg.file_names == ['dir/train/train3.txt', 'dir/train/train2.txt', 'dir/train/train1.txt']

Now, at the command line:

mkdir -p dir/train
touch dir/train/train1.txt
touch dir/train/train2.txt
touch dir/train/train3.txt
python my_app.py  # the assertion passes

[abhijeetdhakane]

Thanks for the answer @Jasha10 , I have a follow up question, where should I include this custom resolver. My current yaml looks like:

train:
   _target_: src.dataamodule.custom_dataloader.TextLoader
  filenames: ['dir/train/train3.txt', 'dir/train/train2.txt', 'dir/train/train1.txt']

TextLoader is a torch custom dataset

[abhijeetdhakane]

I am using this template https://github.com/ashleve/lightning-hydra-template

[Jasha10]

where should I include this custom resolver

You can include it anywhere, e.g. inside of your main function, or just before you define your main function. Just make sure that you call OmegaConf.register_new_resolver before you access the cfg.file_names attribute. The resolver will be looked up lazily when you call cfg.__getattr__("file_names").

You can use assert OmegaConf.has_resolver("pathlib_glob") to check that your resolver has been properly registered.