Improve performance

[chrisgorgo]

Initializing the Layout object requires indexing the whole directory recursively. This can be time consuming (especially on S3 see #45). We should investigate ways to improve this (for example #37).

[anibalsolon]

That's a good point.

I've been thinking about a lazy indexing scheme, but it would be more changes for grabbit than grabbids. So for example when querying for entities with parameters based on {{root}} directory (such as subject), it could sustain the laziness. For parameters based only on patterns (such as modality for BIDS), a full transverse would become necessary.

And, of course, in order to use pybids to validate the whole tree, it is necessary to transverse it. So it need to be configurable.

In the S3 example, since everything on S3 is an object, and there are no folders per se, the API call for list-objects would "transverse" the tree only limiting by pagination (1000 objs / req). Thus, I think it would not be a bottleneck.

[tyarkoni]

I agree with @anibalsolon that this is probably worth raising in the grabbit repo, but probably won't buy us much in grabbids, since the new validator (see #65) will indeed need to traverse the whole tree. Also, since grabbit now supports writing/reading indexes to/from files as of 0.0.8, the indexing probably only needs to be done once per project. So I'm not sure it's worth worrying about the remaining use case where someone has an enormous project, doesn't need validation, and can't wait a few minutes to build the index once. Closing this here, but feel free to re-open if you think it still needs consideration.