Delayed import of scikit-learn and pandas + other dependency tweaks

[danielhollas]

Various tweaks to dependencies:

1. Delayed import of scikit-learn and bokeh.
   Both of these are very large packages that may negatively impact startup time, so it makes sense to import them only when necessary. It would be great to get rid of them altogether. We use pandas only to generate HTML tables from python dictionaries. Perhaps there is a more lightweight solution for that. Sklearn is only used to do PCA in SmilesWidget, I'll try to remove it in another PR.

2. Declare rdkit and sklearn as optional dependencies for SmilesWidget.

3. Add basic test for SmilesWidget

4. Updated test dependencies and moved them from setup.cfg to requirements_test.txt, same way as in QeApp. The pip installation on GHA went from 1 minute to 5 seconds so this is worthwhile.

5. Removed stale requirements.json file. I am assuming it is not needed given the setup.cfg but correct me if I am wrong.

I added this to the milestone, since we're removing sklearn as a direct dependency, which might potentially break apps that relied on it. Note that SmilesWidget was already broken in principle, since rdkit is not installed in the new docker stack. At least now apps can reliably install needed dependencies by specifying aiidalab-widgets-base[smiles].

[danielhollas]

Tests are failing, not sure why. Will enable screenshots first in #411 to debug this.

[unkcpz]

To reproduce the failing test, you can launch a docker container and run the failed notebook with the branch.

[unkcpz]

Hi @danielhollas, just request review again when you think this is ready.

[danielhollas]

@unkcpz I need to fix one of the XPath selector to fix the new test, but otherwise this is good to review.

EDIT: Tests are passing

[unkcpz]

I am not sure about this, if you want "smiles" related packages installed by aiidalab install, it not supports install from extras dependencies yet.

[danielhollas]

That's okay I think, because I can declare the dependency in my setup.cfg with these extras dependencies, so it should pull the package from pip.

[unkcpz]

Ah, this is clever, but only for packages like aiidalab-widgets-base and when we not treat it as "app" anymore. Anyway, good for the change here.

[danielhollas]

Yes, that is a good point. But @yakutovicha said many times he wants to turn AWB into library anyways, so it should be okay.

[unkcpz]

@danielhollas I think you have permission to merge, correct? I'll leave it to you in case you still want to add things. It is all good to me.
This one actually bring the improvement of the performance for some widget, it reminds me that we want to add the benchmark test to the widgets, but we can do in a separate PR.

[danielhollas]

I don't have permission to merge actually. This is good to go from my side.

This one actually bring the improvement of the performance for some widget, it reminds me that we want to add the benchmark test to the widgets, but we can do in a separate PR.

Yes, that would be great. I suspect this PR alone does not bring performance improvements much. Pandas is still imported by optimade-client. But I don't know how to measure this well.

[danielhollas]

@yakutovicha can you confirm it's okay to delete requirements.txt?

[yakutovicha]

I am a bit concerned by some changes in this PR. Specifically, I don't see a need of separating dev into dev and testing. All packages will be installed during the CI run anyways. Some while ago we actually did the opposite thing: we merged them together into dev.