jupyter: ruff v0.1.4 ignores variables with the same name as magic commands

[njzjz]

This is the previous Jupyter Notebook file, and it had no lint errors with ruff v0.1.3.

When ruff was upgraded to v0.1.4 (deepmodeling/dpdata#573), import dpdata got removed,

Besides, F821 was complained:

ruff.....................................................................Failed
- hook id: ruff
- exit code: 1

docs/nb/try_dpdata.ipynb:cell 4:1:1: F821 Undefined name `system`
docs/nb/try_dpdata.ipynb:cell 5:1:1: F821 Undefined name `system`
docs/nb/try_dpdata.ipynb:cell 6:1:1: F821 Undefined name `system`
Found 3 errors.

It's obvious that ruff doesn't consider variables in the other cells.

[charliermarsh]

In this case, it's because system is a Jupyter magic which can be used as an automagic. In Jupyter, you can do system ls in a cell (without any percent signs), and it will treat the command as an ls. We started ignoring cells that look like automagics, because it can be hard (impossible) to understand their behavior, since Jupyter will actually treat them differently depending on what other variables are defined, i.e., the order in which you run your cells.

If you use any other variable name, it will work as expected, but we'll consider how to make this more targeted.

[charliermarsh]

That explanation is probably unclear, here's an image that demonstrates what I'm describing -- the first and third cells are identical, but their behaviors entirely depend on the other of execution:

[charliermarsh]

\cc @dhruvmanila since we discussed this a bit at the time

[dhruvmanila]

Yeah, this is tricky. We could implement the simple solution that we discussed earlier about having support for only pip but then the same problem would appear.

The far fetched solution would be to not ignore such cells and perform some semantic analysis on the parsed code but then it could potentially raise a syntax error (cd ..). For any potential automagic command, the semantic model would probably check if the identifier (system, cd, etc.) is already defined in the scope. If it is, continue as Python code otherwise continue as an automagic.

I'd actually want to spend some time figuring out how IPython does it as it could lead to some insights.

[dhruvmanila]

So, IPython is doing exactly that: https://github.com/ipython/ipython/blob/de97f0032dd96e0109780396e9219e8d73073e29/IPython/core/prefilter.py#L428-L467

For something like pwd = "foo", the AssignmentChecker will run it normally as Python code and then for any other usage of pwd, the AutoMagicChecker checks if it's shadowed or not and runs it as a magic command accordingly.

[charliermarsh]

Hmm yeah. We could apply some similar heuristics though they'd need to be more expansive... We could also try parsing, then fallback to automagic if the code doesn't parse, but that will also be wrong in some cases. I'd just really like to avoid doing semantic analysis on the code block to understand how to parse it.

[charliermarsh]

This is fixed by #9653.