Running cells with declarations multiple times: redefinition error - any workarounds?

[momonala]

If I run a cell where I am defining a variable or function multiple times, I get an error for redefinition of that variable or function, which makes sense. But in Jupyter it is nice to quickly test a few variations of a function without restarting the entire kernel. Is there a clever way around this? Python has a magic function to reset specific variables or the whole namespace, is some version of this available?

[SylvainCorlay]

This is a limitation of cling. The gist of it is that reversing the compilation to restore the previous state of the program is not easy.

[drvink]

fwiw, cling allows this, at least from the cling shell, via .u (for undo). It works as expected: you can travel sequentially back in time, but you cannot pick individual states to remove.

[proto-n]

I've run into this immediately when trying out xeus-cling.

Some input for thought:
Coming from a Python background, what I personally want is not actually to restore a previous state of the program, but for the new variable / function / class to hide the old one. Similar to how name hiding in child classes works. This should be technically much simpler I think than state restoring, but I could be wrong.

I'm fine with the old one still existing, but I want to the name to refer to the new one. I'm fine with calls in old code still referring to the hidden one, previously created objects being of the hidden type, etc. Basically, I want it to behave as if I renamed the function from some_name to some_name_2 and search/replaced every occurrence in the code.

This behavior can be a source of confusion though, so maybe a toggle?

[JohnDoe02]

As an (of course limited) workaround, you can use lambdas instead of real functions. The lambda is anonymous and can therefore be redeclared, e.g.:

[](){
    auto try_and_err = []() {
        // complicated function that I need to edit again and again
        // until it works
    };

    try_and_err();
}();

A cell like this can be evaluated as often as you'd like.

[adibaba]

My colleague @bigerl suggested a similar solution:

Cell n:

#include <iostream>
void (*myFunc)() = nullptr;

Cell n+1:

myFunc = []()->void {
    std::cout << "Hey";           
};
myFunc();

Using this, you can define function names, which can be used for tests later (e.g. for nbgrader).

[phcerdan]

👍 This would be great to have! It will improve the experience of prototyping c++ in jupyter. Does cling have a way to remove a declaration, not worrying about the old state?

Thanks for the lambda workaround for functions, but it would be nice to have it for classes as well.

[StefanRickli]

Just tried xeus-cling out today and I must say that this issue needs to be solved before I'm going to use it further.

Because at the moment it goes like this:
You make a mistake in the definition of a function, run the cell, see an error or wrong output, think 'oh shoot', correct the mistake, restart the kernel and pray for the best. In the worst case rince and repeat.

[certik]

I came to report this same issue, I think this is a big problem.

I have written a similar interactive kernel for Fortran (https://docs.lfortran.org/) and there I simply shadow the previous definition, the example notebook has an example for that, you can also launch it in Binder to play with that interactively.

While C++ is a lot more complicated language than Fortran, I still think this issue is solvable.

[vinayan3]

I also have seen this and coming from the python world it is quite the surprise. @drvink mentioned
there is .u in cling. Is there a possibility to make a magic command to let us undo?

[aGIToz]

Are there any plans to solve this issue ? I mean will this feature of redefining the variables, functions and classes will be integrated in xeus-cling kernel ? If yes this will totally change the game of doing scientific programming in cpp.

[SylvainCorlay]

Yes, the support recently landed in cling (master) but is not in a release yet.

As soon as a release of cling includes it, it will be in xeus-cling.

[aGIToz]

This news just made my sunday!

[hendersonh]

Do they really expect anyone to use this "cling" with C++ in the current state? I mean the basic premise of using a Jupyter notebook is to allow experimenting and quick editing of cells. It is like making a car without the steering wheels and brakes and without an engine. I think this is was a joke being played on us.

[SylvainCorlay]

@hendersonh I think you should ask for a refund.

[rcurtin]

@SylvainCorlay I'm really excited about the support for this in cling (once it's ready, I can build an mlpack notebook! :)). I was watching root-project/cling#325 but it doesn't look like there is a response there. Is there any chance that backporting relevant patches from cling could be a short-term solution?

(Please note that I'm asking this question from a place of ignorance! I don't know how much effort would be needed for that. If you can point me in the right direction with enough instructions, maybe I could help. :))

[asingh-divinisoft]

I had a silly idea. Thought of sharing it. Maybe whenever a redefinition error occurs you can use .undo [n] to go back. Execute the new definition and all the transactions that were undone again.

[zoq]

I build xeus-cling against the latest cling version and packaged cling and xeus-cling (master branch) - https://anaconda.org/zoq/repo. Redefinition support is enabled by default. The solution is not perfect, see the template specialization example, nonetheless, I think this is an improvement for the most common workflows.

Example - https://mybinder.org/v2/gh/zoq/xeus-cling/cling-nighlty-binder?filepath=notebooks/xcpp.ipynb

[SylvainCorlay]

@zoq this is great! I think we should make a build of cling and xeus-cling including that patch on conda-forge!

[SylvainCorlay]

Oh, I notice that you are building from the last commit of the master branch for cling instead of by adding patches to the last release. Unfortunately, we may not be able to do this at this point on conda-forge.

[zoq]

Can we build against the latest tag? Also, I guess I can also create a huge patch for everything between now and v0.6.

[SylvainCorlay]

@zoq the root tag is an internal thing used for the root project.

I filed root-project/cling#325 asking if they would like to tag releases more frequently, which would be great.

[zoq]

Btw. the redefinition feature can be disabled with:

#include "cling/Interpreter/Interpreter.h"
cling::runtime::gCling->allowRedefinition(false);

or enabled with:

#include "cling/Interpreter/Interpreter.h"
cling::runtime::gCling->allowRedefinition();

But to enable/disable the feature, you have to install llvmdev (could patch this out).

[mujina93]

@zoq can your build be installed to try it out with: conda install -c zoq xeus-cling? I get an empty UnsatisfiableError: when trying to install in a fresh conda environment with no other packages inside.

[aGIToz]

I get the same, there is zlib-cuda 10.2 incompatibility also.

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package libgcc-ng conflicts for:
python==3.7.5=h0371630_0 -> libffi[version='>=3.2.1,<3.3a0'] -> libgcc-ng[version='>=7.2.0']
python==3.7.5=h0371630_0 -> libgcc-ng[version='>=7.3.0']

Package libstdcxx-ng conflicts for:
python==3.7.5=h0371630_0 -> libstdcxx-ng[version='>=7.3.0']
python==3.7.5=h0371630_0 -> libffi[version='>=3.2.1,<3.3a0'] -> libstdcxx-ng[version='>=7.2.0']

Package zlib conflicts for:
python==3.7.5=h0371630_0 -> zlib[version='>=1.2.11,<1.3.0a0']
xeus-cling -> zlib[version='>=1.2.11,<1.3.0a0']The following specifications were found to be incompatible with your CUDA driver:

  - feature:/linux-64::__cuda==10.2=0

Your installed CUDA driver is: 10.2

[zoq]

Interesting, will look into the issue over the weekend.

[zoq]

@mujina93 @aGIToz do you use anaconda or minconda to install the package?

[aGIToz]

Anaconda

…

On Mon, Jul 6, 2020, 12:22 AM Marcus Edel ***@***.***> wrote: @mujina93 <https://github.com/mujina93> @aGIToz <https://github.com/aGIToz> do you use anaconda or minconda to install the package? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#91 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJDSHX64OEPTOJNYKGPVWE3R2D4LFANCNFSM4EHLKS4A> .

[zoq]

Anaconda
…
On Mon, Jul 6, 2020, 12:22 AM Marcus Edel @.***> wrote: @mujina93 https://github.com/mujina93 @aGIToz https://github.com/aGIToz do you use anaconda or minconda to install the package? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#91 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJDSHX64OEPTOJNYKGPVWE3R2D4LFANCNFSM4EHLKS4A .

Any chance you are able to test miniconda, if I remember right anaconda has some incompatibilities with pre-installed packages, you should see similar issues with xeus-cling from the conda-forge repository. Also make sure to add the conda-forge repository as well: conda install -c zoq xeus-cling -c conda-forge.

[aGIToz]

Yes it works!
-c conda-forge is important otherwise the same error.
But everytime I am donig conda activate or conda install I am getting this on my shell

bash: export: `=x86_64-conda_cos6-linux-gnu': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-addr2line': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-ar': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-as': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-c++filt': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-elfedit': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-gprof': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-ld': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-ld.gold': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-nm': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-objcopy': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-objdump': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-ranlib': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-readelf': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-size': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-strings': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-strip': not a valid identifier
bash: export: `=x86_64-conda_cos6-linux-gnu': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-cc': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-cpp': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-gcc': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-gcc-ar': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-gcc-nm': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-gcc-ranlib': not a valid identifier
bash: export: `=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/azad/miniconda3/include': not a valid identifier
bash: export: `=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/azad/miniconda3/include': not a valid identifier
bash: export: `=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/azad/miniconda3/lib -Wl,-rpath-link,/home/azad/miniconda3/lib -L/home/azad/miniconda3/lib': not a valid identifier
bash: export: `=-D_DEBUG -D_FORTIFY_SOURCE=2 -Og -isystem /home/azad/miniconda3/include': not a valid identifier
bash: export: `=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /home/azad/miniconda3/include': not a valid identifier
bash: export: `=/home/azad/miniconda3:/home/azad/miniconda3/x86_64-conda_cos6-linux-gnu/sysroot/usr': not a valid identifier
bash: export: `=_sysconfigdata_x86_64_conda_cos6_linux_gnu': not a valid identifier
bash: export: `=/home/azad/miniconda3/x86_64-conda_cos6-linux-gnu/sysroot': not a valid identifier
bash: export: `=x86_64-conda_cos6-linux-gnu': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-c++': not a valid identifier
bash: export: `=/home/azad/miniconda3/bin/x86_64-conda_cos6-linux-gnu-g++': not a valid identifier
bash: export: `=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/azad/miniconda3/include': not a valid identifier
bash: export: `=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /home/azad/miniconda3/include': not a valid identifier

what is this?

[mujina93]

Hi @zoq . I can confirm it also works using anaconda for me, I just had to add the -c conda-forge (I'm not sure what's going on with the dependencies here. What exactly is retrieved from the conda-forge channel? All/part of the dependencies of your xeus-cling package?)

Thanks a lot!

[rlloretb]

Which is the current status on this?

[zoq]

Hi @zoq . I can confirm it also works using anaconda for me, I just had to add the -c conda-forge (I'm not sure what's going on with the dependencies here. What exactly is retrieved from the conda-forge channel? All/part of the dependencies of your xeus-cling package?)

Thanks a lot!

All dependencies, here is the recipe https://github.com/zoq/cling-conda-nightly/blob/master/meta.yaml

Which is the current status on this?

I use the modified kernel over here: https://github.com/mlpack/examples.

Examples:

• https://lab.mlpack.org/v2/gh/mlpack/examples/master?urlpath=lab%2Ftree%2Fforest_covertype_prediction_with_random_forests%2Fcovertype-rf-cpp.ipynb
• https://lab.mlpack.org/v2/gh/mlpack/examples/master?urlpath=lab%2Ftree%2Fbreast_cancer_wisconsin_transformation_with_pca%2Fbreast-cancer-wisconsin-pca-cpp.ipynb

Without encountering any huge issues so far.

[justinblaber]

conda install -c zoq xeus-cling -c conda-forge

Is working great for me so far. Can redefine stuff no problem (yet, at least).

[SylvainCorlay]

cling 0.7.0 and xeus-cling 0.10.0 have been released with this long awaited feature!

Many thanks to everyone involved, and especially @zoq who maintained the nightlt conda build of cling.

cc @justinblaber @zoq @rlloretb @mujina93 @aGIToz @asaaditya8 @rcurtin @phcerdan @vinayan3 @certik @momonala @JohnDoe02 @proto-n

[SylvainCorlay]

🎉

[nthiery]

Yippee! Just in time for the new semester. This already made a difference
this week in class! One less hurdle for our newbies.

Kuddos to everyone who made it happen!