Issues with the new update mechanism

[nadako]

I'll make a list of issues found with the new haxelib update mechanism, so we don't forget about them:

• haxelib remove haxelib fails (at least on Windows) because the directory with run.n file is busy. One option would be to always use bundled haxelib if we're removing haxelib, but that sounds pretty hacky.
• When user upgrades Haxe with a newer bundled haxelib, his installed haxelib may be of lower version than the bundled one. In general, this is a not big issue, since haxelib will show upgrade notification (Available version check #282), but this may be a problem if some newer Haxe version requires a newer haxelib version. We could check if installed haxelib version is greater than the bundled one.
• I still think that we gotta check if there's a noticeable performance regression with haxelib path, since now we do some extra work (extra fs queries, reading haxelib.json, invoking neko run.n). However I wouldn't be surprised if it'll perform on par or even faster on Unixes because haxe --interp mode we used earlier is slower than Neko.
• Should haxelib haxelib be searched in a project-local repository (created with newrepo)? I'd say not.
• Check if it works correctly when haxelib repo path is not configured (no /etc/.haxelib or ~/.haxelib file, no HAXELIB_PATH env var)
• Document the usage of --safe option if haxelib installation is messed up

cc @ibilon

[andyli]

For, haxelib remove haxelib i think it is reasonable to require the use of --safe option.

For haxelib haxelib & project-local repository, that's an interesting question... I think "do not search local repo" would give less surprising result. I have to give it a bit more thought.

[nadako]

For, haxelib remove haxelib i think it is reasonable to require the use of --safe option.

You mean, just exit with an error message (suggesting --safe) if we try to remove haxelib from within installed haxelib? That's an easy option! \o/

[nadako]

UPD: added one more item we need to make sure is handled correctly

Check if it works correctly when haxelib repo path is not configured (no /etc/.haxelib or ~/.haxelib file, no HAXELIB_PATH env var)

[ibilon]

(haxelib remove --safe)
Not sure the safe is required, remove is already blocked against the set version, which would be running.
How far goes the windows directory lock?

(haxe upgrade doesn't upgrade haxelib)
For the haxe upgrade that would indeed not upgrade haxelib anymore (beside the safe version),
but I think that's ok since it'll be notified,
and the semver check would prevent the full use of set/dev granted by running on top of run.

(performance cost)
I'll make some test and see how long it is from the command start to the main in the "real" executed haxelib.

(project local repo)
The haxelib lib is used globally through the haxelib command, it does not make sense to have a local version.
That means haxelib update haxelib should probably ignore it too, otherwise a selfupdate might turn into a local install.

(no setup)
What does getRepository() returns if not setup?
https://github.com/HaxeFoundation/haxelib/blob/development/src/haxelib/client/Main.hx#L359
If not setup the file shouldn’t be found and won't be run.

[nadako]

(haxelib remove --safe)
Not sure the safe is required, remove is already blocked against the set version, which would be running.
How far goes the windows directory lock?

That happens when you do remove haxelib without specifying version. I added the check here: 232f9f8

(no setup)
What does getRepository() returns if not setup?
https://github.com/HaxeFoundation/haxelib/blob/development/src/haxelib/client/Main.hx#L359
If not setup the file shouldn’t be found and won't be run.

Yeah, and I added some error handling here: 55c1414

[nadako]

(project local repo)
The haxelib lib is used globally through the haxelib command, it does not make sense to have a local version.
That means haxelib update haxelib should probably ignore it too, otherwise a selfupdate might turn into a local install.

Sounds reasonable, I guess we have to add some more checks here and there. I'll do it!

[nadako]

and the semver check would prevent the full use of set/dev granted by running on top of run.

Maybe we should at least emit a warning if bundled haxelib is about to run library haxelib of earlier version?

[ibilon]

We already do (almost finished) emit a warning: "update available".

[nadako]

Right. There's a small concern that a newer haxe might DEPEND on a newer haxelib version, but I guess if that ever happens, we can handle it specifically.

[nadako]

I tried updating haxelib to the latest version in haxe and it revealed something:

https://travis-ci.org/HaxeFoundation/haxe/jobs/115524038
https://travis-ci.org/HaxeFoundation/haxe/jobs/115524044

[ibilon]

Oh, haxelib run munit must have leaked it's env var into a call to haxe/haxelib.
That's not an easy one to fix.

I don't understand why path would fail in the second one.

[nadako]

Oh, haxelib run munit must have leaked it's env var into a call to haxe/haxelib.
That's not an easy one to fix.

Oh crap. Am I correct thinking that it's this:

1. haxelib run some -> sets HAXELIB_RUN
2. some/run.hx tries to haxelib run other and that fails because of HAXELIB_RUN

This can be fixed by passing library name via an env var and checking it in haxelib.
At first I thought about changing value 1 of HAXELIB_RUN, but technically it's a breaking change, if someone relies on this concrete value. We gotta think about this some more. Also #272 is related.

I don't understand why path would fail in the second one.

It may have the same root.

[nadako]

A quick search on github shows that we can't change HAXELIB_RUN from 1. So we may have to introduce an extra var like HAXELIB_RUN_NAME.

[nadako]

Okay. That helped (and is actually useful for haxelibs in general). But as with any new public API I have concerns whether it's good or bad. At the moment I can't think of another options, so if noone objects before the release, It'll stay like this. :)

[ibilon]

Sounds good to me.

[ibilon]

I've put a trace(haxe.Timer.stamp()); call at the start of the main function,
got two lines, my haxelib call and the one haxelib is running,
the difference between the first and the second is just 0.018 second,
I think that's ok :)

[nadako]

Sounds fine. What OS are you on btw?

[ibilon]

Linux (openSuse)

[ibilon]

I remembered to compare with the old haxe --run tools.haxelib.Main from the latest haxelib_client on haxelib.

Asking for the path of flixel takes:

• haxe run around 310ms (average)
• haxelib run haxelib around 40ms

[nadako]

Ouf of interest, could you provide these numbers with latest haxe/haxelib? :)

[nadako]

well latest haxelib, don't care about haxe here

[nadako]

I've put a trace(haxe.Timer.stamp()); call at the start of the main function,
got two lines, my haxelib call and the one haxelib is running,
the difference between the first and the second is just 0.018 second,

for me on Windows it's around 0.044. I guess this is fine after all.

[ibilon]

haxe -cp src --run haxelib.client.Main path flixel

An average of 550 ms, adding --safe is basically the same,
so we did make it slower, but since it's not on haxe --run anymore it's better.

[nadako]

That's great!

[nadako]

I wonder why is it slower though, is there a bottleneck in haxe interp regarding to spawning process?

[ibilon]

Since it's more or less the same time with or without --safe then the cost of spawning process + neko exec isn't the issue.

[nadako]

Interesting, since I expected path to become actually faster with a2e7952 and 7c8a593

[nadako]

Regarding project-local haxelib, we settled somewhere in the middle, see #295

[ibilon]

Only one item left, I can update the doc if you want.

[nadako]

Feel free, I have to go afk for an hour or two anyway :)

[nadako]

Maybe --safe should be documented as a part of the new selfupdate documentation :-P

[andyli]

What about renaming --safe? Just don't want to make it sound like every operation is really safe... Maybe --base or --system is more appropriate.

[ibilon]

Or that not using it is unsafe.

I think that system sounds better than base.

[nadako]

It would also be nice to decide on the terminology. How do we call the haxelibs?
"Bundled" doesn't sound right to me when we're talking about something that is installed in OS, "System" may be better inded.
I have no idea how to call the "haxelib haxelib" though :)

[ibilon]

Personally I don't call them haxelibs, that's too confusing, simply libraries.

I think it could be a bit late to find a nice term like ruby's gems.

[nadako]

yeah too bad our language isn't named after diamonds and snakes :-P

[ibilon]

Haxe is usually eaten with potatoes and sauerkraut, not catchy names for libraries :D

So I guess all that's left in this issue is to rename safe into system in the code and doc?

[nadako]

I like "potato"!

So I guess all that's left in this issue is to rename safe into system in the code and doc?

Yeah I think so. Everything seems to work so far :)

[ncannasse]

Could I get the result of a simple Hello class using haxe --times -lib heaps -main Hello (for instance).
On Windows with current haxelib I get between 0.05s and 0.08s execution time for haxelib.

[nadako]

https://gist.github.com/nadako/93e659e10542caab5a2a

As mentioned earlier, the slowdown on Windows is about 0.044s. Hope that's something we can live with. We discussed loading run.n module instead of invoking neko which may speed up things, but haven't tested it, and I personally don't know about any security or other considerations regarding that.

BTW, I'd like to look at these numbers on Linux/OSX.

[ncannasse]

Thanks, that looks good enough for now. And yes loading run.n would prevent having to start one extra process, especially if you just need to resolve the .n directory then call it with the exact same arguments.

[nadako]

Thanks, that looks good enough for now. And yes loading run.n would prevent having to start one extra process, especially if you just need to resolve the .n directory then call it with the exact same arguments.

I'll look into it later and do some measurements.