Proposal: Haxelib shim

[jgranick]

Haxelib Shim

Current Problems

Many users experience problems running haxelib selfupdate

On a Mac or Linux system, "/usr/lib/haxe/haxelib" may require elevated system privileges, and may be locked.

On Windows, running "haxe update.hxml" afterward feels messy.

Experienced developers I know do not run haxelib selfupdate at all -- they have never heard of it.

If we make vital improvements to the Haxe package manager, we need a mechanism that allows users to receive the update.

Improvements to Development Process

There is no simple way to test changes to the Haxelib client, currently. If you could haxelib dev haxelib_client path/to/dev/client it could improve the workflow for improving haxelib.

If more developers can contribute to haxelib, it may help prevent critical issues we have had.

Proposal

I propose that we create a shim that is able to detect updated versions of the haxelib client, and prefer those if they are available. However, I propose that it also includes a full compiled version of the haxelib, itself, which allows it run as normal if an update is not found.

Haxe always comes with a haxelib binary. Alone, the shim functions identically.

If a user would run haxelib upgrade and find a new haxelib_client library, the shim would prefer this version. The user could also haxelib set haxelib_client to choose a version, they could haxelib dev haxelib_client to work with a development version, or do none of these to use the original.

Prototype

I have built a prototype here:

https://github.com/jgranick/haxelib-shim/blob/master/Main.hx
https://github.com/jgranick/haxelib-shim/blob/master/build.hxml

It is currently working perfectly on my system. I can set, dev, or remove each version and it responds properly.

I would be happy to discuss this further.

The risk is very low, and something like this would forever improve the use of haxelib.

[Simn]

There have been similar discussions before but I can't find them right now. I think we agreed that the current version of selfupdate is not really sustainable, but I don't remember if there was a consensus on alternatives.

[ncannasse]

That would be nice if people having update issues could comment on the proposed implementation

[ncannasse]

@jgranick could you confirm on which system you have been testing the changes ?

[jgranick]

I have successfully used this on both my Linux VPS and my Windows system, I will check Mac now

[jgranick]

It worked perfectly on the Mac as well

[jgranick]

I added build instructions to the README https://github.com/jgranick/haxelib-shim/blob/master/README.md

[ousado]

What about the distinction of system-wide and per-user installs and upgrades respectively?

[Simn]

I like the general idea but I'm not sure about the concrete implementation. There are some path conventions going on there (if file x exists do this, else do that) which I was hoping would be something we back away from.

We should give this some more thought and aim for inclusion in the Haxe 3.3 release.

[larsiusprime]

March 2015: Haxe 3.2.0 (currently release candidate)
April 2014: Haxe 3.1.3
May 2013: Haxe 3.0.0
July 2012: Haxe 2.10

Seems that there's about a 1-year release cycle on Haxe. If we wait until 3.3 does that mean it will take another year before this feature becomes available? That's a bit long in my opinion.

[jgranick]

It has haxelib's same logic

[ibilon]

As someone who had to manually help the haxelib selfupdate succeed I'm all for this idea.

I'd even go a little bit further and make haxelib setup download the haxelib_client library, that way haxelib upgrade would also upgrade haxelib itself without the need to activate it with a selfupdate (which at lot of people don't know about).

And for the implementation itself and the way it finds the haxelib folder, seems good to me.

@larsiusprime +1
haxelib isn't as mature as the haxe compiler, could use a smaller release cycle, which is the all point of facilitating haxelib's selfupdate.

[back2dos]

Thanks, I will look into this now. We've still got one week until 3.2.0 final, the way I understood it.

[Simn]

Yes but let's not do this for 3.2.0, this really isn't the kind of change you make between RC and release.

I'm not opposed to making another 3.2.x release in a couple of weeks once we have a bit more data on this than "it works on Joshua's computer".

[ncannasse]

@Simn in general I would agree with you, but it seem many people have had issues updating haxelib, so if we ship the current one with 3.2.0-final and it can't be updated, it means people will be stuck with it until 3.3 (or longer if they don't update to 3.3 immediately). I think we should make sure the update works well, then we can do all the necessary changes post 3.2 and people will be able to update haxelib independently from Haxe

[Simn]

That's why I said I'm not opposed to making another dot release in a couple weeks. I find this change too drastic to just implement it over lunch. It might very well break more than it fixes and then I'm the idiot release manager who let something like this in a stable release.

[ncannasse]

@Simn release is a collective responsibility, don't take it as a personal one because it's not

[Simn]

It's hard not to when I'm the one handling the entirety of it. I didn't really mind having the responsibility, but in combination with not having the power to make final decisions about it it's rather frustrating.

So... you guys do what you think is right. I'll make the 3.2.0 release either way and then we'll see what happens.

[ncannasse]

@Simn what I meant is that while it's your responsibility to take care of the release, don't take it a personal offense when there are broken things that were not because of your choices/actions. I can understand it can be frustrating, let's discuss our release process at WWX

[jgranick]

Any updates on this?

We still are getting reports from users who cannot use OpenFL or Lime due to problems from older haxelib releases. Avoiding this scenario in this future (with the easy upgrade), would REALLY help.

Thanks everyone

[ibilon]

A little feedback, I've been using it for the last 15 days,
compiling, listing, installing, removing, setting dev version, getting path...
no issues for me (linux 64, haxe 3.2 rc.2, git haxelib, neko 2).

[Simn]

So... am I supposed to wait for this or not?

[back2dos]

I don't think so.

This is just a proof of concept and I'm quite certain that it unfortunately won't live up to Joshua's enthusiasm. The build merely uses nekotools boot which produces binaries that still have a dependency to neko (i.e. the required DLLs). Not getting a straight answer on whether that's ok, I'll have to assume it is not.

We could also work around the dependency, if all the required code were statically linked, in which case we roughly have a ~1MB binary on our hands. Which might be fine, I don't know. And even then, with xcross being covered in a thick layer of dust, I have no real idea how to actually build one.

Also, this proposal assumes the presence of neko a second time, by blindly preferring neko binaries. Assuming we bundle precompiled versions of haxelib.n in haxelib_client then running haxelib upgrade on a system without neko, you get a broken haxelib.

In the case of a statically linked executable, we might be able to use neko.vm.Loader to delegate work to the newer neko module. It's been ages since I have played with that, so I don't know how easy it will be.

The general idea is valid of course. But there's still a lot of ground to cover from the status quo to a state where "The risk is very low, and something like this would forever improve the use of haxelib."

So we need:

1. A viable way to run haxelib on every Linux machine. These are the options I see:
   • a neko binary with statically linked dependencies
   • accepting neko as a dependency (in which case we don't need a shim to start with, but ok)
   • a thinner shim that uses haxe --run with the smallest possible dependencies (to not impact compilation speed, which seems to have Joshua worried). This to me seems the most defensive choice.
2. Tests. I think this is big change. Without the slightest bit of coverage, I have absolutely no confidence in such a change at this point of Haxe's release cycle.

I currently lack the time to do the above to a level of quality that I would feel comfortable with, even less so with the pressure of the whole Haxe release depending on it.

Making haxelib a script again should solve the problems that Linux users are experiencing right now. And then we could actually deploy any experiments in this direction to the crowd that dares to use haxelib selfupdate (by having tools.haxelib.Main become the shim, that then invokes neko run.n or --run tools.haxelib.Client or what not), so that by the next Haxe release we have a field tested solution.

[ousado]

• Is the shim only supposed to solve the "issue" of requiring access to privileged directories and if not what else?
• is a haxelib selfupdate supposed to be able to perform an update for a system-wide haxelib installation without admin rights? If yes, how is it supposed to do that, if not, what's the problem?
• How does accepting neko as dependency change the requirement for a shim, re: "accepting neko as a dependency (in which case we don't need a shim to start with, but ok)"?
• What are the semantics of haxelib selfupdate? What's 'update.hxml'?
  A google search: site:haxe.org "haxe update.hxml" selfupdate doesn't answer these questions.
• What are the policies to ensure that the "identically functioning" shim can deal with whatever an updated haxelib did, in case an update becomes unavailable, another update fails, or similar?
• The current prototype just passes arguments on to the respective haxelib binary, does that mean that haxelib selfupdate is constrained to a certain set of command line arguments, or is the shim only "identically functioning" as long as no arguments are added?
• Is there any documentation of the design/architecture/inner workings of haxelib?

[jgranick]

Use of the shim would allow common users to have a current version of haxelib installed.

This would open the door for us to make continued efforts to improve haxelib

[ibilon]

Duplicate of #172?

[nadako]

Not technically a duplicate, but it was solved by #291 anyway.