"Something like haxeshim"

[Simn]

We discuss this at every meetup but never opened an issue about it, so let's do that. The idea is to have something official that works similar to haxeshim. From what I gather, there are two main tasks for such a tool:

• deal with dependencies: figure out the dependencies, e.g. from -lib arguments, and prepare them for the actual compiler to consume
• allow self-updating the compiler: have a separate executable so that the actual compiler executable can easily self-update

I don't know much more about haxeshim, so I'll need some input here from various people: @back2dos @nadako @Gama11

I'm willing to work on this, but I'll need a clear understanding of what exactly we want to do.

[Gama11]

I think you would have to replace all hardcoded haxelib invocations in the compiler - that's not just haxelib path, but also haxelib run hxcpp/hxcs/hxjava. IIRC @back2dos suggested something like haxe -lib hxcpp --run Main could work?

[nadako]

I think these two things can be approached separately:

• locking down compiler version. this should be handled by some frontend executable that reads some file (e.g. `.haxerc) and determines which actual haxe executable to run, as well as setting up necessary environment for it

• locking down library versions. this can be either handled by the same tool (i.e. resolve -lib into -cp+-D and pass it to the real compiler like haxeshim does), or it can be done inside the compiler itself (we can already just look for haxe_libraries or whatever and use it instead of the current haxelib path calling).

these are the things that require work in the compiler itself, however the end user also needs a tool that downloads and places things in the correct places. so either lix should become an official tool, or haxelib should provide the same functionality. or both.

also we need to design some format for locking versions. the current haxeshim approach (.haxerc+haxe_libraries) works fine, but I'm still not sure if we should instead just have a single haxe.json that contains all the info (also potentially more things for the future, e.g. project-specific define descriptions and whatnot)

[RealyUniqueName]

I think we should go another way:

• compiler should not know about libraries and (probably) hxml
• have a separate tool, which gathers classpaths from libraries, handles hxml, composes a final list of compiler arguments and then invokes the compiler. This tool could also become a replacement for haxelib at some point.

[ncannasse]

Could we start by listing the actual problems to solve before discussing the actual solutions ? we want to make sure solutions are fixing actual problems ;)

[ncannasse]

Problem P1 : being able to install side by side several haxe compiler versions and be able to specifiy at compile time which version to use.

[kevinresol]

• The compiler should be self-sufficient. i.e. Remove haxelib dependency (hence neko)
• Lock down library versions per project

[back2dos]

Could we start by listing the actual problems to solve before discussing the actual solutions ?

Fully reproducible builds. That is the main problem. The rest is icing on the cake.

[ncannasse]

For fully reproducible builds we need to be able to :

• P1 select compiler version
• P2 pin the haxelib libraries

P1 can be handled with haxelib : if the haxe compiler is a library, then we can resolve its path, install and publish versions, etc. It then only requires a parameter for the "shim" to select the compiler.

P2 can already be acheived by explicitely setting the version in hxml : -lib name:version, but that's not very practical as one will often forget about it, and you might have several hxml that would require to be synchronized wrt version upgrade.

So for P2 it would require something similar to npm package-lock.json that store the installed version for a set of libraries (+ dependencies). haxelib-lock.json would be read/written by haxelib. No need to have the libs in local repository for that, that's a different problem.

Do I forget anything ?

[nadako]

That's more or less what haxeshim does: it's a "frontend" that runs a correct haxe binary with correct env vars based on selected version. and the haxe_libraries folder serves the same purpose as the "haxelib-lock.json" - it contains full information about libraries: some metadata for installing this exact version and compiler arguments (so the compiler binary don't have to depend on haxelib and run haxelib path itself).

While it would be awesome to have compiler as a haxe library, this would require more thought and work because: a) we need a way to install binaries for different architectures and b) we want an easy way to install nightlies. And I don't see anyone working on haxelib anytime soon. That said, this is not a blocker for implementing the haxeshim-like thing, it doesn't care how libraries/compilers are installed, it simply looks for them in the folders specified by the convention.

Anyway, we can discuss the actual format of the lock files and how to publish haxe as haxelib and whatnot, but in any case we need to implement this "frontend" executable that runs the specified version. I would (again) suggest we start with this first and continue thinking about libraries after that.

[grepsuzette]

Also we need to design some format for locking versions. the current haxeshim approach (.haxerc+haxe_libraries) works fine, but I'm still not sure if we should instead just have a single haxe.json that contains all the info (also potentially more things for the future, e.g. project-specific define descriptions and whatnot)

@nadako Having dependencies as mere files e.g. haxe_libraries/hx-yaml.hxml (as opposed to a big file with all dependencies) is very convenient when you want to copy one from another related project using the terminal, as opposed to having to: edit a file, find the line, ignore all surrounding information, do whatever you want, save, close. (With a locate haxe_libraries/hx-yaml.hxml you could even list all projects having that lib as a dependency). I of course see things from my end, but there are clear advantages with this approach, not even talking about how since it would use json, a cat haxe.json would be hard to read compared to ls haxe_libraries, yet it's something that one would have to use very often.

[benmerckx]

In addition to @emugel's remarks, having separate files for dependencies is more readable for git diffs (and githubs interface might hide the details).

[nadako]

Well, these are good points and I personally have no issue with haxe_libraries :)

[ncannasse]

Been discussed extensively today on Slack, with the following document as result
https://github.com/HaxeFoundation/haxe/wiki/Haxe-haxec-haxelib-plan
This is WIP , please discuss there and don't edit directly ;)

[grepsuzette]

I'm going to challenge the problem Fully reproducible builds.
It is a worthwhile goal, but not in the haxe scope.

Haxe is a compiler.
I sure see the point of having a version manager, being a lix user, but not why this service should be provided by Haxe.

Can gcc, python, java do that?
Does node tries to be node + npm + yarn + nvm ?
The fact that there exists alternative tools is not a bad thing.

Tools necessarily have a shorter life-cycle than compilers.
Merging the compiler and the tools fragilizes the compiler; by swiping away competition and ideas that are brought by it, by sticking to one central idea.

It also goes against the unix principle (doing one thing well).

What is needed in my opinion is merely for Haxe to behave cleanly, so as to facilitate and almost make trivial the work of external tools such as lix. And then focus on being an excellent compiler.

Quoting @RealyUniqueName earlier in this thread:

I think we should go another way:

• Compiler should not know about libraries and (probably) hxml have a separate tool, which gathers classpaths from libraries, handles hxml, composes a final list of compiler arguments and then invokes the compiler.
• This tool could also become a replacement for haxelib at some point.

[Aurel300]

@emugel I think that the latest plan that's on the wiki actually does what you are asking for? (It may be different somehow from what was said earlier in this thread but that's not important.)

Merging the compiler and the tools fragilizes the compiler;

It's the opposite.

The latest proposal is for the compiler, the front-end, and the package manager to all be separate tools. Importantly, -lib should no longer be a part of the compiler and instead should be resolved with a convention in Haxe "lock files". This setup allows a third-party package manage if one happens to pop up. It is actually less locked down than the current situation.

Additionally, although it is a nice sentiment to want to let third-party tools solve the situation (and lix certainly helped), I don't really see the problem of Haxe Foundation also developing the new version of these tools for now. Something like npm does not happen overnight and requires heaps of work and infrastructure.

[mundusnine]

I will chime in with a different perspective:

Firstly, while adopting similar concepts to what as been done in the js world could attract people from that field, we should probably ask ourselves, should we really go there ?

When starting or even attempting to do js stuff people have to grasp packaging, locks and all the throw away knowledge that incurs. Npm is a mess and doubles the information across the web, i.e. projects have a git and put their releases on npm or unpkg or what have you.

I believe a better way for projects to version the compiler and their libs would be to use something most, if not all projects use to version their code base, git.

It's way easier to grasp, in my view, for new users when we use a submodule approach. For new users, we can just say when you clone our repo do: git clone --recursive https://github.com/you/mycoolgitproject.git and they have everything versioned. Also, it's not throw away knowledge to learn what is a submodule or to better understand how to use git.

They comeback in a year and want to compile the project, the project still compiles because the haxe compiler used for the project/framework is versioned to that projects HEAD and the same is true for all its dependencies.

Now the issue that remains is how will libraries version themselves for the outside world ? The best approach in a situation where we use submodules is to have releases based on the compiler version used. That way, if a project is still using the haxe compiler 3.4.7, it will use the releases(or tagged commits with the release) that supports that version of the compiler.

Another advantage of such a method is there is no work to be done to create an npm or packaging tool. We only need to create a standard for haxelib creators to adopt(and dont accept them as official if they dont) and tell frameworks to version their haxe compiler in their git repo that they already use. This means, no need to push a new tool on projects, just use the tool they already use differently. Also, we avoid needing a full wiki page dedicated to understanding this complex scheme that we create for the haxe world and teach people how to better use a tool that is used everywhere.

Kha is a great example of this method working well and gives Fully reproducible builds.

Issue: "But we use perforce ?" Then use perforce streams.

Issue: "But we use svn ?" Then use svn:externals with -r commit_url.

[Aidan63]

Any chance of this happening for haxe 5? I've implemented most of the frontend program described in that haxec wiki page if that helps get the ball rolling. https://github.com/Aidan63/haxef

So I could test things without renaming the current compiler the frontend is just called haxef. It doesn't handle all the next and each mess just yet though, but works apart from that. each and next now works for the sane case, i.e. a single hxml file.

[tobil4sk]

Just to say also, I've got the haxelib side of that proposal WIP as well, currently blocked by: HaxeFoundation/haxelib#607 (which has problems due to some neko issues). I have put some thought into a lot of the details of they way the version locking should work (particularly with git dependencies) so I hope to get back to it some day.

A long time ago I also tried to implement the haxe frontend, but I never got around to finishing it properly and my code wasn't very well organised. I think I got the --each --next behaviour pretty consistent with haxe at the time, just in case you might want to have a look @Aidan63: https://github.com/tobil4sk/haxe