Support for elm 0.19

[Chilinot]

Since Elm 0.19 is now released this should be updated to accommodate. It includes changes to the elm-package.json, one being it has been renamed to elm.json. It also seems to have some structural changes.

[gdotdesign]

I will not be updating this repository anymore because 0.19 makes is much harder to install and use packages with native code.

It would require a lot of time to reverse engineer the new native and install logic, and personally I'm not interested in Elm any more.

If anyone is willing to tackle it, just let me know and I'll add them as a collaborator.

I'll be updating the readme to reflect this.

[spookylukey]

I may be willing to tackle this, or to help.

If my initial inspection looks right, I don't think it can be that difficult. If packages in https://github.com/elm-lang/ and https://github.com/elm-explorations/ can do it, then everyone else ought to be able to. The only problem is that you have to put "elm-lang" or "elm-explorations" in the name field in your elm.json for the compiler to accept it. We could get around that by simply adding another "real-name" field or something like that, for use by elm-github-install

I think there may be significantly more hoops to jump through in your JS code to ensure it is compatible with the new --optimize functionality. Unfortunately the core team don't want to document these, it seems, but they shouldn't be too hard to reverse engineer.

The way I see it, native/kernel code is not going away (it can't, as long as Elm wants to target the web, because you need to use browser APIs), so there is no reason 3rd party native/kernel code is going to go away either. There may be more hoops to jump through, that's all, some of them warranted (e.g. the optimizations may require the JS to conform to certain standards), and some of them pointless, but the workarounds will be there.

[SidneyNemzer]

Someone on the Elm slack pointed out the the new compiler checks for new versions when running elm make, so it might mean the compiler needs to be patched to allow non-published packages

We're using Firebase which heavily relies on its JavaScript library, and since rewriting that in Elm or using ports will take more time than we have, we'll be on 0.18 for the foreseeable future.

[spookylukey]

@SidneyNemzer If elm make does have that behaviour and it can't be turned off, it would be strange, because it would mean you need internet connectivity to compile your code, which sounds nasty to me.

However, if there is no other way, and especially if others would lend a hand, I would be interested in patching the compiler for these use cases. Elm is a huge asset to me, and kernel code is simply the right (or the only) way to do some things. There have got to be enough other people in the same situation to make this workable.

[spookylukey]

@SidneyNemzer some quick checks - elm make downloads packages the first time, but after that doesn't require internet access to work, which implies that if elm-github-install can install stuff in the right way it is possible to get elm make to work without patching

However, the ~/elm/0.19.0 directory now has a lot more inscrutable binary files in them, which is unfortunate, and might require a lot of work to be compatible with.

One workaround for these is the fact that you could make all third party native code appear as if it was part of your own project. You would then (I think) only need to fake the "name" field in your elm.json to get elm make to compile your native code.

[SidneyNemzer]

Yeah, I'd expect the compiler skips checking for updates if there's no internet. But it will download new versions if it can. I wouldn't be surprised if the compiler trips up on packages that aren't published, but maybe it skips them.

[norpan]

I've looked into the compiler source, and it will be very tricky to "fool" elm make into accepting foreign packages. It checks the package server, downloads a list of packages (~/.elm/0.19.0/package/versions.dat) and will reject any package not in that file. The file is overwritten each time it queries the package server.

I think the most straightforward way to get foreign packages working is to patch the compiler source itself, to inject packages into the versions list from other sources too. Working directly with git should be ok, we don't need to have anything "published", we could just fetch all tags from remote, and consider tags like x.y.z as published versions and put them in the list.

I'm going to make some experiments with the compiler source and see what works.

[norpan]

And by the way, if we are patching the compiler, it's pretty easy to remove other restrictions, like access to effect modules/kernel code.

[rtfeldman]

EDIT 5 years later: You can see in the edit history of this comment what I originally wrote here; I was upset and said unkind things that I regret, and which nobody deserved to hear. I apologized at the time and I still feel I should apologize again, unequivocally. I was in the wrong here.

I'm editing this because even 5 years later, people are still linking to this comment (just this week on HN, most recently) and I want to be really clear that I regret this, I apologize for what I wrote, and it's a mistake I've thought about often in the years since. I should have been better.

[norpan]

While I don't quite agree with the strong words (I think it's very possible to see this as a way to try to make your own life easier, not as a way to try to undermine Elm) it makes sense to cool off for a few months and see what state Elm 0.19 will be in. We certainly don't want to alienate the core team.

Right now it works well for us to use 0.18 but as time goes by it will get harderbecause other packages and tools will move on. My issue right now is with the Phoenix saschatimme socket library, which works well in 0.18 with a very nice API but currently is impossible to write in 0.19 even with ports, because access to tasks is missing. The problem here is pretty clear, so I don't see much use in exploring, it's a matter of implementing. The problem is that I can't unless the compiler is patched to allow exploration. I don't see how I can help move that package forward, and this is frustrating.

@rtfeldman surely you can see this more as a question of frustration than hostility? It's easy to get trapped in a filter bubble when you are surrounded by like minded people and very easy to demonize others that just try to get by as best they can.

[rtfeldman]

@norpan I appreciate your perspective, but what you said is very different from what I quoted above. ("There is no reason 3rd party native/kernel code is going to go away either. There may be more hoops to jump through, that's all...")

I completely understand the frustration of being blocked by things not having first-class support in Elm - I've been there myself, many times! Here's a recent example of a monkey patch I was planning to use at work until I found a nicer solution that involved a server-side improvement which removed the need for front-end hacks. Another community member was in a similar boat regarding CSRF tokens, and I talked him through the server-side improvement we ended up going with; that's now what he's planning to do as well.

There is a massive difference between my using hacks to unblock myself and my using hacks to pull an ecosystem away from its stated design goals and toward the design goals I think it should have instead. This repo's description is "An alternative decentralized package manager for Elm," so the goals are clear.

We know the current elm/websockets design doesn't adequately cover Phoenix use cases, which is why it's being redesigned, but in the meantime - as Evan's post noted, if you're blocked on something, let's talk about how to unblock you! That collaborative process can benefit multiple people, by helping inform the redesign of the websockets package.

Ports are not the only tool for getting unblocked, and I'm pretty sure I've seen people on Elm Slack mention how they came up with localized workarounds until there's a websockets package that fully covers Phoenix.

[spookylukey]

@rtfeldman

I completely understand that these efforts will not be met with approval by the core team. I do not want to give a long reply here - because it would further derail this thread, which is attempting to do something positive, although you may find that hard to believe. I would like to continue by email, if possible, because I think we agree far more than we differ and I'm curious as to how our logic diverges.

I am actually trying very hard to contribute to Elm, although attempting to work with the Elm leadership is an extremely frustrating process (and I have several decades of experience working with open source projects). For example, I have been working hard on elm-fluent for some months now (not ready yet), which I hope will one da be a valuable contribution to the ecosystem (currently blocked at some key points on that front) . And I also think Elm-github-install contributes to Elm in a helpful way that aligns with Elm's own states aims.

The questions I have for you are:

Are you ready to assume good faith efforts to those who don't agree with you, instead of accusing them of attacking Elm?

Are you ready to consider that the Elm community is already far wider than the group of people who think that Evan is infallible?

Are you even ready to consider the possibility that our disagreements might stem from your lack of insight, or lack of personal detachment from Evan and Elm, not ours?

Thanks so much for your patience.

[rtfeldman]

I would like to continue by email, if possible

@spookylukey Sure, DM me on Slack and we can move the discussion to email.

[renatomassaro]

We know the current elm/websockets design doesn't adequately cover Phoenix use cases, which is why it's being redesigned

It's great to know that there's a fix/remake being worked on. Is there any place I can follow its progress? There's no other branch in the repository, nor a personal fork from Evan.

[rtfeldman]

@renatomassaro the problems are design challenges, not implementation ones. I suspect once the design gets worked out, implementation will be pretty quick!

[bgourlie]

If you understand the design goals, but don't agree with them, why not channel that in a positive way - e.g. by building something that fits your vision instead of directly working against Elm's design goals?

These kind of responses are extremely condescending considering Elm folks have been encouraging the use of Elm in production environments for years now. People have fought to use Elm at their place of employment only to have things broken in significant ways, and your response is, effectively: "How dare you not foresee future design goals. Stop complaining and write your own thing if you don't like it."

Here's an idea: If you plan on breaking things in significant ways in future releases, be up front about it when trying to sell Elm for production use. And don't be a passive aggressive jerk when people are reasonably upset about it.

[rtfeldman]

People have fought to use Elm at their place of employment only to have things broken in significant ways, and your response is, effectively: "How dare you not foresee future design goals. Stop complaining and write your own thing if you don't like it."

Here's an idea: If you plan on breaking things in significant ways in future releases, be up front about it when trying to sell Elm for production use. And don't be a passive aggressive jerk when people are reasonably upset about it.

Wow, I did not realize this was how I was coming across. I'm really sorry. I do not want to be a jerk, and I apologize.

Let me try to explain where I'm coming from in a more chill way.

The Unavoidable Breakage

The format of Kernel code will unavoidably have serious breaking changes over time, and upgrading it will get increasingly difficult. Evan has already made several (necessarily breaking) changes to its format to improve performance, but the potential future breakages are much, much bigger than they have been so far.

In the future Elm will likely compile to Web Assembly. Web Assembly is not designed to be handwritten by humans, so it's easy to imagine writing some complex kernel code today (e.g. a wrapper around D3.js, like the D3 wrapper in PureScript or the D3 wrapper in BuckleScript—this is not a hypothetical example) which becomes completely infeasible to upgrade in a future where Elm compiles to WASM.

Knowing this is coming is one of the reasons that even inside the core Elm libraries—the use case Kernel code was designed for—its use is being minimized. The new elm/time package does not rely on JavaScript's Date in part because there is no Date primitive in WASM. The 0.19 core data structures use more Elm code and less Kernel code in anticipation of that Kernel code needing to be rewritten in WASM.

This unavoidable problem also applies to iOS and Objective-C bytecode as a compilation target. As Elm evolves over time, there will be more and more compelling reasons to make changes to this kernel, so repeated breakage to Kernel code is inevitable no matter what.

Communication Breakdown

In the past, we've tried (unsuccessfully) to prevent people from having bad experiences with this unstable API by communicating publicly about its design goals, including how it's going to break and shouldn't be relied on.

Here is a post Evan wrote in 2015 outlining these design goals. This was not successful, so he wrote another post about these design goals in 2017. That didn't do the trick either, which led to a third one in 2018.

The only official Elm documentation about this private, unstable API has been posts like these, advising against relying on it. Whenever people have asked that we document the API, we've instead directed them to these posts. We've really tried to be clear about the fact that the Kernel API is not something anyone should rely on!

One reason we learned the "communicate publicly" strategy has not worked is that when some folks read the source for elm-lang/core, they saw what they assumed was a typical language feature, and had no way of knowing it was a private, undocumented API that was never designed for use outside core libraries.

That is totally reasonable! After all, how could they possibly infer that from the source code of the core libraries—the one place it was actually designed to be used?

Preventing Breakage

This was one of the motivations behind having the compiler give an error: it's the one communication channel everyone sees. The compiler error conveys things much more clearly than any number of public posts possibly could, and in retrospect this is what the original design should have been all along.

Kernel code is going to break. It's unavoidable, given wasm, iOS, etc. If people use it at work, their applications will end up being broken in future releases of Elm, and unlike other backwards-incompatible changes to the language, tools like the compiler and elm-upgrade won't be able to help.

One of the reasons for announcing the compiler change several months in advance of the release was so that we could collectively help anyone who was currently relying on it migrate to a stable alternative. (Ports, custom elements, etc.) It was really heartwarming to see people step up and help each other out like this!

You commented "People have fought to use Elm at their place of employment only to have things broken in significant ways." My goal in writing this is to prevent future breakages.

Since we know that Kernel code is going to break unavoidably, having elm make give an error is the best tool we have for preventing people from relying on it and having a painful upgrade experience in the future. The idea of distributing an altered version of elm make that is designed to remove this warning, so that people can build on an unstable foundation and inevitably get burned by it in the future...gave me a visceral reaction. Again, I apologize for reacting that way.

Hopefully this clarifies where I'm coming from. I really want people to have a good experience with Elm—not only today, but over the next 5, 10, 20 years as well. ❤️

[Chilinot]

@rtfeldman

That was very well written, and feels like a good closure to this ticket.

[bgourlie]

You commented "People have fought to use Elm at their place of employment only to have things broken in significant ways." My goal in writing this is to prevent future breakages.

I think there's an important distinction that should be made here: People understand things will break. People don't expect undocumented APIs to be stable. That's not what people are upset about here. People are upset about the fact that Elm essentially implemented package DRM.

Steps were taken to intentionally break things (such as this very project) that attempted to solve real-world problems that people had while using Elm. Hell, steps were taken to prevent people from even tinkering locally with kernel code. There's a not-so-subtle line between designing a language with certain goals and going out of your way to prevent users from using it in a way that you don't approve of (which, by the way, is totally counter to the spirit of open source).

Ultimately, the argument being made here is "we're just saving you from yourself!" Which again is rather condescending.

The reasoning is also mildly annoying. The tree shaking point is based entirely on the straw-man argument that people want to write kernel code for the sole purpose of creating bindings to monolithic javascript libraries, instead of the far more common desire to create minimal bindings to browser APIs. The "we want to eventually target web assembly" argument ignores the fact that again, these are minimal bindings, and most of the value comes from having a nice interface in the Elm world for kernel-dependent things.

All this sounds like fluff to cover for something that Evan has fairly plainly stated in the past: He wants to have a say in the design of everything that interfaces with kernel code, and that he's concerned about the risk of people adopting things he doesn't like. And, while I don't agree with this, and acknowledge he's well within his right to operate in this fashion, at least it's honest and gives people a much better idea for what they're in for when deciding whether or not to use Elm.