diff --git a/_posts/2024-12-20-jank.md b/_posts/2024-12-20-jank.md new file mode 100644 index 0000000..b73a371 --- /dev/null +++ b/_posts/2024-12-20-jank.md @@ -0,0 +1,235 @@ +--- +title: "The jank programming language" +layout: post +excerpt: "Learn about jank, the native Clojure dialect on LLVM, +and how it uses Clang and CppInterOp to stand out from the pack. +Discover how Clojure users develop and how jank brings that to +the native world." +sitemap: false +author: Jeaye Wilkerson +permalink: blogs/jank_intro/ +banner_image: /images/blog/jank_intro/logo.png +date: 2024-12-20 +tags: jank clojure clang clang-repl cppinterop +--- + +# Intro +Hi everyone! I'm [Jeaye Wilkerson](https://github.com/jeaye), creator of the +[jank programming language](https://jank-lang.org). jank is a dialect of +[Clojure](https://clojure.org) which I've been working on for around eight years +now. To start with, jank was an exploration of what my ideal programming +language would be. Coming from a background of systems programming and game +development, I was very comfortable with C++, but I also was exploring the +world of functional programming. All of this led me to try Clojure, which +ultimately refined the direction of jank altogether. + +My work on jank is made possible by Clang, LLVM, and the tools coming out of the +[Compiler Research Group](https://compiler-research.org/), including Cling/Clang +REPL and [CppInterOp](https://github.com/compiler-research/CppInterOp). + +The past two years of jank development have been exciting as I've focused on +building jank into a native Clojure dialect on LLVM. This marries the lispy +world of REPL-driven, data-oriented, interactive programming and the C++ world +of light, fast, native runtimes. jank isn't production ready yet, but it's come +a long way in the past two years. + + + + +# What is Clojure? +Clojure is a modern, practical take on Lisp. It's functional-first, has +persistent, immutable data structures by default, but allows for adhoc side +effects. As it's a lisp, it has a powerful macro system which allows devs to +transform their own code using the same functions they use to transform any +other data. Clojure is dynamically typed and also garbage collected. + +Clojure has a rich tooling story for interactive development. Clojure devs +generally connect their text editor directly to their running programs, via a +REPL (read, eval, print, loop) client/server. This allows us to send code from +our editor to our program to be evaluated; the results come back as data which +we can then continue to work on in our editor. This whole workflow allows any +function to be redefined at any time during development and it allows for a +development iteration loop like nothing else. + +Clojure is a hosted language, which means it has a symbiotic relationship with +the host environment on which it runs. By default, Clojure is on the JVM. +Clojure JVM enables seamless interop with other JVM languages like Java, Kotlin, +Scala, etc. However, Clojure runs on many other hosts. For example, +ClojureScript runs on JavaScript (browser or node). ClojureCLR runs on the CLR +(where C#, F#, etc run). There's also ClojureDart, ClojErl (BEAM), and many +others. Each of these dialects provides seamless interop with its host. Since +Clojure is always hosted, things like socket libs, filesystem libs, etc are +never included in Clojure; they just come from the host. + +Clojure on a native host, though, is where jank comes in. + +# What makes jank special? +jank provides all of the interactive functionality of Clojure, which involves +JIT compiling native code with LLVM, while also providing seamless interop with +the native world. For those who are familiar, jank for Clojure is basically +analogous to Clasp for Common Lisp. However, compared to Clasp, jank aims to +provide even more seamless native interop which doesn't require making "bridge" +files. + +## Interop features +In jank, when you require a module (like including a header in C++ or importing +a module in Python), that module may be backed by a jank file or by a C++ file. +If it's backed by a C++ file, jank will JIT compile that file and then bootstrap +it by invoking a well-known function. This allows you to vender C++ code +alongside your jank code within the same project seamlessly. + +For example, take a look at this jank file which requires both a jank dependency +and a C++ dependency. They're both required in the same way. When +`clojure.pprint` is required, jank will find that the module is backed by a +`.jank` file, so it will JIT compile that file. However, when `sleep-native` is +required, jank will find that it's backed by a C++ file and it will JIT compile +that using `clang::Interpreter`. In either of these cases, if a `.o` is present +and up to date, jank will load that instead. + +```clojure +(ns nap-time + (:require [clojure.pprint :as p] ; JIT loads a jank module + [sleep-native :as s])) ; JIT loads a C++ module + +(defn main [] + (s/sleep 500) + (p/pprint "napped!")) +``` + +The `sleep-native` module can be backed by a C++ file which looks something like +this. jank will look for a special `jank_load_sleep_native` function in the +module, based on the name of the module itself. Note, the `-native` suffix is +just a common pattern used for modules backed by C++ files. + +```cpp +object_ptr jank_load_sleep_native() +{ + /* Create the sleep-native namespace so it's exposed to the jank world. */ + auto const ns{ _rt_ctx->intern_ns("sleep-native") }; + auto const intern_fn{ [=](native_persistent_string const &name, auto const fn) { + auto const boxed_fn{ make_box(fn) }; + ns->intern_var(name)->bind_root(boxed_fn); + } }; + + /* Create a var holding a pointer to the sleep function. */ + intern_fn("sleep", [](object_ptr const ms) -> object_ptr { + auto const duration(std::chrono::milliseconds(to_int(ms))); + std::this_thread::sleep_for(duration); + return nil_const; + }); + + return nil_const; +} +``` + +Overall, this automatic module support allows for arbitrarily complex C++ files +to be loaded, which will generally be used to wrap some existing native code and +use jank's C or C++ API to expose that code to the rest of the jank system. +However, this is all still roughly equivalent to "bridge files" and I aim to do +better. + +On top of that, jank will also support interop directly from jank code into C++ +land, allowing for the instantiation of native C++ objects as locals, calling +native functions, and also instantiating C++ templates on the fly. This is where +[CppInterOp](https://github.com/compiler-research/CppInterOp) comes in. At this +point, it's only in design phase, but I aim to utilize CppInterOp to its fullest +so that we can allow for arbitrary C++ values within jank, calling C++ +functions, and working with C++ data types. The working design looks something +like this: + +```clojure +; Feed some C++ into Clang so we can start working on it. +; Including files can also be done in a similar way. +(c++/declare "struct person{ std::string name; };") + +; `let` is a Clojure construct, but `c++/person.` creates a value +; of the `person` struct we just defined above, in automatic memory. +(let [s (c++/person. "sally siu") + ; We can then access structs using Clojure's normal interop syntax. + n (.-name s) + ; We can call member functions on native values, too. + ; Here we call std::string::size on the name member. + l (.size n)] + ; When we try to gives these native values to `println`, jank will + ; detect that they need boxing and will automatically find a + ; conversion function from their native type to jank's boxed + ; `object_ptr` type. If such a function doesn't exist, the + ; jank compiler fails with a type error. + (println n l)) +``` + +All of this will work with RAII, allow auto-boxing, but also give complete +access to C++ from within jank. + +## AOT features +With jank, you can AOT compile your programs to two different runtimes: + +1. A dynamic runtime which enables REPL usage, further JIT compilation, + redefining functions, etc +2. A static runtime which strips out all JIT capabilities, enables LTO, and + directly links functions instead of going through vars + +Static runtime binaries will also be able to be statically linked, providing a +portable binary which will very easily run on any distro or in any docker +container. + +# Why would people use jank? +There are two main audiences for jank. + +## Clojure users +jank's runtime is significantly lighter than the JVM. Since its runtime is +specifically crafted for jank, rather than being a generic VM, it's also able to +compete with the JVM in terms of runtime performance while keeping memory usage +lower. + +On top of that, anyone who is using Clojure and wanting easier access to native +libraries, tighter AOT compilation, etc, without sacrificing the fundamental +Clojure interactive development workflows will choose jank. + +## Native users +In game development, where my career has been focused, we build our engines and +games in C++, generally. However, we very often include a scripting language +like Lua in the engine so that we don't need to write all of our game logic in +C++. jank is an excellent fit here, too. Not only does it provide seamless +interop with the existing C++ engine/game code, but the REPL-driven workflows it +supports will allow you to spin up your game once and then send code to it, get +data back, and continue iterating on the code and data shapes without ever +restarting your game. + +Naturally, this is a game development focused use case, but the same thing +applies to any existing native code base which wants to utilize a higher level +language for a tighter iteration loop. + +# Next steps +jank is not yet released in production, but I aim to have it out the door in 2025. +In fact, as of January 2025, I'll have quit my job at Electronic Arts to +focus on jank full-time, unpaid aside from some small sponsors. I'm aiming to +get jank released, gather initial feedback, and help existing Clojure users +onboard jank into their systems. + +The three big tasks that are remaining before jank is released are: + +1. Improved error handling (following the recent work on Elm, Rust, and others) +2. Finalized C++ interop support (using the CppInterOp library) +3. Finalized AOT compilation support + +# Future plans +In the long term, since I'm building my dream language, I won't stop at just a +native Clojure dialect. jank will always be able to just be that, but I'll also +be extending it to support gradual typing (maybe linear typing), explicit memory +management, stronger pattern matching, value-based errors, and more. This will +allow another axis of control, where some parts of the program can remain +entirely dynamic and garbage collected while others are thoroughly controlled +and better optimized. That's exactly the control I want when programming. + +# Thank you! +For everyone who has worked on Cling, Clang REPL, and CppInterOp: Thank you! +jank is possible because of the magic you folks have created. To Vassil, in +particular, thanks for all of the support over the past couple of years as I +onboarded with Cling, ported to Clang, and then ultimately picked up +CppInterOp as a means to tackle seamless C++ interop. + +## Would you like to join in? +1. Join the jank community on [Slack](https://clojurians.slack.com/archives/C03SRH97FDK) +2. Join the jank design discussions or pick up a ticket on [GitHub](https://github.com/jank-lang/jank) +3. Join the Compiler Research community on [Discord](https://discord.gg/Vkv3ne4zVK) diff --git a/images/blog/jank_intro/logo.png b/images/blog/jank_intro/logo.png new file mode 100644 index 0000000..da0c8a7 Binary files /dev/null and b/images/blog/jank_intro/logo.png differ diff --git a/images/blog/jank_intro/star-history.png b/images/blog/jank_intro/star-history.png new file mode 100644 index 0000000..75aedfe Binary files /dev/null and b/images/blog/jank_intro/star-history.png differ