Challenging the Rune Primitive

[josepot]

In this write-up, I aim to initiate an open discussion surrounding the Rune primitive, the foundation upon which CAPI is built. Additionally, I will scrutinize the CAPI approach itself:

Capi interactions are defined as "Runes", which are declarative, portable and strongly-typed building blocks for describing client-chain interactions. Runes enable potentially-complex interactions to be folded into their most minimal and parallelized form, so that Capi developers need not think about redundancy and timing.

While Runes are intended to simplify complex interactions by reducing redundancy and optimizing parallelism, I believe they fall short of achieving this goal. Instead, I think that Runes can often complicate developers' experiences with regards of redundancy, state management, and timing.

Full Disclosure

To provide full transparency, I must disclose that I am the author of another work-in-progress library that aims to serve as a viable alternative or replacement for PolkadotJs. I embarked on developing this library before CAPI's inception, and I temporarily paused its progress upon learning about CAPI: a Parity team's endeavor to create a library aligned with similar core objectives.

More recently, following my unsuccessful attempt to utilize CAPI for building a relatively simple dApp for the W3F, I decided to resume work on my WIP library.

I am also the co-author of a state-management library for using RxJS with React called React RxJS, and I am a contributor to the RxJS library itself. It's worth mentioning that I maintain a somewhat critical stance towards RxJS, which I will delve into further later on.

I am aware that my opinions may be influenced by certain biases that favor different mental models. To foster an honest dialogue, it is essential to acknowledge and be transparent about these biases, which is why I provide this information upfront.

Without further ado, let's delve into the discussion.

Runes vs Observables

Upon reading the Runes comparison with RxJS, it became evident to me that RxJS has struggled to effectively explain itself.

To summarize, the CAPI document comparing Runes with RxJS does the following:

• It suggests an arbitrary default behavior when combining different observables, without providing any justification or explanation for why this behavior should be considered the default or desirable.
• It then attempts to demonstrate the "incredible complexity" of achieving this "consistent" behavior by misusing RxJS.
• Next, it shows how easily this behavior can be achieved using Rune.tuple. However, it is important to note that the behavior shown in the 'Runes' examples does not align with the intended behavior claimed by their own statement: "Runes of a Feather Update Together".

Perhaps if RxJS had explained itself better, then the authors of the Rune system might have realized that achieving the desired behavior with RxJS (without inconsistencies) could be accomplished in just a single line of code:

const tuple = pipe(combineLatest, debounceTime(0))

Allow me to demonstrate this: first example, second example and third example

What a twist! In the end, it turns out that achieving the consistent behavior claimed by Rune.tuple is not only straightforward with RxJS, but the RxJS version accomplishes it without any inconsistencies at all.

It's clear that the authors of the Rune system have misunderstood RxJS. However, it's important to acknowledge that this misunderstanding is not entirely their fault. The confusion surrounding RxJS stems from its history of obscuring its core primitive and lacking proper decoupling between the primitive and its various enhancers (operators). Moreover, RxJS has been burdened with numerous "fluffy" operators in the past, which only added to the complexity.

Fortunately, improvements have been made with RxJS 6 and onwards. The library now puts more effort into explaining its core primitive. Additionally, steps have been taken to remove many of the bogus operators. However, it is worth noting that in my opinion, there is still room for further simplification in RxJS.

Unfortunately, many RxJS users fail to grasp the Observable primitive, leading to self-inflicted problems when misusing different operators.

Do I believe that the RxJS Observable is perfect? No, I don't. Personally, I think the RxJS Observable could be even simpler. Despite some ergonomic issues I find with its current API, I still consider it one of the finest Observable libraries available. Given its widespread adoption, it has become a de facto standard. Rather than reinventing the wheel, I prefer to contribute feedback and ideas to make it even better. However, I would much rather to see the Observable primitive being part of ECMAScript itself. Unfortunately, such a development appears unlikely in the foreseeable future.

All that being said, what I really like about RxJS Observables is that they are as simple as simple can be, The Observable primitive possesses the following characteristics:

• Laziness: Observables are evaluated only when a subscriber is present.
• Unicast: Each subscriber receives its own independent execution of the Observable.
• Composability: Observables can be combined and transformed in various ways.
• Stateless: The default Observable does not maintain internal state between emissions.
• Disposable (Abortable): Observables can be terminated prematurely if needed.
• Ability to emit values synchronously or asynchronously.

This simplicity allows for the addition of extra behaviors through enhancers (aka operators). For example, an enhancer can transform the source Observable into a multicast Observable, another enhancer can introduce statefulness by replaying the latest value, while another can dispose of the Observable under certain conditions, and so on. The beauty lies in the ability to customize and combine these enhancers in a way that best suits your requirements.

However, what if some of these "enhanced" behaviors were already integrated into the core primitive? It would limit the flexibility of creating composable enhancers. This limitation would be unfortunate.

This brings me to a drawback I find in Runes: they come bundled with excessive embedded behaviors and oversimplified assumptions, with no straightforward way to opt-out from them.

Runes are not a good primitive

Runes, as a primitive, have certain drawbacks that make them less ideal. Firstly, they come with embedded behaviors that consumers cannot opt-out from. Additionally, creating Runes from scratch to model push-based interactions is challenging.

The Rune primitive exhibits the following characteristics:

• Laziness
• Statefulness (including built-in memoization, which can easily translate into memory-leaks)
• Opinionated asynchronous value emission
• Non-abortable nature (which poses significant problems)
• Limited composability
• Multicasting behavior

Considering these pre-defined assumptions, it becomes difficult to customize the combination of asynchronous behaviors to best suit the needs of a dApp. The lack of flexibility is evident, and I will provide specific evidence of this later on.

While I could delve into each aspect in detail, let's focus on one critical aspect: composability. Regrettably, I have been unable to grasp how to compose Runes effectively.

For example, suppose I intend to create my own "runic" version of the map operator. How would I achieve this using Runes?

To illustrate, here's an implementation of a DIY map operator in RxJS:

const map = <I, O>(mapFn: (x: I) => O) =>
  (source: Observable<I>): Observable<O> =>
    new Observable((observer) =>
      source.subscribe({
        next(value) {
          observer.next(mapFn(value))
        },
        error(e) {
          observer.error(e)
        },
        complete() {
          observer.complete()
        },
      }),
    )

However, I have struggled to determine the equivalent process for accomplishing this using Runes. Despite my efforts to explore the codebase, it appears that achieving such functionality would be less straightforward.

Runes: Leaky and Not Light-Client Friendly

I find it perplexing that Runes have chosen async iterators for modeling push-based interactions. In my opinion, this choice is problematic on multiple levels, although it's challenging to pinpoint where to begin. Instead of delving into a lengthy explanation, I will illustrate the issue with a concrete example from their documentation.

If you have meticulously read the CAPI docs, you might have noticed the lack of a proper explanation on how to create a custom Rune for modeling events from an external source. However, there is a specific section where they provide the following code snippet:

function timer(ms: number) {
  return Rune.asyncIter(async function*() {
    let i = 0
    while (true) {
      yield i++
      await delay(ms)
    }
  })
}

Setting aside the irony that this snippet is presented in the context of generating synchronized Runes (which they won't be, due to the timer lacking a proper scheduler and causing events to fall out of sync), my immediate concern upon seeing this code was: "How can I stop an ongoing Rune? If I can't, then we have a major problem."

Allow me to demonstrate this problem. Since most developers are not well-acquainted with async iterators, let's first consider an example without Runes. Suppose we want to pull a promise that introduces a one-second delay with each pull, and after five pulls, we want to stop everything and continue. Although contrived, please bear with me. Here's how we can achieve that:

const delay = (ms: number) =>
  new Promise<void>((res) => setTimeout(res, ms))

const timer = (ms: number) => ({
  async *[Symbol.asyncIterator]() {
    let idx = 0
    while (true) {
      yield idx++
      console.log(`starting a ${ms}ms delay...`)
      await delay(ms)
    }
  },
})

let idx = 0
for await (const value of timer(1_000)) {
  console.log(`Done with delay #${++idx}`)
  if (idx === 5) {
    console.log("stop everything!!")
    break
  }
}

When you run this code, you'll notice that once the "Stop everything!!" message is logged, the program exits normally. Fantastic!

Now, let's see what happens if we attempt to use a Rune as suggested here, but with the intention of eventually stopping it, as we did before:

const delay = (ms: number) =>
  new Promise<void>((res) => setTimeout(res, ms))

const timer = (ms: number) => Rune.asyncIter(() => ({
  async *[Symbol.asyncIterator]() {
    let idx = 0
    while (true) {
      yield idx++
      console.log(`starting a ${ms}ms delay...`)
      await delay(ms)
    }
  },
}))

const a = timer(1000).map((n) => `a${n}`)
const b = timer(1500).map((n) => `b${n}`)

const ab = Rune.tuple([a, b])

const start = Date.now()
let idx = 0
for await (const value of ab.iter()) {
  console.log(Date.now() - start, value)
  if (++idx === 5) {
    console.log("stop everything!!")
    break
  }
}

If you try running this code (note: CAPI currently doesn't work on StackBlitz), you'll observe that even after the "Stop everything!!" message appears, the logs of "Starting a XXXXms delay..." continue endlessly 😱. This occurs because the Rune has no knowledge that the consumer of the async iterator returned from the Rune is no longer listening. As a result, the Rune keeps running indefinitely. Unfortunately, this behavior is not just about memory leaks; it is fundamentally flawed as the Rune fails to react to the disinterest of its consumer in the values it provides. I truly hope I'm missing something here 🤞, but based on my observations thus far, Runes do not seem to provide the semantics for canceling them 😬.

As a result, in my opinion, the Rune primitive is not light-client friendly. To properly leverage the new JSON-RPC API, which was designed to meet the requirements of light-clients, we need a primitive that allows us to stop subscriptions for resources that the consumer is no longer interested in. If the primitive lacks this capability, there is no way to fully utilize important features of the new JSON-RPC API. Consequently, since there appears to be no way for a Rune to determine when one of its consumers is no longer interested in a specific storage subscription, the Rune will keep pulling that resource from storage indefinitely. Meaning that, dApps built with Runes will eventually reach the maximum subscription limit imposed by the light-client.

Lack of Interoperability with Runes

In JavaScript, the absence of a standardized Observable primitive has led to the proliferation of various custom implementations. Mobx, XState, Redux store (although not ideal), Solid, Svelte, and many others have their own flavors of Observables. Despite this diversity, there is typically a certain degree of interoperability among these Observables. For example, an RxJS Observable can be seamlessly used as a Svelte store without any issues. Most of these Observables exhibit high levels of interoperability, and there are tools available to simplify the process.

However, Runes stand as a distinct entity, rendering them unsuitable for achieving basic interoperability with other Observable-like primitives. Runes differ significantly from the established patterns, making it challenging to establish seamless interoperability.

Conclusions and Challenge

While I initially intended to include two more sections, I realize that this write-up has become significantly longer than anticipated. If you've managed to reach this point, congratulations! 🤨 Although it's likely that you may have skipped ahead to the "Conclusions" section, right? If that's the case, I kindly request you to at least read the "Runes: Leaky and Not Light-Client Friendly" section, if possible. 🙏

In summary, my conclusions align with what I stated at the beginning: Runes not only fall short of delivering on their promised benefits but also introduce unnecessary complexity when dealing with asynchronous interactions.

However, I would genuinely appreciate being proven wrong. Therefore, I would like to present a challenge to the CAPI team: Solve a real-life problem, which is relatively simple yet non-contrived, using Runes. This problem is the same one that led me to abandon CAPI when building a dApp for the W3F. Initially, I attempted to use CAPI, but it proved inadequate for the task, so I reverted to PJS. Unfortunately, PJS exhibited sub-optimal behaviors that hindered light-client friendliness in the dApp. As a result, I decided to use my own work-in-progress library, which allowed the dApp to efficiently utilize the light-client.

Once the dApp was production-ready, I decoupled the section responsible for asynchronously pulling and combining all the validator data into its own repository. This facilitated easy access to the latest dataset for our colleagues at the W3F, enabling them to conduct their studies. The code in that repository is straightforward, and solving the same problem using PJS is relatively simple (though less performant). Hence, my request to the CAPI team is to tackle the same problem using CAPI. I genuinely believe that it should take them no more than a few hours to do so. This exercise would benefit everyone by facilitating a comparison of different mental models and trade-offs presented by the different libraries.

[jeluard]

@josepot Interesting read! Do you maybe have some writeup detailing Unfortunately, PJS exhibited sub-optimal behaviors that hindered light-client friendliness in the dApp. ?

Also, how is your own wip library different and adresses that?

Thanks

[josepot]

@josepot Interesting read!

Thank you!

Do you maybe have some writeup detailing "Unfortunately, PJS exhibited sub-optimal behaviors that hindered light-client friendliness in the dApp."?

I don't currently have a writeup available, but I would be more than willing to create one if there is sufficient interest!

In the meantime, you can implement this code using PJS and compare its performance to the version using unstoppablejs. You can refer to this existing code built with PJS as a reference, which the W3F was using before I built my own solution.

In a nutshell, the challenges arise from the fact that PJS was developed before smoldot was a thing. Assumptions were made based on the library's communication with a conventional RPC node, resulting in a mismatch with smoldot. This impedance mismatch makes it difficult to effectively consume smoldot using PJS. It's not just the lack of support for the new JSON-RPC API; there are also suboptimal behaviors within PJS, such as waiting for all the keys of a large storage entry before retrieving their values, which could be parallelized. While these optimizations may be insignificant when working with a conventional RPC node, they significantly impact performance when using the light-client. I've provided a high-level summary here, but I'd be happy to delve into more detail in a comprehensive writeup.

Also, how is your own WIP library different and addressing these issues?

Regarding my own WIP library, it's not appropriate to discuss it in detail here. However, I aim to create a library that avoids imposing new or unfamiliar primitives on its consumers, even with RxJS Observables not being first-class citizens of unstoppablejs. I hope to have opportunities in the future to engage in comprehensive discussions and comparisons of different approaches. For now, let's keep the focus on discussing the Rune primitive in this forum.

Thanks

Thank you!

[statictype]

If understand correctly, you are saying that the whole reason that the Capi team created this Rune system is because they were not able to achieve some behaviour with pure RX.js, when in fact is was an easy one-liner? Wow, that would be very sad. Looking forward to an explanation from the capi team and in general more technical docs!

[josepot]

If understand correctly, you are saying that the whole reason that the Capi team created this Rune system is because they were not able to achieve some behaviour with pure RX.js, when in fact is was an easy one-liner? Wow, that would be very sad

I can see how, when comparing the way CAPI has presented Runes in contrast to RxJS, along with the evidence I've presented in this writeup, one might arrive at such a conclusion. However, I want to clarify that this is not the precise point I'm trying to convey.

The truth is that I don't fully comprehend the reasons behind the creation of Runes or the specific problem they seek to address. However, I do have concerns that the creators of Runes may have a limited understanding of prior art in this field. Furthermore, considering the reliance of Runes on async-iterators, I worry that the associated trade-offs of this choice may not have been fully explored and understood.

Looking forward to an explanation from the capi team and in general more technical docs!

Likewise!

[tjjfvi]

I'll follow up here shortly, but for now, I'm going to merge all the conversation here.

[tjjfvi]

There are a lot of things being brought up here, and I hope to respond to them completely. To that end:

• To serve as a basis for discussion, I'm going to first explain what Rune is and how it works.
• I'm not going to focus on RxJS in this response, but focus on discussing "the Rune primitive". (Comparing Rune and RxJS has merit, and I can give an example of where RxJS cannot meet our needs in a separate thread.)

Note: a bit of this isn't yet covered in the docs. It should be, and it will be (#1146).

Rune

Rune has a "timing system". (It's important to note that the "time" in this system does not refer to clock time, but rather a Rune-specific construct of time that is used to track updates).

Its timing system is designed to ensure consistent results when combining subscriptions. Here, "consistent results" means that, in any pair of dependent runes (whether a direct or indirect dependency), the values yielded by the dependent rune each correspond to exactly one value yielded by the independent rune (i.e. the dependency).

For example, take the following rune dependency graph:

  a
 / \
b   c
 \ /
  d

Here, d is dependent on both b and c, each of which are dependent on a. Rune ensures that the values yielded by d each correspond to exactly one value yielded by a – meaning, the values of b and c from which any given value of d derives are themselves both derived from the same value of a.

Rune is designed such that this property holds in any Rune dependency graph.

This consistency is important for Capi, as it deals with combining subscriptions very frequently. For example, the above dependency graph appears in a very basic usage of Capi, where:

• a is a subscription to the most recent block hash
• b tracks the block number corresponding to a
• c reads the Timestamp.Now storage value at the block a
• d combines b and c to log out block numbers and their corresponding timestamps

If Runes did not guarantee consistency, this setup could produce inaccurate logs, where the block number and timestamps corresponded to two different block hashes.

The core properties of Rune's timing system can be viewed as follows:

• there is a shared timeline
• the timeline is indexed by times, which are natural numbers
• the timeline can be extended with new times
• times semantically correspond to external events
• runes can update at a specific time
• when a rune updates, its dependencies also update
• runes can be evaluated for a given time
• when a rune is evaluated for a time where it did not update, it keeps its old value
• when runes are evaluated for a given time, they evaluate their dependencies for that same time

So, in the above abcd example, a is the only "event source" (formally, the only rune that is extending the timeline). Whenever an external event comes in (in the Capi case, whenever there is a new block hash), it extends the timeline with a new time t, and updates at that time t. Consequently, b and c both update at t, which causes d to update at t. Then, d is evaluated for t, which uses the evaluation of b and c for t, which both use the evaluation of a for t.

Because evaluation of runes happens corresponding to a specific time, consistency is ensured – all parts of d's value will correspond to the time t, even if new a events occur while d is still evaluating for time t.

Responses to...

"Runes vs Observables"

However, it is important to note that the behavior shown in the 'Runes' examples does not align with the intended behavior claimed by their own statement: "Runes of a Feather Update Together".

In that issue, you:

• describe what you think Rune's intended behavior is
• observe that Rune does not conform to what you describe
• conclude that this is a bug in Rune

Your description of Rune's intended behavior is not accurate. Rune's timing system is not based on the task-queue or macrotasks. In the example, two different timers are used to intentionally create two independent event sources. The examples demonstrate that Rune's behavior does align with the actual intended behavior, as described above.

achieving the desired behavior with RxJS (without inconsistencies) could be accomplished in just a single line of code:

const tuple = pipe(combineLatest, debounceTime(0))

This accomplishes the behavior you identified, but not the actual intended behavior of Rune.

achieving the consistent behavior claimed by Rune.tuple is not only straightforward with RxJS, but the RxJS version accomplishes it without any inconsistencies at all.

This statement is based on your description of Rune.tuple's intended behavior, which is not accurate. (But I'll reserve further discussion of the RxJS comparison for another thread.)

A 3 ms discrepancy after so few emitted values seems to indicate that there is some serious overhead going on with the Runes, which will IMO not help at producing consistent behaviors.

This is not due to "serious overhead" in Rune; this is just run-of-the-mill timer drift.

Even if this was overhead, it wouldn't interfere with producing consistent values, as Rune is designed to produce consistent values no matter how much clock time certain things take.

"Runes: Leaky and Not Light-Client Friendly"

I find it perplexing that Runes have chosen async iterators for modeling push-based interactions. In my opinion, this choice is problematic on multiple levels, although it's challenging to pinpoint where to begin.

Rune doesn't use async iterators under the hood; they're simply used at the top level for a convenient API. It's worth noting that Rune is not solely push-based. Updates are push-based, but evaluation is pull-based. This combination is a big part of what allows Rune to yield consistent results.

If you have meticulously read the CAPI docs, you might have noticed the lack of a proper explanation on how to create a custom Rune for modeling events from an external source.

When we previously discussed this in person, I acknowledged that this does need to be documented, and it will be (#1149).

this snippet is presented in the context of generating synchronized Runes

This snippet is not presented in the context of generating synchronized runes. In the example, different timers are used to intentionally create two independent event sources.

This occurs because the Rune has no knowledge that the consumer of the async iterator returned from the Rune is no longer listening. As a result, the Rune keeps running indefinitely. Unfortunately, this behavior is not just about memory leaks; it is fundamentally flawed as the Rune fails to react to the disinterest of its consumer in the values it provides. I truly hope I'm missing something here 🤞, but based on my observations thus far, Runes do not seem to provide the semantics for canceling them 😬.

Runes can be cleaned up, and they are cleaned up when you break out of the for await. All of Capi's examples exit gracefully because of this automatic cleanup.

What you've observed is that when Rune.asyncIter is cleaned up, it does not stop pulling from the async iterator. This is a bug with Rune.asyncIter, and I'll fix it soon; thanks for noticing it.

(Aside: Rune.asyncIter is only used in a few tests (with finite iterators) and the docs, for simplicity – it's not used in Capi, as its pull-based API is not very useful in practice. Nevertheless, it is a bug and will be fixed (#1147).)

As a result, in my opinion, the Rune primitive is not light-client friendly. To properly leverage the new JSON-RPC API, which was designed to meet the requirements of light-clients, we need a primitive that allows us to stop subscriptions for resources that the consumer is no longer interested in. If the primitive lacks this capability, there is no way to fully utilize important features of the new JSON-RPC API. Consequently, since there appears to be no way for a Rune to determine when one of its consumers is no longer interested in a specific storage subscription, the Rune will keep pulling that resource from storage indefinitely. Meaning that, dApps built with Runes will eventually reach the maximum subscription limit imposed by the light-client.

"The Rune primitive" does support abortion, and no-longer-used subscriptions are cleaned up. There is no aspect of Rune that is not suitable for use with light clients.

"Runes are not a good primitive"

Firstly, they come with embedded behaviors that consumers cannot opt-out from.

When we previously discussed this in person, I explained that, though Rune does default to producing consistent results (as this is necessary for Capi), there are operators one can use to opt-out of this behavior where desired.

• Statefulness (including built-in memoization, which can easily translate into memory-leaks)

Runes are only memoized for as long as they are actively used. When a rune is no longer used, it is removed from the memoization. The memoization thus doesn't leak memory.

• Opinionated asynchronous value emission

Yes, Runes are fundamentally asynchronous, as this best aligns with Capi's use case. Everything Capi does is asynchronous, so Runes being fundamentally asynchronous is appropriate for using Capi.

• Non-abortable nature (which poses significant problems)

Runes are abortable, and get cleaned up automatically.

• Limited composability

There isn't a limitation on composing runes. Writing operators for runes does require more care than for observables, as runes are more complex (as they solve more problems than observables). Nevertheless, runes can be combined in many different ways.

• Multicasting behavior

Yes, this behavior is ideal for Capi – for example, when you submit an extrinsic and watch its status, it should only be submitted once, even if you listen to its status multiple times.

For example, suppose I intend to create my own "runic" version of the map operator. How would I achieve this using Runes?

Rune has a built-in map, and its implementation is not privileged.

It can be implemented as follows:

const map = <T1, U, T2>(source: Rune<T1, U>, fn: (x: T1) => T2): ValueRune<T2, U> =>
  ValueRune.new(RunMap, this, fn)

class RunMap<T1, U, T2> extends Run<T2, U> {
  source: Run<T1, U>
  constructor(
    runner: Runner,
    source: Rune<T1, U>,
    readonly fn: (value: T1) => T2 | Promise<T2>,
  ) {
    super(runner)
    this.source = this.use(source) // instantiate `source` and register the dependency
  }

  lastValue!: T2
  async _evaluate(time: number, receipt: Receipt) {
    // evaluate the source rune at the given time
    const source = await this.source.evaluate(time, receipt)
    // if the source rune is not ready, or it has not changed...
    if (!receipt.ready || !receipt.novel) {
      // skip calling `this.fn` and return the last value
      return this.lastValue
    }
    // otherwise, call `this.fn` and update `this.lastValue`
    return this.lastValue = await this.fn(source)
  }
}

"Lack of Interoperability with Runes"

However, Runes stand as a distinct entity, rendering them unsuitable for achieving basic interoperability with other Observable-like primitives. Runes differ significantly from the established patterns, making it challenging to establish seamless interoperability.

When we previously discussed this in person, I explained that, yes, because runes carry more information than observables, interop with observable libraries is more complex, but that it is not impossible, and that we will be building libraries/guides for this interop (in both directions of consumption) in the future (#1148).

"Conclusions and Challenge"

Given the information presented above, I suggest the conclusions be reconsidered.

Therefore, I would like to present a challenge to the CAPI team: Solve a real-life problem, which is relatively simple yet non-contrived, using Runes.

This is an interesting proposal; we will consider it and follow up here when we can.

[josepot]

excluding the bug with one particular rune

And when you mention "one particular rune," you're referring to the only vaguely documented means of creating a Rune from an external source, correct? I assume that's why you would be willing to share the proper way of creating a Rune from an external source that doesn't encounter this issue. I would greatly appreciate your insights on this matter. 🙏

[tjjfvi]

@josepot See https://docs.capi.dev/rune/advanced/custom-runes and https://docs.capi.dev/rune/interop#observable-libraries (paritytech/docs.capi.dev#20)

[josepot]

Thanks @tjjfvi ! A couple of things, though:

On the topic of cancelation:

1. How can I cancel a running Rune without waiting for the current Promise to resolve? For example, what if I need to abort the operation because the async value is taking too long to resolve? Is that possible, or is cancellation only possible between async iterations using break?

2. Additionally, how can I cancel a running Rune (with a pending Promise) if another Rune produces a value indicating that we should stop listening immediately?

Also, interoperability with Observables is broken.

[tjjfvi]

Is that possible, or is cancellation only possible between async iterations using break?

Good point – currently, using the .iter() api, this is not possible. I don't see anything under-the-hood that should prevent this, though. Perhaps an AbortSignal argument to .iter() is in order.

Additionally, how can I cancel a running Rune (with a pending Promise) if another Rune produces a value indicating that we should stop listening immediately?

This sounds technically feasible, but I'm not sure I fully understand your usecase.

Could you potentially give an example of this in RxJS?

---

@josepot Would you be willing to file issues for each of these?

[josepot]

Good point – currently, using the .iter() api, this is not possible

I appreciate that you've finally acknowledged the reality that, as it stands, Runes do not inherently possess a functionality or mechanism to enable proper cancellation.

Perhaps an AbortSignal argument to .iter() is in order.

It's important to understand that AbortSignal is not a feature of ECMAScript and that the specifications for DOM and Node diverge considerably from each other. Even its initial advocates have come to deeply regret attempting to integrate this concept into JavaScript runtimes. You are, of course, free to proceed as you see fit. However, I maintain my stance that Runes are fundamentally flawed. While you could potentially bypass this issue using AbortSignals, this workaround would inevitably lead you to confront the more challenging drawbacks associated with both Promises and Observables.

As such, I strongly encourage you to consider the possibility that Runes may not be a good primitive.

but I'm not sure I fully understand your usecase

I find myself at a loss for words...

Could you potentially give an example of this in RxJS?

Many essential RxJS operators capitalize on this behavior, which is very useful when building reactive applications with Observables. These include takeUntil, switchMap, race, and others. It's quite surprising to encounter someone seemingly not well-versed in these pre-established tools, which have been effectively used by millions, yet feels confident in proposing what they perceive as a superior primitive.

@josepot Would you be willing to file issues for each of these?

I don't see the point, to be honest.

[statictype]

Thank you for this detailed explanation!

Looking forward to the Rx.JS comparison clarification, where I'm hoping to understand more about what the actual intended behaviour of the Rune is and why existing, battle-tested observable libraries could not satisfy Capi needs, making the creation of these distinct entities a pragmatic choice.

I would like to urge once again the Capi team to test their claims outside the existing Deno examples, like in a node.js web app. We've had plenty memory leaks from Capi.

[statictype]

the gamma release breaks functionality. we were unable to update the app in a while unfortunately
paritytech/capi-multisig-app#217
so for what it's worth the memory leaks still happen in the multsig app. you can always try submitting a tx on https://multisig.capi.dev/ and see for yourself.

[harrysolovay]

This discussion is about Rune. #1054 is not Rune-related (see nodejs/node#47615).

Regarding your screenshot––I actually opened #1139 on your behalf. Can we pick this up there?

[statictype]

you are right about #1054, edited my comment.
#1139 is not at all about the screenshot. it's actually about the broken functionality in the latest Capi release.
the screenshot, which is a Rune related memory leak, is actually a much older issue.

[harrysolovay]

Ah, so the issue in the screenshot is resolved. You eluded to another memory leak issue ("plenty")?

Regarding #1139, I followed up there. If I missed anything...

Please open an issue with a link to the code or a reproduction.

[statictype]

You eluded to another memory leak issue ("plenty")?

plenty as in "it has been happening with every transaction for a couple months now"

Ah, so the issue in the screenshot is resolved.

I'm not sure! Not able to test all interactions with the new release where it's supposed to be solved due to this newly introduced error.