Encapsulate successful or failed function execution

[elizarov]

This issue is for discussion of the proposal to provide a Standard Library class to encapsulate successful or failed function execution.

Proposal text: result.md

[pakoito]

Re. flatMap and the current API: even if the Scala code looks similar it doesn't do the same. Don't fall in that trap. It's rewritten behind the scenes the same way coroutines do to a chain of operations. The return type is a Try<List<T>>, so to be equivalent your example would have to return SuccessOrFailure<List<T>>, and the same has to be returned from the other methods.

So taking the original code:

fun getURLContent(url: String): List<String> {
    val url = parseURL(url) // here parseURL returns URL, throws on failure
    val connection = url.openConnection()
    val input = connection.getInputStream()
    val source = Source.fromInputStream(input)
    return source.getLines()
}

What happens when parseURL(url), openConnection(), getInputStream() return SuccessOrFailure<T>? Do you have to use the getOrNull and nullcheck it every line? Then you lose information about the exception, and makes it more verbose. If you have to getOrThrow and try/catch every line then you don't have any improvement over the current state of the world with normal exceptions. I'll write 4 snippets each with a different way of chaining these operations and are all valid (save for bugs :P) with the proposed API:

fun getURLContent(url: String): SuccessOrFailure<List<String>> {
    val url = try { 
                  parseURL(url).getOrThrow()
                } catch (e: ParseException) { 
                  return SuccessOrFailure.failure(e)
                }
    val connection = try {  
                  url.openConnection().getOrThrow()
                } catch (e: ConnectionException) { 
                  return SuccessOrFailure.failure(e)
                }
    val input = connection.getInputStream().getOrThrow() // anything thrown here is non-recoverable
    val source = try {  
                  Source.fromInputStream(input).getOrThrow()
                } catch (e: IOException) { 
                  return SuccessOrFailure.failure(e)
                }
    return SuccesOrFailure.success(source.getLines())
}

fun getURLContent(url: String): SuccessOrFailure<List<String>> {
    val url = parseURL(url).getOrNull()
    if (url != null) {
      val connection = url.openConnection().getOrNull()
      if (connection != null) {
        val input = connection.getInputStream().getOrThrow()
        val source = Source.fromInputStream(input)
        if (source != null) {
          return SuccessOrFailure.success(source.getLines())
        } else  {
          return SuccessOrFailure.failure(SomethingException("1"))
        }
      } else {
         return SuccessOrFailure.failure(ConnectionException())
      }
    } else {
      return SuccessOrFailure.failure(ParseException())
    }

fun getURLContent(url: String): SuccessOrFailure<List<String>> {
  return runCatching {
      val url = parseURL(url).recover { if (it is ParseException) { return SuccessOrFailure.failure(it) } else { throw it }  }.getOrThrow()
      val connection = url.openConnection().recover { if (it is ConnectionException) { return SuccessOrFailure.failure(it) } else { throw it }  }.getOrThrow()
      val input = connection.getInputStream().getOrThrow()
      val source = Source.fromInputStream(input).recover { if (it is IOException) { return SuccessOrFailure.failure(it)  } else { throw it }  }.getOrThrow()
      source.getLines()
  }
  .recover { if (it is InputStreamException) { throw it } else { return SuccessOrFailure.failure(it) }  }
}

fun getURLContent(url: String): SuccessOrFailure<List<String>> {
    try {
      val url = parseURL(url).getOrThrow()
      val connection = url.openConnection().getOrThrow()
      val input = connection.getInputStream().getOrThrow()
      val source = Source.fromInputStream(input).getOrThrow()
      return  SuccessOrFailure.success(source.getLines())
    } catch (e: ConnectionException) { 
        return SuccessOrFailure.failure(e)
    } catch (e: ParseException) { 
        return SuccessOrFailure.failure(e)
    } catch (e: InputStreamException) {
       throw e // so it doesn't get caught by IOException
    } catch (e: IOException) { 
       return SuccessOrFailure.failure(e)
    }
}

None of the snippets is an improvement over the current situation of try/catching, and the API is ripe for abuse as it doesn't provide a clear, unique way of chaining SuccessOrFailure. The non-local returns required for coroutine rewrites also enable more possibilities for breaking control flow, and will create a copy of frequently called APIs on every callsite.

Every company, library, and employee will have their own preferred style, which means every time you come to a file using SuccessOrFailure it will look different and have different control flow that's not the one JetBrains had in mind for this proposal. You're trading learning a concept once, for all libraries, with having to read an implementation to understand its behavior every time.

If you let the exception propagate then your API is lying to you, it's not a SuccessOrFailure<List<T>>, it's a hidden exception instead like most Java APIs do. It's not an improvement, it's a step back with a new broken way of writing APIs.

You have the body of work that Arrow has done around this already. The flatMap way of chaining constructs is principled and shared under other names like bind, andThen and selectMany across JavaScript, C#, Java, TypeScript, Rust, Scala, Clojure, Haskell, F#, OCaml, ReasonML, a handful of libraries for each, and even other parts of Kotlin. It's well understood that flatMap chaining is not equivalent to exception propagation even with checked exceptions, nor the two snippets provided are the same and the alternatives described above are even more verbose than regular exceptions.

We have the implementation, the test suite, the documentation, entire communities of untyped languages that understand the concept and enough examples across all of them. I don't understand the pushback.

[raulraja]

https://github.com/Kotlin/KEEP/blob/success-or-failure/proposals/stdlib/success-or-failure.md#appendix-why-flatmap-is-missing just want to note that this section is not accurate. Direct style is not equivalent to monadic comprehensions unless in the Id Monad. All other monads take care of their effects including those capable of failure without blowing the stack throwing exceptions. From all the things Kotlin has gotten from Scala monad comprehensions is the big one missing. The entire Scala community even those uninterested by FP like for comprehensions. I think a lot of people in the Kotlin community would benefit from safer error handling and generalized syntax for types that structurally include a way to flatMap like List, Deferred, Observable,...

[pakoito]

Someone has pointed me that if you look closely,
parseURL(url).mapCatching { url -> url.openConnection().getOrThrow() }
is equivalent to
parseURL(url).flatMap { url -> url.openConnection() }.

Given that Raul and I have been the first ones to reply, it's worth mentioning that we don't have any agenda with this topic other than trying to help with the API based of experience with similar constructs, and what's happened in the past in other communities. For example, the JS community ignored a similar API design correction citing "aesthetics" in Promises/A+ that years later they're still paying for, and ended up spinning up the whole fantasyland ecosystem and a set of alternative Promise libraries and extensions trying to fix this mistake.

If we're not being helpful or insightful, we'll just add SuccessOrFailure#flatMap as an extension function in a single function library and everyone will get the API what they want :D

[elizarov]

@pakoito Let me try to answer your concerns

What happens when parseURL(url), openConnection(), getInputStream() return SuccessOrFailure<T>?

We explicitly discourage writing functions that return SuccessOrFailure<T>. Don't fall in the trap of thinking that we are proposing to add Arrow's Try class in disguise into the Kotlin Standard library.

That is why we don't have to answer this "what happens" question nor do we have to write any of the verbose alternatives you are suggesting. Moreover, we explicitly discourage using SuccessOrFailure to catch some exceptions, while letting others fall through. SuccessOrFailure is explicitly designed to encapsulate any failure in the function execution.

We do encourage using domain-specific classes in case where you need more fine-grained control over failures that need to be caught and the ones that need to go through. We are not planning to eat Arrow's bread here and that is one the reasons we gave such a long SuccessOrFailure name to the stdlib primitive. We purposefully avoided taking over names that functional libraries are using. You are welcome to continue Arrow's primitives without confusing anyone who is reading the code on the intended semantics of those primitives.

I will expand the corresponding section in the KEEP to better explain the similarities and differences between monadic approach and the use-cases that SuccessOrFailure is designed to address.

[elizarov]

@raulraja The KEEP explicitly lists it pretty narrow goals. Introducing alternative to exceptions for error-handling is out-of-scope of this KEEP. This KEEP is narrowly focused on the problem of dealing with exceptions in concurrent and functional code.

[elizarov]

Let me give the summary of various way to represent errors in Kotlin code:

• Nullable types -- have very good support in stdlib and in the language.
• Exceptions -- have some support in the language and this KEEP is improving support in stdlib.
• Sealed classes -- have good support in the language, but it is quite verbose to compose domain-specific sealed classes. Might be an area of further improvement in the language and/or in stdlib (out of the scope of this KEEP).
• Error classes in other libraries (like Arrow's Try) -- have very good support in the corresponding libraries, it is an open question if and how additional language support is needed (out of the scope of this KEEP).

[elizarov]

@pakoito

Someone has pointed me that if you look closely,
parseURL(url).mapCatching { url -> url.openConnection().getOrThrow() }
is equivalent to
parseURL(url).flatMap { url -> url.openConnection() }.

That is correct. One can easily write flatMap as an extension, but since we discourage using SuccessOrFailure as return type of functions, there is no point in having it in the standard library as its use would contradict the design goals behind SuccessOrFailure.

[elizarov]

@pakoito

We have the implementation, the test suite, the documentation, entire communities of untyped languages that understand the concept and enough examples across all of them. I don't understand the pushback.

That is great. There is no pushback. I, personally, really like the way Arrow community is working and the resulting functional libraries and I would like to see it thrive and expand. It works really well as a separate library that people can use if they develop code in mostly functional style and I see no point in stealing any of the Arrow's functionality into the Kotlin Standard Library. Arrow is already quite popular and works really well the way it is.

From my side, I will try to give a better explanation on differences between Arrow's Try and the SuccessOrFailure that this KEEP is proposing and to focus more attention on the additional benefits that Try is bringing onto the table to encourage people who are interested in those to use Arrow library. I did not intent this appendix to be perceived as a pushback on Arrow. I'll need to do a better writing job there.

[ktkit]

Thank you, Roman.
It's great this type has reached the stage of KEEP.
Yet still, I strongly believe its value is underestimated.

Let's consider a simple scenario.
We've designed the following function that throws an exception, having in mind the user code just propagates it on the higher level. So we document its dangerous behavior and voila:

/** Returns [URL].
@throws SomeException... */
fun parseURL(url: String): URL {
  // parses a string and may throw an exception in case of a malformed url
  return theUrl
}

Then a user calls it in some place where error propagation is expected:

fun getURLContent(url: String): List<String> {
  val url = parseURL(url) // it throws but we expect it
  // ... process content
  return content
}

But some time later the user sees this usage of parseURL function which obviously returns URL, and copy-pastes it to that function:

fun showURLContent(url: String) {
  val url = parseURL(url)  // oops .. Application is terminated due to an unexpected error
  // ... process content
  textField.text = content 
}

And oops. It's slipped from UnitTests and real users experience the bug :(

Such scenarios happen all around because the relation between method signature, its returned type and the fact that it's dangerous, is held only on short-term memory of the developer who uses it. No other hints available.

Kotlin removed an explicit Java-like method annotation throws (thank god!) but provides nothing safe in exchange (Null-safety is cool! But it's about references, not a behavior of functions).
And so we have the same "billion-dollar mistake" problem that we had in Java with null references. But now in Kotlin, and on the higher level of functions. Figuratively speaking it's the same as holding dynamite in candy wrap. Looks sweet, touch it, and you'll find yourself in chocolate up to the ears :)

And lo! The SuccessOrFailure type which literally introduces the new paradigm of exception handling:
Any function that may throw an expected exception returns SuccessOrFailure object.

fun parseURL(url: String): SuccessOrFailure<URL> {
  // parses string and may throw an exception in case of a malformed url
  return successOrFailureUrl
}

And since then nothing wrong can happen.

fun showURLContent(url: String) {
  val url: URL = parseURL(url).onFailure{return}.getOrThrow() // continue safely with a URL object
  // ... process content
  textField.text = content
}

And if one needs an error propagation behavior, the following explicit alternative to the parseURL can be defined:

fun parseURLThrowing(url: String): URL {
  // parses string and may throw an exception in case of a malformed url
  return theUrl
}

Or the safe version can be used this way:

fun getURLContentThrowing(url: String): List<String> {
  val url = parseURL(url).getOrThrow()
  // ... process content
  return content
}

And since the implementation of the SuccessOrFailure class is highly optimized, it costs nothing to box-unbox this object the way above.

I expect objections from developers who got used to the "unsafe" style and/or has a bunch of code written that way. So yes, it's a tough decision to make. But I believe it's worth it.
Anyway, it would be great to have this type in Standard Lib. But you sell it for 1% of its real value.

P.S. In case of reconsidering the role of the SuccessOrFailure class, the argument that this class "can abuse as a return type of a function" becomes not effective. And the name "Result" and other shorter names can still be under discussion.

P.P.S: The unsafe method signatures is the reason of the discussions about deprecation of Java Optional.get()

Java 8 was a huge improvement to the platform, but one of the few mistakes we made was the naming of Optional.get(), because the name just invites people to call it without calling isPresent(), undermining the whole point of using Optional in the first place. (If this was the worst mistake we made in such a big release, then we did pretty well.)

(I'm Markus M from this discussion)

The bottom line is: This type is so elegant exception handling solution that it by design encourages developers to use it as a return type from functions that throw expected exceptions.

[pakoito]

We explicitly discourage writing functions that return SuccessOrFailure

So if I understand this correctly this KEEP exists almost exclusively to address error handling in coroutines, as its current design requires try/catching and it doesn't compose, which is why we ended up extending it on Arrow. Why does SuccessOrFailure have to be added into the language instead of kotlinx.coroutines directly then?

My concern with adding it to the language is that even if you put documentation, give a stern talk at kotlinconf, put a linter in IJ, a compiler warning and post about it on several blogs, I suspect many people will ignore it and still use them as function returns because that's the perceived reason a Result-like type has to exist. Your examples already include Deferred<SuccessOrFailure<T>> and List<SuccessOrFailure<Data>> as function returns that escape the scope of coroutines.

The other concern I have is that the API is too barebones just for aesthetical purposes. We know from user data gathered over the past year that most users come ask about a function sequence() that goes from List<SuccessOrFailure<Data>> to SuccessOrFailure<List<Data>>, a shape that's easier to work with in their heads. Same as flatMap they'll reimplement it in non-canonical, untested ways, or will get it from a library. People who are not willing to learn map, flatMap, fold and sequence already use when and for loops to recreate the same functionality; which doesn't mean that you can still provide the canonical functions for everyone else. There are no downsides, just aesthetical reasons, and the community will not necessarily use SuccessOrFailure the same way this KEEP intends because the lack of helpers forces them to look for their own solutions.

As @ktkit said if this is a construct added for the language, might as well go all-in the first time around and copy Rust's Result or any other proven body of work you prefer. At Arrow we're eager to deprecate Try if it's added to the language, proper. If that is not the intention of the KEEP, a couple of extension functions may suffice instead.

[elizarov]

@pakoito The primary use-case for this SuccessOrFailure class, as explicitly spelled out in the corresponding KEEP is indeed error handling in coroutines. We have to put it into the Kotlin Standard library because Continuation interface from the Kotlin Standard Library depends on it. We cannot put it into kotlinx.coroutines, but we can put it into kotlin.coroutines. However, since we've identified a number of use-cases beyond coroutines (and they are all listed in the KEEP) we decided to put into kotlin package for now.

We very much share your concern that people will use it as a return type and we'll try our best to discourage people from doing so.... we cannot entirely prevent bad code, though.

Giving additional extensions for things like List<SuccessOrFailure<T>> was raised as an open issue. I'll add it as an open issue to the KEEP, but we'll need to make sure that we are not encouraging abuse of SuccessOrFailure when doing so. In this respect, barebones API is done on purpose. Again, we don't want to eat Arrow's bread. This API is designed for very narrow use cases, not as something you'll use all over the place in your application.

We could add something broader into the language in the future, but that is definitely out of the scope of this KEEP. Don't deprecate your Try please. We are not in a position to replace it yet. Replacing all the exception-handling patterns you can do with Try and putting them into the language in some shape is extremely complicated endeavor. We have not even narrowed down any potential direction we might take on this road (and there are many), yet alone figured out appropriate design.

[raulraja]

The question for me is why isn't this for a more generic Either in the std lib and let users decide how they want to use it. I'm in favor of removing types from Arrow that surface in the stdlib in some kind of compatible shape with the functional combinators. If we had Either in the std lib you could have in Deferred the operations you need to handle errors even when ops go async without blowing the stack:

fun <A> Deferred<A>.attempt(): Deferred<Either<Throwable, A>>
fun <A> List<Deferred<A>>.sequence(): Deferred<List<A>> // shortcircuits on first error 

val op1: Deferred<Int> = async { 1 }
val op2: Deferred<Int> = async { throw RuntimeException("can't fetch 2") }
val op3: Deferred<Int> = async { 3 }

val result: Either<Throwable, List<Int>> = 
  listOf(op1, op2, op3).sequence().attempt().await()

result.fold(::showError, ::doWithSuccess)

I think the issue with error handling in async stuff is that we instead encourage people to do things that break composition when exceptions are thrown, at least a lot of the code I see with coroutines looks like calling await anywhere is fine forcing then again to try/catch or it blows with an exception.

When you say:
| We explicitly discourage writing functions that return SuccessOrFailure
I think that leads to people having to handle errors, async/sync etc explicitly on each layer or resort to throwing exceptions and back to the original style of try/catch.

val result: List<Int> = try {
   listOf(op1,await(), op2.await(), op3.await())
} catch (e: Throwable) {
  emptyList<Int>()
}

I think an Either type for the standard library that allows others like Coroutines solve their own problems would be more appropriate specially when there is already a proliferation of Result types in many libraries and a bunch of custom ones that people ad-hoc put in their code bases and none of them solve the async biz.

[fvasco]

A personal consideration about @ktkit Optional mention.

Optional does not fix the billion-dollar mistake, it is only a verbose workaround.
Every Java method can return null instead of Optional, so the mistake remains unchanged.
To fix this issue is required a language change.

Likewise for SuccessOrFailure, there is no way to guarantee a result instead of an exception.

In the previous example:

fun parseURL(url: String): SuccessOrFailure<URL> {
  // parses string and may throw an exception in case of a malformed url
  TODO() // return successOrFailureUrl
}

So SuccessOrFailure as return type is trustworthy like Optional in Java.

[elizarov]

@ktkit Thanks for taking time to explain the dangers of exceptions. Let's dissect your example a little further:

/** Returns [URL].
@throws SomeException... */
fun parseURL(url: String): URL

We really encourage Kotlin programmers NOT to declare such functions because of all the problems with exceptions you've outlined. If you have signalling exception that is supposed to be caught in your code and handled locally you should not have used exception for that in the first place. An idiomatically designed Kotlin API for this case should be defined like this:

/** Returns [URL] or `null` if it cannot be parsed. */
fun parseUrlOrNull(url: String): URL?

With this signature you get all the nice typesafety and extensive support from Kotlin compiler and standard library, which allows you to write succinct, yet correct and safe code like this:

fun showURLContent(url: String) {
  val url: URL = parseUrlOrNull(url) ?: return // continue safely with a URL object
  // ... process content
  textField.text = content
}

It is out of the scope of this KEEP to design similar mechanics for wider classes of failure that a class like SuccessOrFailure might represent, that is why we discourage writing functions that return SuccessOrFailure object. Please, use nullable type in the code like that.

[elizarov]

@raulraja We are painfully aware of exception handling problems with concurrent code and how it is very hard to write correct code using async (without running into very subtle problems). We are trying to address it in kotlinx.coroutines via a more structured concurrency approach, see Kotlin/kotlinx.coroutines#410 for details.

[elizarov]

UPDATES: @pakoito @raulraja @ktkit

1. I've expanded Appendix with a more fleshed out comparison of SuccessOrFailure and Try to explicitly address the differences in how they work and how they are used.

2. I've added Additional APIs for collections into the list of open questions.

3. I've worked out future direction for Integration into the language to show a potential path on how one day we might realize a Kotlin language mechanism of finer-grained control of exceptions that could replace Java checked exceptions with a safer and more explicit approach in Kotlin style.

[raulraja]

I reread the Appendix and these are not equivalent:

def getURLContent(url: String): Try[Iterator[String]] =
  for {
    url <- parseURL(url) // here parseURL returns Try[URL], encapsulates failure
    connection <- Try(url.openConnection())
    input <- Try(connection.getInputStream)
    source = Source.fromInputStream(input)
  } yield source.getLines()

fun getURLContent(url: String): List<String> {
    val url = parseURL(url) // here parseURL returns URL, throws on failure
    val connection = url.openConnection()
    val input = connection.getInputStream()
    val source = Source.fromInputStream(input)
    return source.getLines()
}

The first returns Failure(e) whereas the second just throws and blows the stack. The whole point of the result types is that they are an Algebraic Datatype with multiple potential states. In this case Success(lines) and Failure(e) they are not equivalent and flatMap which is what for comprehensions desugar to is always biased toward success. The user is not forced to handle the exception but her function does not throw an exception either. Case 1 is exception free.

I also find the encouragement of losing information when handling errors bad practice suggesting that fun parseUrl(url: String): URL? is better. That type signature causes loss of information and models error handling where the type of error should be known or at least a Throwable with a type that it's used to model the absence of a value. Instead we are encouraging to disregard exceptions and errors and model them as absence. This style of narrowing to ? forces users to always be explicit about error handling in their function bodies to go from E -> A? and that is not necessary when the data type already provides an error context like for example Deferred, Observable, Either, Try, etc. These are all types which have an error context and don't need to have users explicitly try catch around them ever. There is a way to compute over these types in a sane way which is via comprehensions that delegate to flatMap. I understand that is not the issue this KEEP addresses but really comprehensions allow for easy composition and imperative style and it works in the same way in sync and async code.

Here is the example above with a pseudo keyword bind that I'm making up to tell the compiler to suspend and use a coroutine context to invoke flatMap behind the scenes until it completes binding it's result to the left hand side. (alternatively the compiler could just desugar the tree to a flatMap chain)

fun doUrlStuffSync: Try<Array<String>> {
  bind url = parseURL(url) //returns Try<Url>
  bind connection = Try { url.openConnection() }
  bind input = Try { connection.getInputStream }
  val source = Source.fromInputStream(input)
  return source.getLines()
}

Now here is the difference with deferred:

fun doUrlStuffAsync: Deferred<Array<String>> {
  bind url = parseURL(url) //returns Deferred<Url>
  bind connection = async { url.openConnection() }
  bind input = async { connection.getInputStream }
  val source = Source.fromInputStream(input)
  return source.getLines()
}

Same syntax if we have A?#flatMap for values returning ? and same syntax for List since there is List#flatMap as well.

Same syntax for all types. Folks don't need to be explicit about exception handling everywhere and types are still forcing them to deal with the exceptional case at some point without resorting to try/catch also we are not encouraging flatMap chains but imperative style top to bottom.

The problems of Either, Try, Deferred etc in terms of composition and sequential computation are the same. All these types compose with flatMap, short-circuit in their first error and carry internally an error context that flatMap ignores. If we end up adding special syntax for all of them as suggested in:

|User throws NotFoundException, MalformedNameException

we are gonna end up with special cases for each type and it's going to be a mess composing values like Deferred<Either<MalformedNameException, User>> where you have to resort to a bunch of different machinery to reduce those boxed values to the values they contain within.

[pakoito]

Re. the integration into the language section, you already have a primitive with its own syntax that handles errors with propagation: coroutines and the Deferred return type. If you manage to make them cheaper, or even better, depend on the implementation of an operator similar to flatMap like @raulraja described above you won't need this new syntax.

If we continue through this path unpacking a Deferred<SuccessOrFailure<NewErrorType<T?>>> is going to require lots of special syntax baked into the language that'll be hard on newcomers. This is a common issue in monad transformer stacks, where you can end up with dumpsterfires like ReaderT<ForIO, Environment, EitherT<ForIO, Option<T?>>> that without implicit arguments or coherent typeclasses are quite cumbersome.

With the flatMap operator that burden is shifted to library and language implementors instead, as it unifies the behavior. The code will still use the "fast path" version of the coroutines machinery behind the scenes, with the same syntax.

[Wasabi375]

I don't see why you would get a Deferred<SuccessOrFailure<NewErrorType<T?>>> anywhere. The example with deferred does not force this on you.

val outcomes1: List<T> = deferreds.map { it.await() }
// ...
val outcomes3: List<SuccessOrFailure<T>> = deferreds.map { it.awaitCatching() } 

You can still use await instead of awaitCatching. As SuccessOrFailure should not be used as a return type I don't see why you would ever get a SuccessOrFailure<NewErrorType>. The usage of SuccessOrFailure is totally optional, you as the user only use runCatching at the place where you actually want to handle the error so you don't need to worry about stacking error types as this one only gets used temporarily to handle the error.
And as state in the proposal, if you need complex error handling you should look to different options anyways

If your code needs fine-grained exception handling policy, we'd recommend to design your code in such a way, that exceptions are not used at all for any kinds of locally-handled failures (see section on style for example code with nullable types and sealed data classes). In the context of this appendix, parseURL could return a nullable result (of type URL?) to indicate parsing failure or return its own purpose-designed sealed class that would provide all the additional details about failure (like the exact failure position in input string) if that is needed for some business function (like setting cursor to the place of failure in the user interface). In cases when you need to distinguish between different kinds of failures and these approaches do not work for you, you are welcome to write your own utility libraries or use libraries like Arrow that provide the corresponding utilities.

[Wasabi375]

@elizarov btw, the awaitCatching functions are not in the API details

[elizarov]

@raulraja I'll try to answer some of your concerns:

I reread the Appendix and these are not equivalent: ... The first returns Failure(e) whereas the second just throws and blows the stack.

It looks like the following sentence from Appendix answers it:

If callers of this function need an encapsulated failure, they can always use runCatching { getURLContent(url) } expression.

With respect to this:

I also find the encouragement of losing information when handling errors bad practice suggesting that fun parseUrl(url: String): URL? is better.

I'll quote this part:

If your code needs fine-grained exception handling policy, we'd recommend to design your code in such a way, that exceptions are not used at all.

The key part here is "if your code needs...". Not all cases are the same. Sometimes we need to "blow up the stack", because we don't plan any local, business-specific handling, and that is where exceptions come in handy, since they contain all the vital debugging information (stack trace and message). But none of that debugging information is useful for business logic. If you are planning to handle URL parsing failure in the business part of your code (what I call "locally"), then having exception at hand does not give you any additional value. Quite contrary, you loose tons of performance constructing this exception only to get a single bit of useful information out of it -- whether parseURL had failed or not. Nullable types are much better here.

Same syntax for all types ...

You are suggesting to unify all the error-containg types, but we are trying to do exactly the opposite. We are trying to discourage usage of exception-containing types in the code. We want to have a world where Kotlin code has as few uses of SuccessOrFailure and Deferred types as possible. These types are noise. Programmers should be focused on the business logic of their code, while error-handling and concurrency aspects of their code should be optional layers -- they should not obscure or encumber the main logic of the program, they should not force the programmers to write the main logic of their code in any different way, but we also need to make sure that the shortest code you write is still safe, correct, and does not loose any resources or exceptions (we are still far from this bold goal, but we are trying to get there).

You can see it in the design of all the Kotlin things. That's why, for example, we have suspend fun(...): T concept instead of fun (...): Deferred<T> as many other languages chose to do, because we believe that programmer should not be forced to change the way they write their business logic because they now have some asynchronous operations in their code.

UPDATE: ^^^ this also answers @pakoito concern. In short, we don't want to encourage things like fun foo(): Deferred<SuccessOrFailure<NewErrorType<T?>>> in Kotlin code (though we cannot prevent people from writing them, of course). It should be suspend fun foo(): T? and nothing more.

[elizarov]

@Wasabi375 awaitCatching is an example. It cannot be part of the Kotlin Standard Library, because awaitCatching is an example of extension you could define on Deferred class which is defined in kotlinx.coroutines library.

[pakoito]

Okay, I understand the intents and purposes of this KEEP and my concerns have been heard. I'll help spread the word not to use SuccessOrFailure as a function return, add some helpful extfuns to arrow-core and promote Try/Either as the choice for public APIs instead :D If there's something else I can do to help just tell me!

[ktkit]

@elizarov Thanks Roman, I like your idea of dissection. Let's just take the whole pie, i.e. the handling of functions that are supposed to fail under specified conditions in which case they should signal it somehow to the caller (in short, fallible functions), and review this subject in evolutionary perspective.

Pre-SuccessOrFailure era:
Available scenarios:

1. A failure is univocal or extra info isn't relevant. Example: Division by zero. Solutions:
   a. If the returning type is non-nullable: return nullable type T?, where null indicates a failure.
   b. Otherwise return Optional, where empty indicates a failure.
2. Info about a failure is/may be relevant. Example: parseURL which failure may indicate what had gone wrong. Solutions:
   a. Throw an exception that carries relevant info, hoping it's caught somewhere in the stack.
   b. Devise a (or use a 3-rd party) custom class or hierarchy of classes to carry this info.

Notes:
1a is the case you've mentioned in your response.
2a is the case, you agreed, is "dangerous". However, most of the developers did not append the name of such functions with "OrThrow" (or alike) since there wasn't "non-dangerous" alternative.
2b is seen as an ancestor of this KEEP subject.

Post-SuccessOrFailure era:
Available scenarios:

1. A failure is univocal or extra info isn't relevant. Solutions:
   a. If the returning type is non-nullable: return nullable type T?, where null indicates a failure.
   b. Otherwise return SuccessOrFailure instead of Optional.
2. Info about a failure is/may be relevant. Solutions:
   a. Return SuccessOrFailure since it's the efficient functional value-exception container.

Notes:
1a is the only case left where handling of a failure is still more efficient than the SuccessOrFailure way, since specific info about a failure isn't required.
The throwing version of a fallible function that relies on catching can still exist but with some intelligible name extension like "OrThrow".

For all other cases, the SuccessOrFailure type represents an evolutionary leap to safety and simplicity.

Please, correct me if I'm wrong.

[elizarov]

@ktkit There is nothing efficient about SuccessOrFailure in case of a failure. On a failure it contains an exception that is extremely expensive to create. It should not be used for case where you expect to have a failure condition that would be handled by some business logic. Exceptions should be only used for truly exceptional cases of the kind that crash your application and are usually handled somewhere on the top level, that is why they are designed to propagate automatically and to carry all the metadata for debugging.

For cases where you expect some kind of potential error in function execution you should use either nullable types or domain-specific sealed class hierarchies. If you write all your code in a monadic style, then you should use the corresponding monadic libraries (like Arrow) which integrate the corresponding monads.

[raulraja]

@elizarov Would mapCatching and other similar methods catch all Throwables or would it perform some kind of discrimination like Scala does with NonFatal https://www.scala-lang.org/api/current/scala/util/control/NonFatal$.html ?

[ktkit]

@elizarov I admit that the fact that the default Throwable.fillInStackTrace method is expensive slipped out of my attention. Now i see, you've got a fair reason to discourage from using this type in cases when failures in a fallible function happen quite often and thus induce pressure on CPU.
And this fact, I believe, should be explained in the API docs as the reason why you discourage from abusing this type since it isn't that obvious.

However, a couple of optimizations can eliminate this adverse cost effect.

Let's review scenarios of handling a fallible function in a new perspective:

1. A failure is univocal or extra info isn't relevant:
   a. If the returning type is non-nullable
   b. If the returning type is nullable
2. Info about a failure is/may be relevant:
   a. The exception stack trace is relevant
   b. Local info about the failure is enough

Solutions:

• 1a: return nullable type T?, where null indicates a failure. (the same)
• 1b: return a predefined failure with an exception, which can be defined in a custom fashion or in the SuccessOrFailure companion object as private val unitException = SomeStacklessException(); val failure = SuccessOrFailure<Unit>.failure(unitException).
• 2a: as intended, return SuccessOrFailure object with an actual exception. Note: in case the exception propagation required, calling result.getOrThrow() doesn't do an extra work compared to throwing method alternative, since an exception is created and thrown once.
• 2b: as an alternative to defining a sealed hierarchy of failure classes we may consider using a stackless exception, i.e. an exception with overridden fillInStackTrace method that does nothing. Such exception has about the same creation cost as any other object.

So maybe instead of or in complement to discouraging developers from abusing this type, informing them about alternatives is not a bad idea?

[elizarov]

@raulraja We don't have anything similar to NonFatal in Kotlin at the moment and designing something like that is complicated endeavor that is outside of the scope of this KEEP, so all xxxCatching functions just catch all Throwable exceptions. That is just another reason why SuccessOrFailure should not be used outside of the narrow use-cases it is designed for. My own experience in enterprise shows that distinction between "fatal" and "non-fatal" is extremely domain-specific and business-specific. Hardcoding it into the language (standard library) would definitely work against the "general purpose" nature of Kotlin. Having it defined in 3rd party libraries is perfectly fine, though, since those are domain-specific by definition.

I've prototyped a version of SuccessOrFailure that would let you a specify a (subtype) of exceptions that are caught, but it does not look very well in Kotlin's generics at this moment (there is no ability to specify default generic type, nor a way for partial type specification). We have to defer it to potential future enhancements. I would rather fold SuccessOrFailure into the language in the future (see updated section on this possibility) as a better replacement of Java's checked exceptions that works well with higher order function (unlike Java's checked exceptions), since it is actually expressed and implemented as a return type of a function. That would be a major enhancement in the language, though.

[elizarov]

@pakoito suspend fun main is planned https://youtrack.jetbrains.com/issue/KT-17679

[Redrield]

When I originally voted for KT-18608, the proposal on YouTrack for a Result type before this KEEP was written, I had hoped that it could be something much more similar to Rust's Result type. What I feel is proposed by this KEEP needlessly limits the scope of what could otherwise be a very powerful tool for developers.

Take this snippet of Rust code

type Result<T> = std::result::Result<T, failure::Error> // Make all of the results parameterized by E=failure::Error

fn could_fail() -> Result<String> {
    foo()?.bar()?.baz() // In this context, ?. is binding Results. (Scala monad comprehension)
}

fn foo() -> Result<Bar>

struct Bar;
impl Bar {
    fn baz() -> Result<Baz>
}

As can be seen in could_fail, an operator is used in the place of try {...}catch(_) { return SuccessOrFailure.failure(..) } in function calls. That pattern could benefit from being a part of Kotlin for error handling the same way that ?. for null safety benefits Kotlin massively. That's something that functional companions like Arrow can't do due to their nature as third party libraries.

This proposal feels like a sad waste of what could otherwise be a very useful part of the standard library.

[elizarov]

@Redrield I personally love design of Result type in Rust and its integration into the Rust language. However, adding Rust-likeResult design into Kotlin would require vast changes to the Kotlin language itself with highly non-trivial consequences to Kotlin-Java interoperability (and interoperability is a very important goal at this stage of Kotlin language evolution).

It is definitely out of the scope of this KEEP (which is focused only on the standard library), but it is still on the table for future releases. One possible direction is covered in this section.

However, in order to move this work on potential future languages changes forward we first and foremost identify use-cases (with code examples) that demonstrate weak points of Kotlin's error-handling design that Rust-like Result could address. For example, this particular KEEP has quite a narrow list of use-cases that are explicitly spelled out in the text and are supported by the proposed standard library class. To do something beyond that, we'd need use-cases beyond that.

We also need a clear story of how Kotlin developers are supposed to use the feature that languages is providing them with. As of today, your Rust example in Kotlin should be, most likely, rewritten in the following idiomatic Kotlin way:

fun couldFail(): String = // means Result<String,Throwable> in Kotlin
    foo().baz() // Here Kotlin does comprehension over Try monad

fun foo(): Bar // means Result<Bar,Throwable> in Kotlin

class Bar {
    fun baz(): String // means Result<String,Throwable> in Kotlin
}

Because the original Rust code used Result<T, failure::Error> I assume that callers of this function are not supposed to actually care about what kind of failure::Error the function had failed with, but, most likely, just propagate it to the caller with ? operator to log the error for developer's trouble-shooting at the top level, so that is why, based on this assumption, I'm rewriting it with equivalent Kotlin mechanism of exceptions.

[ktkit]

@elizarov I still can't see why you suggest this fallible function style:

fun couldFail(): String // can throw an exception
// Or
fun String.toInt(): Int // can throw an exception

For all functions of that style that already exist (what you call JVM ecosystem) just let them be. But with the SuccessOrFailure at disposal, it's much safer to write:

fun couldFail(): SuccessOrFailure<String> // can throw an exception
// Or
fun String.toInt(): SuccessOrFailure<Int> // can throw an exception

And handle them whichever way you prefer:

fun rethrow(): String = couldFail().getOrThrow() // re-throws an exception
// Or
fun handle(v: String): Int = v.toInt().getOrElse(-1) // handle an exception locally

As for performance, there is no extra costs in both success and failed outcomes. In a success case the result won't be even boxed in a SuccessOrFailure since it's inlined (i might be wrong here, but not in the gist), and in case of a failure, the hard work has already been done anyway in the callee by creating the exception.

So actually this warning from the KEEP:
This SuccessOrFailure class SHOULD NOT be used directly as a return type of Kotlin functions with a few exceptions ...
could be narrowed down to:
This SuccessOrFailure class SHOULD NOT be used directly as a return type only in case the failure is handled by the caller locally. Where nullable type and sealed class hierarchy could be used instead.

And if the user is aware of stackless exceptions, those can be used with SuccessOrFailure without extra costs, thus eliminating even the sealed-class hierarchy case.

I'm sorry, I know I can be a pain in the chair :)

__
By the way I've just noticed the SuccessOrFailure core class API lacks a useful, to my mind, function:

fun getOrElse(altValue: T): T

It lets write:

getInt().getOrElse(-1)
// instead of 
getInt().getOrElse { -1 }

[pakoito]

Those two are ambiguous because the second getOrElse could be returning a function. We desambiguated them as getOrElse and getOrDefault IIRC.

[ktkit]

@pakoito As I can see from REPLing the following sample, those functions are distinct, i.e. treated as overloaded.
And as I can see from the current KEEP API, there's no getOrDefault.

class A<T>(val value: T) {
  fun getOrElse(v: T): T = v
  fun getOrElse(code: () -> T): T = code()
}

fun use() {
  val a = A(1)
  val x: Int = a.getOrElse(2)
  println("x=$x") // 2
  val y: Int = a.getOrElse { 3 }
  println("y=$y") // 3
}

[pakoito]

In val y = a.getOrElse({ 3 }), y is 3 rather than a function () -> Int as intended. Hinting doesn't fix it.

[ktkit]

@pakoito I do not quite understand what you're trying to get.
val y: Int = a.getOrElse({ 3 }) invokes fun getOrElse(code: () -> T): T, where y is of type Int, and the argument is the function { 3 }.
What am I missing?

[pakoito]

What I want is val y: () -> Int = a.getOrElse({ 3 }), because a holds a function of type () -> Int.

[ktkit]

@pakoito Charming! Thanks for the clue. When type T is a function, those overloads really become ambiguous. Although, I can't grasp the reason why. Looks like a language limitation or bug.

Anyway, the current SuccessOrFailure API lacks getOrDefault.

[elizarov]

@ktkit getOrDefault is indeed missing. We'll consider adding it: https://youtrack.jetbrains.com/issue/KT-25659

W.r.t. the style of fallible functions. If you are planning to handle error locally, then you should not use exceptions (nor SuccessOrFailure, for that matter). Kotlin provides two idiomatic mechanisms for locally handled errors -- nullable types and sealed classes. I am not suggesting you should write this if you plan to handle error locally:

fun String.toInt(): Int // can throw an exception, use it for globally-handled errors

I'm suggesting that you write this:

fun String.toIntOrNull(): Int? // can return null, use it for locally handled errors

Bottom-line: Don't use neither exceptions nor SuccessOrFailure for failures that should be handled locally. Exceptions are designed to be handled globally (at the top level) and SuccessOrFailure is designed to be simply a wrapper on top of them for cases where you need to work with multiple pending (non handled yet) exceptions at the same time. If you have just one exception at hand, then it should be always thrown.

P.S. If you write all your code in monadic style, that is a different case. Use an appropriate library (like Arrow) that has an integrated set of monads for all your needs.

[ktkit]

@elizarov Yes, I agree, and I got it. What I'm trying to highlight is the gist of the warning about the usage of SuccessOrFailure.
One thing to say: it SHOULD NOT be used ... with a few exceptions.
And another thing to say: it SHOULD NOT be used in two cases ... or even CAN be used with two exceptions ... (which are the nullable type and the sealed class hierarchy).

When you view this type in the latter perspective, it doesn't look that unattractive. And it better corresponds its nature.

And by the way, Scala's Try type has the same usage issue, while the documentation doesn't say a word about it. So, even mentioning those two limitations is a step forward.

[qwwdfsad]

From the KEEP standpoint of view, it seems like SuccessOrFailure is a coroutines-specific primitive and any other usages are highly discouraged.

SuccessOrFailure is very similar to Try monad but lacks of monad combinators. And when something looks like a Try monad, it eventually will be [ab]used and sneaks into popular libraries, business logic etc. The fact that "real" Try monad is present in Arrow won't help, because it's not as common as stdlib and is not easily discoverable.

We already have Java 8 Optional experience: even with a tremendous educational effort we're still seeing stuff like equals written with optionals for "null comparison" or functional null-checks like Optional.ofNullable(value).ifPresent(value -> { log(value) }); (these are examples from popular github projects btw). I don't see how SuccessOrFailure may be different from the Optional pitfalls in its current state.

Unpopular opinion:
If we don't want users to accidentally misuse SuccessOrFailure, we should move it to coroutines package and make all useful extensions internal, making it unusable standalone.

[Wasabi375]

@qwwdfsad I agree with you. I think experience shows that most people won't look at a documentation before they are forced to by an error and even then. The problem is though that making SuccessOrFailure internal wont work as the whole idea is that the users interact with SuccessOrFailures returned by runCatching.

IMO the most useful feature SuccessOrFailure would provide is the ability to properly handle errors in parallel executing code.

val outcomes: List<SuccessOrFailure<T>> = deferreds.map { it.awaitCatching() }

I don't see how this is possible for the user when SuccessOrFailure would be internal.
What about making the compiler create a warning whenever a function returns a SuccessOrFailure or expects one as a parameter? In the stdlib we could just suppress the warning.
That way improper use of this will be discouraged without relying on documentation.

[qwwdfsad]

@Wasabi375
I don't propose making SuccessOrFailure internal, only its functional extensions. SuccessOrFailure will still be used in Continuation API, but won't have anything which can assist stdlib users. Continuation is a very low-level API with a few consumers, so it won't be a problem.

In your example, awaitCatching will be a handwritten extension on Deferred (with manual try {} catch {} inside).
But hopefully kotlinx.coroutines will have better builders and mechanisms for parallel decomposition than map { awaitCatching() }

[ktkit]

@qwwdfsad @Wasabi375 You have a point of course. However, real life is a way of compromises. You can't get rid of "rake walkers". Does it mean we should forbid rakes? No one can stop one from doing this:

val nullable: Any? = null
nullable!!.toString() // oops :) 

And it doesn't mean there's something wrong with nullable types. There's myriad of ways how one can screw one's code.

The fact is, there is Try type in Scala stdlib which does the same the SuccessOrFailure does. And they do not sweat it, because it's the safe and elegant way to handle fallible function.
The only issue is the cost of filling the exception stack at the moment of its creation, that is about 10 times slower than the object creation, yet is not that critical (roughly 1000 ns instead of 100 ns). Which brings the only group of cases when usage of the subject has to be limited. And that's the job of the API documentation.

I also believe the subject deserves a more consistent name.

[elizarov]

The Kotlin class to encapsulate success or failure was renamed to Result and additional restrictions on its usages are now enforced by compiler to allow for future enhancement. Details can be found in the new section of the KEEP document: https://github.com/Kotlin/KEEP/blob/master/proposals/stdlib/result.md#limitations

[pakoito]

Those seem like a set of arbitrary restrictions that only make sense from the implementer's perspective. I'm not sure it'll be easy to explain the users why they're in place, specially if they're going to start being used in standard library functions.

Do these restriction exist only to prevent the Railway-Oriented Programming and promote the direct style, or is there any technical reasoning behind them?

[elizarov]

@pakoito If the goal was just to prevent ROP, then we would have made it a warning. We made it a compiler error, because if we allow it in Kotlin 1.3, then this code might break in future versions of Kotlin, because of the future changes to the language we might be introducing around Result type.

P.S. We cannot prevent ROP, anyway. Users are free to use any 3rd-party library with an appropriate result type.

[pakoito]

Okay, so binary compatibility. It makes sense, thanks :)

[Dico200]

Wouldn't it be a good idea to implement sealed inline class into the language before 1.3 release, and change Result implementation, so we don't need to keep the current source and binary interface of Result forever?

[elizarov]

@Dico200 The switch to sealed inline class in the future should not affect existing binary interface.

[throwable-one]

Hello all,

This KEEP is more about failure handling in coroutines and parallel decomposition, right?
In the first paragraph it speaks about exceptions usages for sequential code.

But Kotlin does not have checked exceptions and this KEEP is not easy to use for failure handling becuse Result can't be used as public methods and kotlin does not have any syntax support for it (as it has for nulls for example).

Here is my case: some function returns result or error, so it is literally Success T | Failure F. I want to be sure that user checks for error before using result. In Java I use checked exceptions, so one can't simply ignore error.

In Kotlin exceptions are unchecked, and the only way to implement it is to return null. But with null, I can't return error object.

From my point of view, this is very close to nullability support, but instead of null we must handle error somehow.

For example:

fun getUser(): Result<User, String>() {
return if (is42()) User() else Error("This function is not available on Mondays")
}
///
val user = getUser().onError { error -> User("Unknown") } 
val user = getUser().onError { error ->
 log(error)
 User("Unknown") 
}
val user = getUser().onError { error -> throw BadUserException(error) } 
val user = getUser().onError { error -> 
 showMessage("Can't get user: $error")
  return
} 

Could we have something like that?

[pakoito]

Given that Result was scoped as a coroutines-only API, you have to look elsewhere for now. There are several already several existing implementations of Result in libraries, and writing one is quite simple.

Feel free to rip off either (any error) or try (only Throwable) from arrow:
https://github.com/arrow-kt/arrow/blob/master/modules/core/arrow-core/src/main/kotlin/arrow/core/Try.kt
https://github.com/arrow-kt/arrow/blob/master/modules/core/arrow-core/src/main/kotlin/arrow/core/Either.kt

// shill
Or even better, use arrow-core instead :D
// shill

[throwable-one]

@pakoito thank you.

It seems that I should talk to Roman becuase he wrote that such things "Might be an area of further improvement"

I really want to have it in stdlib

[elizarov]

Closing the issue as it had been implemented.