Home

Scala Expect

Introduction

Scala expect is a Scala implementation of a very small subset of the widely known TCL Expect.

It allows you to automate interactions with programs that expose a text terminal interface (CLI). However unlike TCL expect the scala expect does not use a pseudo terminal to interact with the programs, it simply reads from the standard out and standard error.

Use cases

You should only use scala expect if you cannot find a library that allows you to use the intended program directly. Or in other words you should use it as your last resort.

A good example for using it would be Kerberos. In Java world there are a lot of ways to obtain a Kerberos ticket, however there isn't a single library that allows you to perform management operations on a realm, such as creating and deleting principals. In theses cases you can either use JNI to invoke the Kerberos library directly using C (which is extremely error prone) or you can parse the output from the kadmin program.

A bad example would be using scala expect for obtaining a webpage source (by expecting wget or curl) or using it to interact with PostgreSQL shell in order to populate a database. These are bad examples since there are libraries that can perform these operations directly and much more efficiently.

Disadvantages

Parsing the output from a program is error prone since the format of that output might change in the future. You can still work around this problem by externalizing the expected output, however this will make your program much harder to read.

There is also the problem of dealing with timeouts since you can't know how long the program is going to take to answer your queries.

Anatomy of an Expect

We will use this real word example to explain the basic concepts. This example uses the core flavor.

import work.martins.simon.expect.core._

sealed trait ErrorCase
case object PasswordIncorrect extends ErrorCase
case object InsufficientPermissions extends ErrorCase
case object UnknownError extends ErrorCase

val unknownError: Either[ErrorCase, Boolean] = Left(UnknownError)

val fullPrincipal = "kadmin/admin@EXAMPLE.COM"
val password = "<PASSWORD>"

val e = new Expect(s"kadmin -p $fullPrincipal", defaultValue = unknownError)(
  new ExpectBlock(
    new StringWhen(s"Password for $fullPrincipal: ")(
      Sendln(password)
    )
  ),
  new ExpectBlock(
    new StringWhen("Incorrect password")(
      Returning(() => Left(PasswordIncorrect)),
      Exit
    ),
    new StringWhen("kadmin: ")(
      Sendln(s"add_principal -nokey somePrincipal")
    )
  ),
  new ExpectBlock(
    new StringWhen(s"""Principal "$fullPrincipal" added.""")(
      Returning(() => Right(true))
    ),
    new StringWhen(s"""Operation requires ``add'' privilege""")(
      Returning(() => Left(InsufficientPermissions))
    )
  )
)

Expect

The Expect is the starting point. To create one you need to specify at least three things:

• command - the command to execute, in this case: s"kadmin -p $fullPrincipal".
• defaultValue - the value that is returned by your expect if you don't have any returning action. This should be an error value because you have to assume, by default, the command is going to fail.
• expectBlocks - these are defined in the second argument list. If you don't define any, the default value will be returned when you run the expect.

val e = new Expect(s"kadmin -p $fullPrincipal", defaultValue = unknownError)(
  new ExpectBlock(
    //Whens
  )
)

ExpectBlock

An ExpectBlock represents the possible outcomes of executing a command. In this example after you send the principal password to kadmin you may either get an "Incorrect password" or the kadmin shell prompt.

In order to create an expect block you need to specify at least one When.

  new ExpectBlock(
    new StringWhen("Incorrect password")(
      //Actions
    ),
    new StringWhen("kadmin: ")(
      //Actions
    )
  )

When

A When represents the list of actions that will be executed if the condition is met. In this case: if output contains "Incorrect password".

To construct a When you need to specify a list of actions which may be empty. If the list is empty then expect will simply move on to the next expect block.

    new StringWhen("Incorrect password")(
      Returning(() => Left(PasswordIncorrect)),
      Exit
    )

How expect runs

The pseudo-code of what happens when the Expect.run method is invoked is the following:

class Expect[R](command: String, defaultValue: R)(expectBlocks: ExpectBlock[R]*) {
  def run(...): Future[R] = {
    val process = new Process(command)
    val intermediateResult = IntermediateResult(
      output = "",
      value = defaultValue,
      executionAction = Continue)

    def innerRun(result: IntermediateResult[R],
                 expectBlocks: Seq[ExpectBlock[R]]): Future[R] = {
      if (expectBlocks is empty) {
        process.destroy() //Make sure to destroy the OS process
        Future.successful(intermediateResult.value)
      } else {
        expectBlocks.head.run(process, result).flatmap {
          case result @ IntermediateResult(_, _, action) => action match {
            case Continue => innerRun(result, expectBlocks.tail)
            case Terminate => innerRun(result, Seq.empty)
          }
        }
      }
    }
    
    innerRun(intermediateResult, expectBlocks)
  }
}

ExpectBlock.run corresponds roughly to the following:

def run(process: Process,
        result: IntermediateResult[R]): Future[IntermediateResult[R]] = {
  if (there is a when that matches against result.output) {
    Future(when.execute(process, result))
  } else {
    runWithMoreOutput(process, result)
  }
}

def runWithMoreOutput(process: Process,
                      result: IntermediateResult[R]): Future[IntermediateResult[R]] = {
  Future {
    val newOutput = result.output + process.read()
    if (there is a when that matches against newOutput) {
      when.execute(process, result.copy(output = newOutput))
    } else {
      throw new NoMatchingPatternException(newOutput)
    }
  } recoverWith {
    case NoMatchingPatternException(newOutput) =>
      runWithMoreOutput(process, result.copy(output = newOutput))
    case e: TimeoutException | EOFException =>
      //If a TimeoutWhen | EndOfFileWhen exists execute it.
      //Otherwise return Future.failed(e)
  }
}

And finally When.execute corresponds roughly to the following:

def execute(process: Process, result: IntermediateResult[R]): IntermediateResult[R] = {
  val trimmedOutput = result.output minus all the text up to 
                     (including) the text matched by the when
  var newResult = result.copy(output = trimmedOutput)
  actions foreach {
    case Send(text) =>
      process.print(text)
    case Returning(newValue) =>
      newResult = newResult.copy(value = newValue)
    case Exit =>
      //Preemptive exit to guarantee anything after Exit does not get executed
      return newResult.copy(executionAction = Terminate)
  }
  newResult
}