New benchmarks discussion

[MatthewCaseres]

Desired improvements to benchmarks

Two things I'd like to improve upon from the benchmarks in the Learn repo -

• Benchmarks should be representative of real workloads from industry
• Benchmarks should run in reproducible (containerized) infrastructure in the cloud
  • Benchmarks will probably be reported on for different infrastructure configs. Like single CPU, multi CPU, single GPU, multi GPU

Preliminary research

• I am planning on using GCP Vertex AI to run these benchmarks, primarily because they let you use containers from Docker Hub (unlike AWS SageMaker which requires that containers be on ECR).
• I've written some code and published a container to Docker Hub.
  • This code isn't representative of a real workload from industry, just a demo
• I am waiting for GCP to provision me the GPU to run these experiments, I have requested a quota increase from 0 to 4 T4 GPUs for custom training jobs.

Potential for collaboration?

I imagine a lifelib style project living within the JuliaActuary GitHub organization that can be competed against. Would you be open to creating a lifelib.jl project under JuliaActuary where I can try to implement some more complicated actuarial models in hopes of racing against them with JAX? The ask isn't for development effort, but for some oversight and quality control.

I will likely maintain some repo of config files for running the containers over on the actuarialopensource github org.

[alecloudenback]

Re: Benchmarks

All good ideas - I originally wanted to create runnable containers for the reasons you listed but gave up for the time being due to the complexity of putting it all into containers.

One thing about cloud benchmarks though - unless you are running on dedicated hardware, you can get temporary slowdowns as the server you are running on gets a heavy request from another user. You can smooth this out by averaging across many attempts, but for the extra runtime you might as well buy dedicated server time.

Re: Workloads

Agreed about the workloads - comments on the existing:

• "Life Modeling Problem" - this toy example was proposed by someone on LinkedIn as a way to compare benchmarks and I just used it. @DimitarVanguelov has pointed out that it really should be done on longer vectors as well to represent more typical calculations, I just haven't gone through the trouble of implementing.
• "IRR" - IIRC, this was another one where someone had posted about how much faster better-irr was than numpy-financial so I had to test this :)
• Black-Scholes - I was experimenting with how to most flexibly design a BS function (still not wholly satisfied) and just made the comparison out of my own curiosity

Re: Lifelib alternative

I love the idea of a more fully featured modeling option. Something I've thought a lot about and ultimately want to build up to. It should probably not use the same "lifelib" name to allow lifelib to retain its well-deserved acclaim.

What are you thinking for design of such a package?

Some misc thoughts I've had:

• I'm not sure that a dependency cell structure is necessary at the formula level? Maybe at the cell level though.
• Every actuarial model of sufficient size ultimately needs accounting logic. (a prototype interface)
• Is there a place for creating a DSL to assist with model definition?
• Ideally, each model point should be able to be projected forward (e.g. inner projections, restarting partial projections)

[alecloudenback]

I'll take a look by this weekend.

[MatthewCaseres]

cool. I guess I'm curious if it is even desirable to formulate life actuarial concepts in terms of "tensor" operations, or if the idiomatic Julia thing is just to use structs and loops.

[alecloudenback]

I guess I'm curious if it is even desirable to formulate life actuarial concepts in terms of "tensor" operations, or if the idiomatic Julia thing is just to use structs and loops.

Broadcasting and fused broadcasting of arrays is common in Julia, and much of the same syntax in that case will work if using GPU (CUDA.jl, Metal.jl, etc).

The code that I am writing in JAX has no nice translation to Julia just because of how mortalitytables.jl represents select tables. The use of offset arrays and having different lengths of select mortality vectors won't play well with the pattern I've been using in JAX for this

Two comments:

1. I found that treating mortality tables as a collection of different length vectors works much better in practice than as a matrix when you have a heterogeneous population of policies where issue ages and durations can be all over the place.

2. I think the way of constructing this is actually pretty simple:

using MortalityTables

tbls = [MortalityTables.table(i) for i in 3299:3308];
issue_ages = 18:51
durations = 1:25

qs = [
	tbl.select[issue_age][issue_age+duration-1]
	for   duration in durations, issue_age in issue_ages,  tbl in tbls
]

face = one.(qs) * 1_000_000 # create an array of face amounts the same dimensions as `qs`

And use, e.g. CuArray(qs) from CUDA.jl if you want it to be an array that will operate on the GPU if the operations you want are supported by the GPU.

I think those ~15 lines are accomplishing everything these ~50 lines do:

import jax.numpy as jnp
import jax
from pymort import getIdGroup, MortXML
import equinox as eqx
from jaxtyping import f, i # declaring what the axes are is nice

class GetQ(eqx.Module):
  select: f["table_index issue_age duration"] 
  ultimate: f["table_index attained_age"]

  def __init__(self, collection_id: int):
    ids = getIdGroup(collection_id).ids
    self.select = jnp.array([MortXML(id).Tables[0].Values.unstack().values for id in ids])
    self.ultimate = jnp.array([MortXML(id).Tables[1].Values.unstack().values for id in ids])

  def __call__(self, table_index: f["policies"], issue_age: f["policies"], duration_projected: f["timesteps policies"]) -> f["timesteps policies"]:
    return jnp.where(
        duration_projected < self.select.shape[-1],
        self.select[table_index, issue_age - 18, duration_projected], # offset because jax arrays are 0-indexed
        self.ultimate[table_index, (issue_age - 18) + duration_projected],
    )

get_q = GetQ(3299)

class ModelPoints(eqx.Module):
  mortality_table_index: i["policies"]
  issue_age: i["policies"]
  duration: i["policies"]
  face: i["policies"]

  def projected_q(self, timesteps: int) -> f["timesteps policies"]:
    """
    Get the mortality rates q for the policies across time.
    """
    time_axis = jnp.arange(timesteps)[:, jnp.newaxis]
    duration_projected = time_axis + self.duration
    return get_q(self.mortality_table_index, self.issue_age, duration_projected)

# every combination of mortality table index from [0,10), duration from [0, 25), and issue_age from [18, 51)
# 10*25*(51-18) = 8250 modelpoints in total
mortality_table_index = jnp.arange(10)
duration = jnp.arange(1)
issue_age = jnp.arange(18, 51)
# like a cartesian product, generate all combinations of each class/duration/issue_age
mortality_table_index, duration, issue_age = [x.flatten() for x in jnp.meshgrid(mortality_table_index, duration, issue_age)]
# all policies have same face value
face = jnp.ones_like(issue_age) * 1_000_000
mp = ModelPoints(mortality_table_index, issue_age, duration, face)

Here's a pluto notebook which I think does the prefix sum example in a similar way (multidimensional array operations):
https://gist.github.com/alecloudenback/1263f8d3fbcc0d3763fd2884b72c2732

[MatthewCaseres]

many thanks for sending the code over. I think the mortality table software benchmarks are complete. Julia is very fast, and I think your software is well designed too.

https://github.com/actuarialopensource/benchmarks

[alecloudenback]

Thanks!