policy parameters w/ vanilla Python expressions instead of Numpy functions

[MattHJensen]

In #1081, @martinholmer created the capability to create a policy parameter that is a Python expression rather than a number. This is a major innovation that implements an idea originally raised by @feenberg in #429.

The implementation in #1081 allows the user to interact with whitelisted records variables using Numpy array functions (maximum, minimum, and where).

It would be valuable if the user could use vanilla python expressions (max, min, if, and, or) in addition to or instead of Numpy array functions. The most important of these is to allow if statements whether than where functions.

One reason for preferring vanilla Python syntax is that experience from Tax-Calculator's early days shows that using where statements for tax logic can get very messy. See https://github.com/talumbau/Tax-Calculator/blob/4cdc5841fefc5d175c9e5e538c7fd9c670fe35d1/translation.py#L1243-L1367s. (We also learned that Python's and and or are much nicer for tax logic than Numpy's alternatives.)

A second reason for preferring vanilla Python syntax is that we use vanilla Python syntax wherever possible in functions.py, and using Numpy syntax for these "hooks" will require users to familiarize themselves with reading tax logic in two syntaxes.

A third reason for preferring vanilla Python syntax is that I expect it will be generally more familiar to TaxBrain users from non-Python backgrounds.

@talumbau, I'm wondering if some of the @iterate_jit technology that you developed for Tax-Calculator functions.py could also be applied to these hooks.

[martinholmer]

@MattHJensen said:

The implementation in #1081 allows the user to interact with whitelisted records variables using Numpy array functions (maximum, minimum, and where).

Actually, that is not precisely true. The functions available for use in parameter expressions at the moment are "max", "min", and "where". This is an important point because issue #1102 is largely about syntatic sugar. The "max" function is an alias for numpy.maximum and the "min" function is an alias for the numpy.minimum function. So, "max" and "min" in policy parameter expressions are identical to the "vanilla python" functions.

It would be valuable if the user could use vanilla python expressions (max, min, if, and, or) in addition to or instead of Numpy array functions. The most important of these is to allow if statements rather than where functions.

Perhaps this syntactic sugar would make expression writing somewhat sweeter for some users, but there is a cost in development complexity and increased potential for errors that will be very difficult to detect. I think it is way too early in the history of policy parameter expressions to know exactly how big a problem it will be to ask users to use a "where" function rather than an "if" statement. Why not wait to discuss this when we have some experience with using the parameter expression capability? How many of those on the development team have used the capabilities on the master branch to write parameter expressions? Can those who have used this feature provide us examples of the expressions you have used? And can you share with the rest of the development team any expressions that you couldn't write with the current "max", "min", and "where" functions? And can you share expressions that became difficult to read/understand because of the nature of the "where" function?

@feenberg @Amy-Xu @zrisher @GoFroggyRun @andersonfrailey @codykallen

[feenberg]

On Sun, 11 Dec 2016, Matt Jensen wrote: In #1081, @martinholmer created the capability to create a policy parameter that is a Python expression rather than a number. This is a major innovation that implements an idea originally raised by @feenberg in #429. The implementation in #1081 allows the user to interact with whitelisted records variables using Numpy array functions (maximum, minimum, and where). It would be valuable if the user could use vanilla python expressions (max, min, if, and, or) in addition to or instead of Numpy array functions. The most important of these is to allow if statements whether than where functions.

I expect mechanically converting Python max and min to numpy for the user would be fairly easy, but if-then-else requires some higher CS skills. Don't we have that available? Is it feasible? The where function is a very inferior if-then and is deservably unfamiliar to users. dan

[talumbau]

just one clarifying question: is the desire to use the "ternary operator" form (or rather, Python's equivalent) which looks like this:

   x = y if cond else z

or do you want to use the full Python if/else construct:

    if cond:
        statement1
    elif cond2:
        statement2
    else:
        statement 3

[MattHJensen]

@martinholmer said:

Why not wait to discuss this when we have some experience with using the parameter expression capability?

You're absolutely right. I've had an example in my head for a long time now but realize I haven't communicated that to anyone else. Here it is:

#1103 (comment)

@talumbau, the example above also answers your question.

[feenberg]

On Mon, 12 Dec 2016, T.J. Alumbaugh wrote: just one clarifying question: is the desire to use the "ternary operator" form (or rather, Python's equivalent) which looks like this: x = y if cond else z

I wasn't aware of that possibility. The key is how it looks when there is more than a binary choice. Does this look right? x = ( .05 if year==2016 else ((.1 if year==2017 else ( .15 if year==2018 else .20)))) but it looks like the where function could be expressed in similar format: x = where(year==2016,.05, where(year==2017..10, where(year==2018,.15),.2)) so that it was at least somewhat readable for very simple expressions. While this is a stylistic matter, the where expressions I saw in the calculator were never formatted for even this much readability. So I don't think the if-cond operator is much help. Really the question is if we want to allow users to supply a function or just an expression. I understand that an expression can be fully general, but I don't think users will be able to translate all of their ideas into functions without getting confused. Earlier I suggested that we allow Python if statements and assignments. We could copy those into a prebuilt dummy function (after some vetting) and execute as normal (slow) Python without numpy or compilation. As I understand it, we can limit access from the function to other Python facilities in a fairly robust and straightforward manner. Is it not possible to mix normal Python with our system? The slowness of such a small part of the system wouldn't be a terrible burden on us, and would greatly expand the acceptability to users. Nevertheless, if we can't do everything, we should still do something. dan

…

or do you want to use the full Python if/else construct: if cond: statement1 elif cond2: statement2 else: statement 3 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.[AHvQVQJnPPj-BBW8XEQqBZ0P_eOj5ZpMks5rHUd_gaJpZM4LKH70.gif]

[martinholmer]

The parameter-code capability was developed and then removed from Tax-Calculator as described in pull request #1221.