Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for tuples #1235

Merged
merged 11 commits into from
Mar 6, 2019
Merged

Add support for tuples #1235

merged 11 commits into from
Mar 6, 2019

Conversation

davesque
Copy link
Contributor

@davesque davesque commented Feb 5, 2019

What was wrong?

This is a rebuild of PR #1147 . Commits retain authorship from original PR where appropriate.

How was it fixed?

Adds support for tuples by modifying low-level utility functions used for ABI dict processing and encodability checks.

Cute Animal Picture

Put a link to a cute animal picture inside the parenthesis-->

@davesque
Copy link
Contributor Author

davesque commented Feb 5, 2019

I'm adding things bit by bit here. The most recent chunk of work done was to rewrite the get_abi_inputs utility function to be simpler and more robust.

Eventually, I'll give things another once over to make sure I didn't miss anything.

Known issues:

  • Updated get_abi_*_types functions need to have updated tests
  • New get_abi_inputs function needs more tests. Will probably add some or all of the tests written by @feuGeneA for that function.

# Component is tuple (possibly tuple list)...

array_dims = abi_type_str[TUPLE_PREFIX_LEN:]
if array_dims:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we tighten this up at all? Match against a regex or something? This jumps out as a good way to end up with weird errors jumping out in the case that the ABI type string is malformed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. I'll add that.

new_comp = copy.copy(comp)
new_comp['type'] = TUPLE_PREFIX

# Assume `arg` is iterable
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, easy to do a quick runtime check that isinstance(arg, collections.abc.Iterable) or maybe tighter as not isinstance(arg, (bytes, str)) and isinstance(arg, collections.abc.Sequence)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a runtime type check using eth_utils.is_list_like.


if isinstance(arg, abc.Mapping):
# Arg is mapping. Convert to properly ordered sequence.
arg = tuple(arg[c['name']] for c in sub_comps)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's going to take me a long time to not have a knee-jerk reflex that python dictionaries aren't ordered (since they now are)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I know they're ordered. I still think that most computer scientists assume that mappings provide no guarantees about any order that results from the internal representation of the data structure. Users could also provide a dictionary with all the right keys but the wrong order.

Copy link
Contributor Author

@davesque davesque Feb 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you meant sub_comps, I believe that should always be a list instance or an itertools.repeat iterator.

@davesque davesque force-pushed the tuples branch 6 times, most recently from f4fdafc to b35c254 Compare February 7, 2019 21:29
@davesque davesque force-pushed the tuples branch 6 times, most recently from c7c6d19 to 85c1837 Compare February 23, 2019 07:15
@davesque davesque changed the title [WIP] Add support for tuples Add support for tuples Feb 23, 2019
@davesque davesque force-pushed the tuples branch 3 times, most recently from 8e76bd8 to 808fa92 Compare February 23, 2019 08:24
Copy link
Collaborator

@carver carver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most comments are about style: For example, I find single-letter and abbreviated variables a chore to read through, even in single-line comprehensions. There are quite a few, and I only commented on a couple of them.

The only substantive piece is that we should probably add tests for nested arrays, and some other deep-nested scenarios, like a double-nested tuple in a double-nested array.

"""
CONTRACT_TUPLE_CODE = '0x608060405234801561001057600080fd5b506108ca806100206000396000f300608060405260043610610041576000357c0100000000000000000000000000000000000000000000000000000000900463ffffffff1680638e1ae3c714610046575b600080fd5b34801561005257600080fd5b5061006d60048036036100689190810190610403565b610083565b60405161007a9190610696565b60405180910390f35b61008b610093565b819050919050565b6060604051908101604052806000815260200160608152602001606081525090565b60006100c18235610850565b905092915050565b600082601f83011215156100dc57600080fd5b81356100ef6100ea826106e5565b6106b8565b9150818183526020840193506020810190508385602084028201111561011457600080fd5b60005b83811015610144578161012a88826100b5565b845260208401935060208301925050600181019050610117565b5050505092915050565b600082601f830112151561016157600080fd5b600261017461016f8261070d565b6106b8565b9150818385602084028201111561018a57600080fd5b60005b838110156101ba57816101a088826102bf565b84526020840193506020830192505060018101905061018d565b5050505092915050565b600082601f83011215156101d757600080fd5b81356101ea6101e58261072f565b6106b8565b9150818183526020840193506020810190508360005b8381101561023057813586016102168882610377565b845260208401935060208301925050600181019050610200565b5050505092915050565b600082601f830112151561024d57600080fd5b813561026061025b82610757565b6106b8565b9150818183526020840193506020810190508385602084028201111561028557600080fd5b60005b838110156102b5578161029b88826103ef565b845260208401935060208301925050600181019050610288565b5050505092915050565b60006102cb8235610870565b905092915050565b60006102df823561087c565b905092915050565b6000606082840312156102f957600080fd5b61030360606106b8565b90506000610313848285016103ef565b600083015250602082013567ffffffffffffffff81111561033357600080fd5b61033f8482850161023a565b602083015250604082013567ffffffffffffffff81111561035f57600080fd5b61036b848285016101c4565b60408301525092915050565b60006080828403121561038957600080fd5b61039360606106b8565b905060006103a3848285016102d3565b60008301525060206103b78482850161014e565b602083015250606082013567ffffffffffffffff8111156103d757600080fd5b6103e3848285016100c9565b60408301525092915050565b60006103fb8235610886565b905092915050565b60006020828403121561041557600080fd5b600082013567ffffffffffffffff81111561042f57600080fd5b61043b848285016102e7565b91505092915050565b61044d81610810565b82525050565b600061045e826107b0565b8084526020840193506104708361077f565b60005b828110156104a257610486868351610444565b61048f826107dc565b9150602086019550600181019050610473565b50849250505092915050565b6104b7816107bb565b6104c08261078c565b60005b828110156104f2576104d68583516105c2565b6104df826107e9565b91506020850194506001810190506104c3565b5050505050565b6000610504826107c6565b8084526020840193508360208202850161051d85610796565b60005b84811015610556578383038852610538838351610637565b9250610543826107f6565b9150602088019750600181019050610520565b508196508694505050505092915050565b6000610572826107d1565b808452602084019350610584836107a3565b60005b828110156105b65761059a868351610687565b6105a382610803565b9150602086019550600181019050610587565b50849250505092915050565b6105cb81610830565b82525050565b6105da8161083c565b82525050565b60006060830160008301516105f86000860182610687565b50602083015184820360208601526106108282610567565b9150506040830151848203604086015261062a82826104f9565b9150508091505092915050565b600060808301600083015161064f60008601826105d1565b50602083015161066260208601826104ae565b506040830151848203606086015261067a8282610453565b9150508091505092915050565b61069081610846565b82525050565b600060208201905081810360008301526106b081846105e0565b905092915050565b6000604051905081810181811067ffffffffffffffff821117156106db57600080fd5b8060405250919050565b600067ffffffffffffffff8211156106fc57600080fd5b602082029050602081019050919050565b600067ffffffffffffffff82111561072457600080fd5b602082029050919050565b600067ffffffffffffffff82111561074657600080fd5b602082029050602081019050919050565b600067ffffffffffffffff82111561076e57600080fd5b602082029050602081019050919050565b6000602082019050919050565b6000819050919050565b6000602082019050919050565b6000602082019050919050565b600081519050919050565b600060029050919050565b600081519050919050565b600081519050919050565b6000602082019050919050565b6000602082019050919050565b6000602082019050919050565b6000602082019050919050565b600073ffffffffffffffffffffffffffffffffffffffff82169050919050565b60008115159050919050565b6000819050919050565b6000819050919050565b600073ffffffffffffffffffffffffffffffffffffffff82169050919050565b60008115159050919050565b6000819050919050565b60008190509190505600a265627a7a72305820fb4abae14c4bcd2cd8a12f35862cb09a2e0f9004eb5fdbfdb76658a753570c8c6c6578706572696d656e74616cf50037' # noqa: E501
CONTRACT_TUPLE_RUNTIME = '0x608060405260043610610041576000357c0100000000000000000000000000000000000000000000000000000000900463ffffffff1680638e1ae3c714610046575b600080fd5b34801561005257600080fd5b5061006d60048036036100689190810190610403565b610083565b60405161007a9190610696565b60405180910390f35b61008b610093565b819050919050565b6060604051908101604052806000815260200160608152602001606081525090565b60006100c18235610850565b905092915050565b600082601f83011215156100dc57600080fd5b81356100ef6100ea826106e5565b6106b8565b9150818183526020840193506020810190508385602084028201111561011457600080fd5b60005b83811015610144578161012a88826100b5565b845260208401935060208301925050600181019050610117565b5050505092915050565b600082601f830112151561016157600080fd5b600261017461016f8261070d565b6106b8565b9150818385602084028201111561018a57600080fd5b60005b838110156101ba57816101a088826102bf565b84526020840193506020830192505060018101905061018d565b5050505092915050565b600082601f83011215156101d757600080fd5b81356101ea6101e58261072f565b6106b8565b9150818183526020840193506020810190508360005b8381101561023057813586016102168882610377565b845260208401935060208301925050600181019050610200565b5050505092915050565b600082601f830112151561024d57600080fd5b813561026061025b82610757565b6106b8565b9150818183526020840193506020810190508385602084028201111561028557600080fd5b60005b838110156102b5578161029b88826103ef565b845260208401935060208301925050600181019050610288565b5050505092915050565b60006102cb8235610870565b905092915050565b60006102df823561087c565b905092915050565b6000606082840312156102f957600080fd5b61030360606106b8565b90506000610313848285016103ef565b600083015250602082013567ffffffffffffffff81111561033357600080fd5b61033f8482850161023a565b602083015250604082013567ffffffffffffffff81111561035f57600080fd5b61036b848285016101c4565b60408301525092915050565b60006080828403121561038957600080fd5b61039360606106b8565b905060006103a3848285016102d3565b60008301525060206103b78482850161014e565b602083015250606082013567ffffffffffffffff8111156103d757600080fd5b6103e3848285016100c9565b60408301525092915050565b60006103fb8235610886565b905092915050565b60006020828403121561041557600080fd5b600082013567ffffffffffffffff81111561042f57600080fd5b61043b848285016102e7565b91505092915050565b61044d81610810565b82525050565b600061045e826107b0565b8084526020840193506104708361077f565b60005b828110156104a257610486868351610444565b61048f826107dc565b9150602086019550600181019050610473565b50849250505092915050565b6104b7816107bb565b6104c08261078c565b60005b828110156104f2576104d68583516105c2565b6104df826107e9565b91506020850194506001810190506104c3565b5050505050565b6000610504826107c6565b8084526020840193508360208202850161051d85610796565b60005b84811015610556578383038852610538838351610637565b9250610543826107f6565b9150602088019750600181019050610520565b508196508694505050505092915050565b6000610572826107d1565b808452602084019350610584836107a3565b60005b828110156105b65761059a868351610687565b6105a382610803565b9150602086019550600181019050610587565b50849250505092915050565b6105cb81610830565b82525050565b6105da8161083c565b82525050565b60006060830160008301516105f86000860182610687565b50602083015184820360208601526106108282610567565b9150506040830151848203604086015261062a82826104f9565b9150508091505092915050565b600060808301600083015161064f60008601826105d1565b50602083015161066260208601826104ae565b506040830151848203606086015261067a8282610453565b9150508091505092915050565b61069081610846565b82525050565b600060208201905081810360008301526106b081846105e0565b905092915050565b6000604051905081810181811067ffffffffffffffff821117156106db57600080fd5b8060405250919050565b600067ffffffffffffffff8211156106fc57600080fd5b602082029050602081019050919050565b600067ffffffffffffffff82111561072457600080fd5b602082029050919050565b600067ffffffffffffffff82111561074657600080fd5b602082029050602081019050919050565b600067ffffffffffffffff82111561076e57600080fd5b602082029050602081019050919050565b6000602082019050919050565b6000819050919050565b6000602082019050919050565b6000602082019050919050565b600081519050919050565b600060029050919050565b600081519050919050565b600081519050919050565b6000602082019050919050565b6000602082019050919050565b6000602082019050919050565b6000602082019050919050565b600073ffffffffffffffffffffffffffffffffffffffff82169050919050565b60008115159050919050565b6000819050919050565b6000819050919050565b600073ffffffffffffffffffffffffffffffffffffffff82169050919050565b60008115159050919050565b6000819050919050565b60008190509190505600a265627a7a72305820fb4abae14c4bcd2cd8a12f35862cb09a2e0f9004eb5fdbfdb76658a753570c8c6c6578706572696d656e74616cf50037' # noqa: E501
CONTRACT_TUPLE_ABI = json.loads('[{"constant":true,"inputs":[{"components":[{"name":"a","type":"uint256"},{"name":"b","type":"uint256[]"},{"components":[{"name":"x","type":"int256"},{"name":"y","type":"bool[2]"},{"name":"z","type":"address[]"}],"name":"c","type":"tuple[]"}],"name":"s","type":"tuple"}],"name":"method","outputs":[{"components":[{"name":"a","type":"uint256"},{"name":"b","type":"uint256[]"},{"components":[{"name":"x","type":"int256"},{"name":"y","type":"bool[2]"},{"name":"z","type":"address[]"}],"name":"c","type":"tuple[]"}],"name":"","type":"tuple"}],"payable":false,"stateMutability":"pure","type":"function"}]') # noqa: E501
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit opinion: it's easier to understand what's going on in an ABI with a native python list, like:

CONTRACT_TUPLE_ABI = [
  {
    "constant": True,
    "inputs": [
      {
        "components":[
          {"name":"a", "type":"uint256"},
          {"name":"b", "type":"uint256[]"},
          {
            "components":[
              {"name":"x", "type":"int256"},
              {"name":"y", "type":"bool[2]"},
              {"name":"z", "type":"address[]"}
            ],
            "name":"c",
            "type":"tuple[]",
          }
        ],
        "name":"s",
        "type":"tuple",
      }
    ],
    "name":"method",
    "outputs":[{
      "components":[
        {"name":"a", "type":"uint256"},
        {"name":"b", "type":"uint256[]"},
        {"components":[
          {"name":"x", "type":"int256"},

...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just following the style of existing fixture code with that one.

FALLBACK_FUNCTION = json.loads('[{"constant": false, "inputs": [], "name": "getData", "outputs": [{"name": "r", "type": "uint256"}], "payable": false, "stateMutability": "nonpayable", "type": "function"}, {"payable": false, "stateMutability": "nonpayable", "type": "fallback"}]') # noqa: E501
MULTIPLE_FUNCTIONS = json.loads('''
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ABI's supplied as strings are already parsed as json, so it's unnecessary to do a json.loads. Although I still prefer native python over embedding json here. (Definitely not a PR blocker, though)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was following the style of existing code.

@@ -226,7 +234,10 @@ def check_if_arguments_can_be_encoded(function_abi, args, kwargs):
if len(function_abi.get('inputs', [])) != len(arguments):
return False

types = get_abi_input_types(function_abi)
try:
types, arguments = get_abi_inputs(function_abi, arguments)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit overwriting the local variable arguments. I assume this is tuple related, so maybe we need a different name to distinguish arguments from typed_arguments (or something). Since now I guess we don't want to use the old arguments value for anything except a length check.


if isinstance(args, abc.Mapping):
# `args` is mapping, convert to properly ordered sequence
args = tuple(args[i['name']] for i in inputs)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
args = tuple(args[i['name']] for i in inputs)
args = tuple(args[input['name']] for input in inputs)

nit opinion: even in short comprehensions, I think it's easier to read with a full variable name. It prevents your eyes having to dart back and forth, so you can more or less read it straight through in one pass.


def _convert_abi_input(comp, arg):
"""
Converts an argument ``arg`` corresponding to an ABI component ``comp``
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

component here seems confusing because each element of a tuple is called component. Maybe this is a arg_type, or arg_spec since it includes a name?

]),
),
),
)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this is already working, but might be nice to add a regression test for nested arrays of tuples. Seems like the kind of thing that could break in a rewrite/refactor.

# If type is array, determine item type and annotate all
# items in iterable with that type
item_type_str = abi_type.item_type.to_type_str()
data_value = [abi_sub_tree(item_type_str, i) for i in data_value]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe item instead of i? Not sure if that's what i is short for, here.

# items in iterable respectively with those types
data_value = type(data_value)(
abi_sub_tree(comp_type.to_type_str(), i)
for comp_type, i in zip(abi_type.components, data_value)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe component instead of i here?

@@ -240,6 +241,8 @@ def get_function_info(fn_name, contract_abi=None, fn_abi=None, args=None, kwargs

fn_arguments = merge_args_and_kwargs(fn_abi, args, kwargs)

_, fn_arguments = get_abi_inputs(fn_abi, fn_arguments)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit overwrote fn_arguments, but now it has a slightly different meaning. Maybe typed_arguments or annotated_arguments? (same as the one in web3/_utils/abi I commented on)

@davesque
Copy link
Contributor Author

@carver I sort of feel like avoiding single letter variables can be just as confusing. In a lot of the examples cited, the cognitive overhead of the short variable names seems pretty minimal. Also, giving things longer names often leads to a code section taking up two or three lines when it could have taken up one. I feel like the overhead of reading a statement that runs across multiple lines is as much if not more than the overhead of reading a statement that uses meaningful abbreviations. Additionally, the typical alternative of using a variable name that simply lacks pluralization is pretty confusing because then two variables only differ by one letter and it's easy to get typos.

Anyway, I'll change it if we really want but I've often felt like I don't understand the aversion to that coding pattern. I wanted to explain some of my thoughts about that.

@davesque
Copy link
Contributor Author

davesque commented Feb 26, 2019

@carver I went ahead and addressed most of your feedback. As far as the testing of deeper tuple nesting goes, is this what you had in mind?

@carver
Copy link
Collaborator

carver commented Mar 1, 2019

As far as the testing of deeper tuple nesting goes, is this what you had in mind?

Yup, that's about it @davesque ! :) I suspect there are maybe funkier things that could go wrong if someone messes about with this code, but I think this is plenty of coverage for now. 👍 :shipit:

@pipermerriam
Copy link
Member

@davesque can you confirm this is 👍 to merge (i.e. you don't have anything outstanding that you know needs to be addressed).

@kclowes or maybe @njgheorghita . IIUC correctly, this gives us support for tuples, but only as raw tuples, and not any rich object representations.

I think what comes next is some version of support for those.

  1. named tuples
  2. dictionaries
  3. arbitrary objects with attributes.

It isn't clear to me exactly how these should be implemented but I think doing them in this order probably makes sense. The NamedTuple one is probably easiest since we can generate the class from the ABI definition and return it in place of a plain tuple. This could be a core part of how web3.py extends the eth-abi API.

@davesque
Copy link
Contributor Author

davesque commented Mar 5, 2019

@pipermerriam Yes it's ready to merge. I was waiting for your review. It seemed like a fairly complex and important piece of work so I didn't want to merge after just one review.

Copy link
Member

@pipermerriam pipermerriam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two somewhat nitpicky bits of feedback. Both might be good candidates to just quickly write up in a "Good First Issue" ticket as candy for a new contributor to fix-up.


if isinstance(arg, abc.Mapping):
# Arg is mapping. Align values according to abi order.
arg = tuple(arg[abi['name']] for abi in sub_abis)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a convention of avoiding overwriting local variables, preferring to instead use new variable names. Maybe we can call this aligned_arg

if is_mapping(arg):
    aligned_arg = tuple(...)
else:
    aligned_arg = arg

NOTE: the above statement has an implicit else clause which is also somewhat against our conventions. It might should look something like.

if is_mapping(arg):
    ...
elif is_list_like(arg):
    ...
else:
    raise TypeError("Something something invalid")

This would simply be moving the TypeError from below up to be part of this clause.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also noting this relates to #1267 as it appears we already support dictionaries for the input arguments for tuple types.

# If type is array, determine item type and annotate all
# items in iterable with that type
item_type_str = abi_type.item_type.to_type_str()
data_value = [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here with convention of not overwriting local variables which if fixed will require this if/elif to get a final else clause.

@davesque
Copy link
Contributor Author

davesque commented Mar 6, 2019

@carver @pipermerriam Thanks guys for the reviews. When the tests pass, I'll go ahead and merge this.

@davesque davesque merged commit eef7c40 into ethereum:master Mar 6, 2019
@kclowes kclowes mentioned this pull request Mar 11, 2019
5 tasks
@davesque davesque deleted the tuples branch May 28, 2019 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants