feat(event_source): deserialize dynamodb new_image, old_image as dict

[shanab]

Issue number: #1432

Summary

Add DynamoDBImageDeserializer class to data classes. The class can be used to parse DynamoDB StreamRecord's old_image and new_image properties of type Dict[str, AttrValue] to Dict with Python types.

Changes

• Add DynamoDBImageDeserializer class to data classes.
• Add unit test.

User experience

Users will be able to convert a StreamRecord's complex image type into a Python dictionary with Python types; without needing to recurse through the data.

Example:

    data = {
        "Keys": {"key1": {"attr1": "value1"}},
        "NewImage": {
            "Name": {"S": "Joe"},
            "Age": {"N": "35"},
            "TypesMap": {
                "M": {
                    "string": {"S": "value"},
                    "number": {"N": "100"},
                    "bool": {"BOOL": True},
                    "dict": {"M": {"key": {"S": "value"}}},
                    "stringSet": {"SS": ["item1", "item2"]},
                    "numberSet": {"NS": ["100", "200", "300"]},
                    "byteSet": {"BS": byte_list},
                    "list": {"L": [{"S": "item1"}, {"N": "3.14159"}, {"BOOL": False}]},
                    "null": {"NULL": True},
                },
            },
        },
    }
    record = StreamRecord(data)
    deserializer = DynamoDBImageDeserializer()
    new_image = deserializer.deserialize(record.new_image)
    assert new_image == {
        "Name": "Joe",
        "Age": "35",
        "TypesMap": {
            "string": "value",
            "number": "100",
            "bool": True,
            "dict": {"key": "value"},
            "stringSet": {"item1", "item2"},
            "numberSet": {"100", "200", "300"},
            "byteSet": set(byte_list),
            "list": ["item1", "3.14159", False],
            "null": None,
        },
    }

Checklist

If your change doesn't seem to apply, please leave them unchecked.

• Meet tenets criteria
• I have performed a self-review of this change
• Changes have been tested
• Changes are documented
• PR title follows conventional commit semantics

Is this a breaking change?

No
RFC issue number:

Checklist:

• Migration process documented
• Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[boring-cyborg]

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our AWS Lambda Powertools Discord: Invite link

[shanab]

Do I need to update the public docs for this change?

[leandrodamascena]

Do I need to update the public docs for this change?

Hi @shanab! First of all thank you so much for submitting this PR and helping to improve this project!

It looks like we have errors in the pipeline. Could you check for errors and fix them? We suggest running make pr before push the code and checking that everything is ok.

[shanab]

Thanks @leandrodamascena! I've fixed the typing errors.

[leandrodamascena]

Thanks @leandrodamascena! I've fixed the typing errors.

Awesome @shanab! I'll review this PR this week and let you know if we need to fix anything, ok?

BTW, we can update the documentation for you, but let us know if you want to update this and push the code 🥇 .

Thank you!

[heitorlessa]

@shanab a few questions if I may as I'm only seeing this now:

• What would you expect customers to do with TypesMap key? Traverse, pick the type they're interested in, or something else?

• Since it's a Map, what's the typical customer expectation on deserialisation? Get it back as a Dict?

• Given we're releasing v2 and we can make a breaking change, would it be easier to get rid of Attribute Value and store the values already deserialised with boto3 TypeDeserializer?

I think what I'm missing is the customer experience after you get the data deserialised like that vs a Dict and its values - v2 is just around the corner, we might simplify maintenance and perf by making any necessary breaking change.

Thanks a lot

[heitorlessa]

Thanks for accepting a 1:1 call on this @shanab ! Here's a quick recap on the conditions we could make it this work if we can get it done until Thursday to not impact V2 launch:

• Update new_image, old_image, and keys properties to return a deserialized Dict - Code here
• Keep Decimals as str to continue compatibility with current implementation, and to support precisions higher than 38 digits (IIRC DynamoDB doesn't by default)
• Let's not cache deserialized data as we might be dealing with thousands of records and we need to keep it memory efficient (we already use a generator for .records)
• Update the docs to describe how to use that data (now being dict) - Docs here
• Switch your base to v2 - it might be easier to create a new branch from v2 given the amount of conflicts right now
• Ping me directly if you need any help - we can do peer review live to get this faster, and I can write the Upgrade Guide too.

[shanab]

I've agreed with @rubenfonseca to move to a new PR, as the diff just blew up once I changed the base branch from develop to v2.
I'll link the new PR here once I do that.

[shanab]

Link to the new PR: #1619