DynamoDBRecord throws decimal.Rounded for large numbers (over 38 digits)

[miikka]

Expected Behaviour

When encountering a DynamoDB Streams event with a large number in the new/old image such as {"N": "110111111111111110000000000000000000000"}, Powertools' DynamoDB Streams data classes should be able to deserialize it like any other event. That is currently not the case if the number string is longer than 38 characters.

According to DynamoDB docs, numbers may have up to 38 digits of precision. However, the trailing zeros do not count towards the precision and creating a document with the number above works fine.

(Note: if you try to create such document using AWS Console, the editor there blocks it unless you use the JSON view. The form editor seems to use a simpler check for the number validity.)

Current Behaviour

A decimal.Rounded exception gets thrown with a stacktrace like this when you try to access record.dynamodb.new_image when using DynamoDBRecord/DynamoDBStreamEvent from DynamoDB Streams data classes to process a DynamoDB Streams events:

Rounded: [<class 'decimal.Rounded'>]
  File "aws_lambda_powertools/middleware_factory/factory.py", line 135, in wrapper
    response = middleware()
  File "aws_lambda_powertools/utilities/data_classes/event_source.py", line 39, in event_source
    return handler(data_class(event), context)
  File "handler.py", line 61, in handler
    xxx
  File "utils.py", line 88, in xxx
    item = record.dynamodb.new_image
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 155, in new_image
    return self._deserialize_dynamodb_dict("NewImage")
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 139, in _deserialize_dynamodb_dict
    return {k: self._deserializer.deserialize(v) for k, v in dynamodb_dict.items()}
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 139, in <dictcomp>
    return {k: self._deserializer.deserialize(v) for k, v in dynamodb_dict.items()}
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 67, in deserialize
    return deserializer(value[dynamodb_type])
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 97, in _deserialize_m
    return {k: self.deserialize(v) for k, v in value.items()}
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 97, in <dictcomp>
    return {k: self.deserialize(v) for k, v in value.items()}
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 67, in deserialize
    return deserializer(value[dynamodb_type])
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 97, in _deserialize_m
    return {k: self.deserialize(v) for k, v in value.items()}
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 97, in <dictcomp>
    return {k: self.deserialize(v) for k, v in value.items()}
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 67, in deserialize
    return deserializer(value[dynamodb_type])
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 94, in _deserialize_l
    return [self.deserialize(v) for v in value]
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 94, in <listcomp>
    return [self.deserialize(v) for v in value]
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 67, in deserialize
    return deserializer(value[dynamodb_type])
  File "aws_lambda_powertools/utilities/data_classes/dynamo_db_stream_event.py", line 76, in _deserialize_n
    return DYNAMODB_CONTEXT.create_decimal(value)

Code snippet

from aws_lambda_powertools.utilities.data_classes.dynamo_db_stream_event import DynamoDBRecord
data = {
  "awsRegion": "eu-central-1",
  "dynamodb": {
    "ApproximateCreationDateTime": 1722318027.0,
    "Keys": {"id": {"S": "test"}},
    "NewImage": {
      "id": {"S": "test"},
      "number": {"N": "110111111111111110000000000000000000000"},
    },
    "OldImage": {
      "id": {"S": "test"},
    }
  }
}
DynamoDBRecord(data).dynamodb.new_image

Possible Solution

No response

Steps to Reproduce

1. Create a lambda that processes DynamoDB Streams events with the Powertools data classes or the batch processing support and call record.dynamodb.new_image for a record in the event.
2. Create a new item in the DynamoDB table with an attribute that has type "number" and value 110111111111111110000000000000000000000

Powertools for AWS Lambda (Python) version

2.40.1

AWS Lambda function runtime

3.11

Packaging format used

Lambda Layers

Debugging logs

No response

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[sthulb]

Hi @miikka I’ll look into this tomorrow. Thanks for reporting it.

[sthulb]

Hi @miikka I've prepped a PR #4863 to handle this, it'll strip off tailing 0s after the 38th char (as per the docs).

Let me know if this would be sufficient.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under 2.43.0 version!