Refactor Processing Function (Extract Artifacts) & Add Calibration Workflow #20
Conversation
| id: test-lambda | ||
| run: | | ||
| # Run curl and write the HTTP status code to a variable | ||
| HTTP_STATUS=$(curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d @lambda_function/tests/test_data/test_eea_event.json) |
There was a problem hiding this comment.
Where does this CURL URL come from? This seems so random?
There was a problem hiding this comment.
This is how you can test the Lambda Function locally; it replicates what happens behind the scenes when a lambda function is invoked by whatever event. You can pass it an event for the invocation, as you can see with me passing the JSON file. https://docs.aws.amazon.com/lambda/latest/dg/images-test.html#images-test-AWSbase
|
|
||
| `docker run -p 9000:8080 -e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID myfunction:latest` | ||
| `docker run -p 9000:8080 -v <directory_for_processed_files>:/test_data -e SDC_AWS_FILE_PATH=/test_data/<file_to_process_name> processing_function:latest` |
There was a problem hiding this comment.
We don't need the Access Key environment variables anymore?
There was a problem hiding this comment.
This was used before when S3 was the only option and you couldn't calibrate a local file; it would actually run the function by needing to make the S3 calls, and that would trigger the pipeline. But since this should be for local testing and pipeline testing should only occur when this is committed to the development pipeline the access keys are no longer needed for local development/testing.
| ) | ||
|
|
||
| try: | ||
| # Initialize the slack client |
There was a problem hiding this comment.
Remind me where this, the timestream, and the CDFTracker were moved to?
There was a problem hiding this comment.
Or did we just remove them from the lambda?
There was a problem hiding this comment.
Everything that generates artifacts was refactored into its own Lambda function, which is triggered by the same S3 Buckets and runs in parallel, you can find it here: https://github.com/HERMES-SOC/sdc_aws_artifacts_lambda
This update refactors the Processing Lambda in a different repository, complementing the Calibration Workflow added in the this PR. This PR needs to be merged first.
The refactor involves removing the artifact generation components and transferring them to a separate lambda function. This new function will be triggered by the same S3 Bucket Events. A new repo will be created and infrastructure will be added for this new function.
Additionally, this update introduces a feature for testing local science files against the pipeline, as well as testing data files already present within the instrument packages. This feature is a critical part of the calibration workflow, which evaluates the calibration of files and then posts the artifacts as downloadable links in a comment on the GitHub page. This workflow is designed to work with the Calibration Workflow, ensuring a streamlined process for calibrating and validating instrument data.
Also updates the README for local testing, for both the methods described above.