Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removes query resources which require where clauses and removes deprecated field #35

Merged

Conversation

tektite-io
Copy link
Contributor

While testing ThreatMapper cloud inventory with very big accounts, error logs in steampipe showed the following

Thu, 10 Oct 2024 09:27:34 +0000 ERR query_resource/query.go:189 Error at querying res: exit status 41 for query: steampipe query --output json "select \"region\" ,
\"resource_id\" , \"service_namespace\" , \"title\" from aws_xxxx.aws_appautoscaling_target "
Thu, 10 Oct 2024 09:27:34 +0000 ERR query_resource/query.go:190 Error: rpc error: code = Internal desc = aws_xxxx: rpc error: code = Internal desc = 'List'
call for table 'aws_appautoscaling_target' is missing 1 required qual: column:'service_namespace' operator: =
 (SQLSTATE HV000)
 Thu, 10 Oct 2024 09:24:43 +0000 ERR query_resource/query.go:189 Error at querying res: exit status 41 for query: steampipe query --output json "select \"account_id\
" , \"compliance_type\" , \"id\" , \"region\" , \"resource_id\" , \"status\" , \"title\" from aws_xxxx.aws_ssm_managed_instance_compliance "
Thu, 10 Oct 2024 09:24:43 +0000 ERR query_resource/query.go:190 Error: rpc error: code = Internal desc = aws_xxxx: rpc error: code = Internal desc = 'List'
call for table 'aws_ssm_managed_instance_compliance' is missing 1 required qual: column:'resource_id' operator: =
(SQLSTATE HV000)

and

steampipe query --output json "select \"account_id\" , \"account_ids\" , \"arn\" , \"name\" , \"owner_type\" , \"partition\" , \"region\" , \"title\" from aws_xxxx.aws_ssm_document "
Error: aws_xxxx: table 'aws_ssm_document' column 'account_ids' requires hydrate data from getAwsSSMDocumentPermissionDetail, which failed with error operation error SSM: DescribeDocumentPermission, exceeded maximum number of attempts, 9, https response error StatusCode: 400, RequestID: 4d8ce258-5a96-459e-9ccd-1f53796a4464, api error ThrottlingException: Rate exceeded.
(SQLSTATE HV000)

aws_appautoscaling_target and aws_ssm_managed_instance_compliance require a where clause (and hence the error) while aws_ssm_document was using account_ids which is deprecated, and it was slowing down the inventory completion due to the numerous errors (with huge accounts for example),

This PR simply remove the account_ids from aws_ssm_document and delete the two tables aws_ssm_managed_instance_compliance and aws_appautoscaling_target

tektite-io and others added 2 commits October 10, 2024 19:14
- removes aws_ssm__managed_instance_compliance
As they are not listing tables, but they requires where clauses
- removes the account_ids from aws_ssm_document as they are deprecated
@ramanan-ravi ramanan-ravi self-requested a review October 11, 2024 15:14
@ramanan-ravi
Copy link
Contributor

Thanks for the contribution @tektite-io

@ramanan-ravi ramanan-ravi merged commit a238500 into deepfence:release-2.4 Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants