Fix a case when v1/action/query returns only InternalServerError

[kpopko]

Fix a case when v1/action/query returns only InternalServerError for the queries with excluded fields

Description

Currently we are unable to do Zuora Catalog Export due to IronBank::Schema.excluded_fields throws an error each time. It could be depended to the latest Zuora Release and it blocks our work for other billing projects.

Please checkout this branch, go into bin/console and run IronBank::Schema.excluded_fields.sort.to_h ( same command billing rake task does ).
The expected output is:

{"Account"=>
  ["LastMetricsUpdate",
   "PartnerAccount",
   "PaymentMethodCascadingConsent",
   "ReservedPaymentAmount",
   "RollUpUsage",
   "SequenceSetId",
   "TaxExemptEntityUseCode",
   "TotalDebitMemoBalance",
   "UnappliedCashAmount",
   "UnappliedCreditMemoAmount"],
 "Amendment"=>["BookingDate", "EffectivePolicy", "NewRatePlanId", "RemovedRatePlanId", "SubType"],
 "CommunicationProfile"=>[],
 "Contact"=>[],
 "Export"=>["Encoding", "SnowflakeWarehouse", "SourceData", "WarehouseSize"],
 "Import"=>[],
 "Invoice"=>
  ["AutoPay",
   "BillRunId",
   "BillToContactId",
   "BillToContactSnapshotId",
   "CreditMemoAmount",
   "Currency",
   "EInvoiceErrorCode",
   "EInvoiceErrorMessage",
   "EInvoiceFileId",
   "EInvoiceStatus",
   "PaymentLink",
   "PaymentTerm",
   "SequenceSetId",
   "SoldToContactId",
   "SoldToContactSnapshotId",
   "SourceType",
   "TaxMessage",
   "TaxStatus",
   "TemplateId"],
 "InvoiceAdjustment"=>[],
 "InvoiceItem"=>
  ["AppliedToChargeNumber",
   "Balance",
   "BookingReference",
   "CommitmentChargeNumber",
   "CommitmentChargeSegmentNumber",
   "ExcludeItemBillingFromRevenueAccounting",
   "NumberOfDeliveries",
   "PurchaseOrderNumber",
   "SourceItemType"],
 "InvoicePayment"=>[],
 "Payment"=>[],
 "PaymentMethod"=>["AchCity", "AchCountry", "AchPostalCode", "AchState"],
 "Product"=>["ProductNumber"],
 "ProductRatePlan"=>[],
 "ProductRatePlanCharge"=>
  ["ApplyToBillingPeriodPartially",
   "CentralizedPrice",
   "CommitmentType",
   "CreditOption",
   "DiscountClass",
   "DiscountClassId",
   "DrawdownRate",
   "DrawdownUom",
   "ExcludeItemBillingFromRevenueAccounting",
   "ExcludeItemBookingFromRevenueAccounting",
   "Formula",
   "IsAllocationEligible",
   "IsCommitted",
   "IsPrepaid",
   "IsRollover",
   "IsStackedDiscount",
   "IsUnbilled",
   "POBPolicy",
   "PrepaidOperationType",
   "PrepaidQuantity",
   "PrepaidTotalQuantity",
   "PrepaidUom",
   "PriceUpsellQuantityStacked",
   "ProductCategory",
   "ProductClass",
   "ProductFamily",
   "ProductLine",
   "ProductRatePlanChargeNumber",
   "ProrationOption",
   "RolloverApply",
   "RolloverPeriodLength",
   "RolloverPeriods",
   "SpecificListPriceBase",
   "SplitUpsellQuantity",
   "ValidityPeriodType"],
 "ProductRatePlanChargeTier"=>["Active", "DiscountAmount", "DiscountPercentage", "IncludedUnits", "OveragePrice"],
 "RatePlan"=>["IsFromExternalCatalog", "OriginalRatePlanId", "ProductRatePlanNumber", "SubscriptionRatePlanNumber"],
 "RatePlanCharge"=>
  ["AmendedByOrderOn",
   "ApplyToBillingPeriodPartially",
   "CommitmentType",
   "CreditOption",
   "DiscountAmount",
   "DiscountClass",
   "DiscountPercentage",
   "DrawdownRate",
   "DrawdownUom",
   "ExcludeItemBillingFromRevenueAccounting",
   "ExcludeItemBookingFromRevenueAccounting",
   "IncludedUnits",
   "IsAllocationEligible",
   "IsCommitted",
   "IsPrepaid",
   "IsProcessed",
   "IsRollover",
   "IsStackedDiscount",
   "IsUnbilled",
   "NumberOfDeliveries",
   "OriginalListPrice",
   "OriginalOrderDate",
   "OveragePrice",
   "POBPolicy",
   "PaymentTermSnapshot",
   "PrepaidCommittedAmount",
   "PrepaidOperationType",
   "PrepaidQuantity",
   "PrepaidTotalQuantity",
   "PrepaidUom",
   "PriceUpsellQuantityStacked",
   "ProductCategory",
   "ProductClass",
   "ProductFamily",
   "ProductLine",
   "ProductRatePlanChargeNumber",
   "ProrationOption",
   "RevenueRecognitionRuleName",
   "RolloverApply",
   "RolloverPeriodLength",
   "RolloverPeriods",
   "SpecificListPriceBase",
   "ValidityPeriodType"],
 "RatePlanChargeTier"=>["Currency", "DiscountAmount", "DiscountPercentage", "IncludedUnits", "OveragePrice"],
 "Subscription"=>
  ["Currency",
   "InvoiceTemplateId",
   "IsLatestVersion",
   "LastBookingDate",
   "PaymentTerm",
   "Revision",
   "SequenceSetId",
   "SubscriptionBillToId",
   "SubscriptionBillToSnapshotId",
   "SubscriptionSoldToId",
   "SubscriptionSoldToSnapshotId"],
 "TaxationItem"=>
  ["Balance",
   "Country",
   "CreditAmount",
   "CustomerCode",
   "ExemptCertificate",
   "PaymentAmount",
   "SellerRegistration",
   "TaxAmountUnrounded",
   "TaxDescription",
   "TaxRuleId"],
 "Usage"=>["UniqueKey"]}

Notes

InternalServerError does not contain any information inside about failed fields.

Recursively dividing field list by two ranges each time until the failing field is not got.

References

• https://zendesk.atlassian.net/browse/VQTA-1241

Risks

Low. This extends the IronBank::Describe::ExcludedFields service capability to handle situations where the response from Zuora does not list the unqueryable fields without changing the original strategy (but at the cost of performance)

[kpopko]

Thanks for the review @jurisgalang

The run output after last commit is:

{"Account"=>["LastMetricsUpdate", "PartnerAccount", "PaymentMethodCascadingConsent", "ReservedPaymentAmount", "RollUpUsage", "SequenceSetId", "TaxExemptEntityUseCode", "TotalDebitMemoBalance", "UnappliedCashAmount", "UnappliedCreditMemoAmount"],
 "Amendment"=>["BookingDate", "EffectivePolicy", "NewRatePlanId", "RemovedRatePlanId", "SubType"],
 "CommunicationProfile"=>[],
 "Contact"=>[],
 "Export"=>["Encoding", "SnowflakeWarehouse", "SourceData", "WarehouseSize"],
 "Import"=>[],
 "Invoice"=>
  ["AutoPay",
   "BillRunId",
   "BillToContactId",
   "BillToContactSnapshotId",
   "CreditMemoAmount",
   "Currency",
   "EInvoiceErrorCode",
   "EInvoiceErrorMessage",
   "EInvoiceFileId",
   "EInvoiceStatus",
   "PaymentLink",
   "PaymentTerm",
   "SequenceSetId",
   "SoldToContactId",
   "SoldToContactSnapshotId",
   "SourceType",
   "TaxMessage",
   "TaxStatus",
   "TemplateId"],
 "InvoiceAdjustment"=>[],
 "InvoiceItem"=>["AppliedToChargeNumber", "Balance", "BookingReference", "CommitmentChargeNumber", "CommitmentChargeSegmentNumber", "ExcludeItemBillingFromRevenueAccounting", "NumberOfDeliveries", "PurchaseOrderNumber", "SourceItemType"],
 "InvoicePayment"=>[],
 "Payment"=>[],
 "PaymentMethod"=>["AchCity", "AchCountry", "AchPostalCode", "AchState"],
 "Product"=>["ProductNumber"],
 "ProductRatePlan"=>[],
 "ProductRatePlanCharge"=>
  ["ApplyToBillingPeriodPartially",
   "CentralizedPrice",
   "CommitmentType",
   "CreditOption",
   "DiscountClass",
   "DiscountClassId",
   "DrawdownRate",
   "DrawdownUom",
   "ExcludeItemBillingFromRevenueAccounting",
   "ExcludeItemBookingFromRevenueAccounting",
   "Formula",
   "IsAllocationEligible",
   "IsCommitted",
   "IsPrepaid",
   "IsRollover",
   "IsStackedDiscount",
   "IsUnbilled",
   "POBPolicy",
   "PrepaidOperationType",
   "PrepaidQuantity",
   "PrepaidTotalQuantity",
   "PrepaidUom",
   "PriceUpsellQuantityStacked",
   "ProductCategory",
   "ProductClass",
   "ProductFamily",
   "ProductLine",
   "ProductRatePlanChargeNumber",
   "ProrationOption",
   "RolloverApply",
   "RolloverPeriodLength",
   "RolloverPeriods",
   "SpecificListPriceBase",
   "SplitUpsellQuantity",
   "ValidityPeriodType"],
 "ProductRatePlanChargeTier"=>["Active", "DiscountAmount", "DiscountPercentage", "IncludedUnits", "OveragePrice"],
 "RatePlan"=>["IsFromExternalCatalog", "OriginalRatePlanId", "ProductRatePlanNumber", "SubscriptionRatePlanNumber"],
 "RatePlanCharge"=>
  ["AmendedByOrderOn",
   "ApplyToBillingPeriodPartially",
   "CommitmentType",
   "CreditOption",
   "DiscountAmount",
   "DiscountClass",
   "DiscountPercentage",
   "DrawdownRate",
   "DrawdownUom",
   "ExcludeItemBillingFromRevenueAccounting",
   "ExcludeItemBookingFromRevenueAccounting",
   "IncludedUnits",
   "IsAllocationEligible",
   "IsCommitted",
   "IsPrepaid",
   "IsProcessed",
   "IsRollover",
   "IsStackedDiscount",
   "IsUnbilled",
   "NumberOfDeliveries",
   "OriginalListPrice",
   "OriginalOrderDate",
   "OveragePrice",
   "POBPolicy",
   "PaymentTermSnapshot",
   "PrepaidCommittedAmount",
   "PrepaidOperationType",
   "PrepaidQuantity",
   "PrepaidTotalQuantity",
   "PrepaidUom",
   "PriceUpsellQuantityStacked",
   "ProductCategory",
   "ProductClass",
   "ProductFamily",
   "ProductLine",
   "ProductRatePlanChargeNumber",
   "ProrationOption",
   "RevenueRecognitionRuleName",
   "RolloverApply",
   "RolloverPeriodLength",
   "RolloverPeriods",
   "SpecificListPriceBase",
   "ValidityPeriodType"],
 "RatePlanChargeTier"=>["Currency", "DiscountAmount", "DiscountPercentage", "IncludedUnits", "OveragePrice"],
 "Subscription"=>["Currency", "InvoiceTemplateId", "IsLatestVersion", "LastBookingDate", "PaymentTerm", "Revision", "SequenceSetId", "SubscriptionBillToId", "SubscriptionBillToSnapshotId", "SubscriptionSoldToId", "SubscriptionSoldToSnapshotId"],
 "TaxationItem"=>["Balance", "Country", "CreditAmount", "CustomerCode", "ExemptCertificate", "PaymentAmount", "SellerRegistration", "TaxAmountUnrounded", "TaxDescription", "TaxRuleId"],
 "Usage"=>["UniqueKey"]}

[kpopko]

@jurisgalang I deployed the custom branch in this https://github.com/zendesk/billing/pull/16354 to billing 999 and verified export tasks works:

The result of first run of export tasks: https://github.com/zendesk/billing/pull/16356, also verified inside ric 999
The result of second run of export tasks: https://github.com/zendesk/billing/pull/16361, some adjustments only to Light Agent rate plan