feat(BA-96): metric based model service autoscaling

[kyujin-cho]

Resolves #2659 (BA-96)

What's changed

• endpoint_auto_scaling_rules table added
• implemented new scheduling logic under scale_services() which automatically increases or decreases EndpointRow.replicas factors based on the rules defined
• Separated aiodataloader handler and actual bulk loading mechanism from both EndpointStatistics and KernelStatistics so that bulk loader can be reused internally

How it works

• Every endpoint (model service) can have one or more auto scaling rules
• Auto scaling rule is defined as:
  • Metric source - inference runtime or kernel
    • inference framework: average value taken from every replicas. Supported only if both AppProxy reports the inference metrics. Check Backend.AI Enterprise guide for more details.
    • kernel: average value taken from every kernels backing the endpoint
  • Metric name (e.g. cuda.shares or vllm_avg_prompt_throughput_toks_per_s)
  • Comparator - method to compare live metrics with threshold value.
    • LESS_THAN: Rule triggered when current metric value goes below the threshold defined
    • LESS_THAN_OR_EQUAL: Rule triggered when current metric value goes below or equals the threshold defined
    • GREATER_THAN: Rule triggered when current metric value goes above the threshold defined
    • GREATER_THAN_OR_EQUAL: Rule triggered when current metric value goes above or equals the threshold defined
  • Step size: size of step of the replica count to be changed when rule is triggered. Can be represented as both positive and negative value - when defined as negative, the rule will decrease number of replicas.
  • Cooldown Seconds: Durations in seconds to skip reapplying the rule right after rule is first triggered.
  • Minimum Replicas: Sets a minimum value for the replica count of the endpoint. Rule will not be triggered if the potential replica count gets below this value.
  • Maximum Replicas: Sets a maximum value for the replica count of the endpoint. Rule will not be triggered if the potential replica count gets above this value.

Checklist: (if applicable)

• Milestone metadata specifying the target backport version
• Mention to the original issue

---

📚 Documentation preview 📚: https://sorna--3277.org.readthedocs.build/en/3277/

---

📚 Documentation preview 📚: https://sorna-ko--3277.org.readthedocs.build/ko/3277/

[achimnol]

As @yomybaby has requested, please update the GraphQL schema to use explicit Enum types for AutoScalingMetricSource and AutoScalingMetricComparator.
Also, currently the GraphQL query results are using field names, not values despite 913569e. Please fix this as well.

[achimnol]

There are some async-database handling issues as well.

[kyujin-cho]

Well, seems like graphene's Enum type expects enum names (not values) as a field input... :(

[HyeockJinKim]

Could you review it again? @achimnol