-
-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'release/v0.9.0' into main
- Loading branch information
Showing
20 changed files
with
423 additions
and
91 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,2 @@ | ||
| erlang 26.0.2 | ||
| elixir 1.15.4-otp-26 | ||
| erlang 26.2.1 | ||
| elixir 1.16.0-otp-26 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Ideas for implementing | ||
|
|
||
| ## Pool Metrics | ||
|
|
||
| I want to make a simple way to analyze running pools to set their optimal configuration. For example, we launched a `pool` in production with a maximum overflow of 0 (we do not want to create more processes than the designated number) and a pool size 200. | ||
|
|
||
| Using metrics, we see that typically, our application uses 10-20 processes, and there are spikes when up to 180 workers are exploited. If our processes are heavyweight and, for example, open persistent connections to storage, then by analyzing metrics, we can significantly save resources. In this case, we can set the pool size to 20 and `max_overflow` to 180. This way, we will have one overall pool size limit of 200, and we will avoid uncontrolled waste of all resources, but at the same time, we will only keep up to 20 processes in memory at times when this is not required. | ||
|
|
||
| ### Metrics to be implemented | ||
|
|
||
| - [x] Pool size | ||
| - [x] Idle workers count | ||
| - [x] Busy workers count | ||
| - [x] Is max_overflow used? | ||
| - [ ] Usage time | ||
| - [ ] How long are workers busy? | ||
| - [ ] How long the application waits of workers from pool? | ||
|
|
||
| ## Implementations metrics | ||
|
|
||
| To be described... |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # Working with metrics | ||
|
|
||
| You can quickly analyze and optimize your pool's production settings with the metrics presented by the library. | ||
|
|
||
| ## Pool size metrics | ||
|
|
||
| The Poolex library presents **an idle/busy worker count metric**. These metrics help estimate a pool load and the number of workers used. | ||
|
|
||
| Also, there is **an overflow metric**. It shows how long pools are forced to use additional workers. | ||
|
|
||
| You can handle them by using `:telemetry.attach/4`: | ||
|
|
||
| ```elixir | ||
| :telemetry.attach( | ||
| "my-lovely-pool-size-metrics", | ||
| [:poolex, :metrics, :pool_size], | ||
| &MyApp.handle_event/4, | ||
| nil | ||
| ) | ||
| ``` | ||
|
|
||
| For example, your application can write metrics to the console: [PoolexExample.MetricsHandler](https://github.com/general-CbIC/poolex/blob/develop/examples/poolex_example/lib/poolex_example/metrics_handler.ex). | ||
|
|
||
| More about using `telemetry` [here](https://hexdocs.pm/telemetry/readme.html). | ||
|
|
||
| ## Integration with PromEx | ||
|
|
||
| There is a plugin that works with the [PromEx](https://github.com/akoutmos/prom_ex) library: [Poolex.PromEx](https://hex.pm/packages/poolex_prom_ex). | ||
|
|
||
| About installation of this plugin you can read [here](https://hexdocs.pm/poolex_prom_ex/readme.html#installation). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
10 changes: 10 additions & 0 deletions
10
examples/poolex_example/lib/poolex_example/metrics_handler.ex
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| defmodule PoolexExample.MetricsHandler do | ||
| def handle_event([:poolex, :metrics, :pool_size], measurements, metadata, _config) do | ||
| IO.puts(""" | ||
| [Pool: #{metadata.pool_id}]: | ||
| - Idle workers: #{measurements.idle_workers_count} | ||
| - Busy workers: #{measurements.busy_workers_count} | ||
| - Overflowed: #{measurements.overflowed} | ||
| """) | ||
| end | ||
| end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,7 @@ | ||
| %{ | ||
| "dialyxir": {:hex, :dialyxir, "1.2.0", "58344b3e87c2e7095304c81a9ae65cb68b613e28340690dfe1a5597fd08dec37", [:mix], [{:erlex, ">= 0.2.6", [hex: :erlex, repo: "hexpm", optional: false]}], "hexpm", "61072136427a851674cab81762be4dbeae7679f85b1272b6d25c3a839aff8463"}, | ||
| "dialyxir": {:hex, :dialyxir, "1.4.3", "edd0124f358f0b9e95bfe53a9fcf806d615d8f838e2202a9f430d59566b6b53b", [:mix], [{:erlex, ">= 0.2.6", [hex: :erlex, repo: "hexpm", optional: false]}], "hexpm", "bf2cfb75cd5c5006bec30141b131663299c661a864ec7fbbc72dfa557487a986"}, | ||
| "erlex": {:hex, :erlex, "0.2.6", "c7987d15e899c7a2f34f5420d2a2ea0d659682c06ac607572df55a43753aa12e", [:mix], [], "hexpm", "2ed2e25711feb44d52b17d2780eabf998452f6efda104877a3881c2f8c0c0c75"}, | ||
| "poolex": {:hex, :poolex, "0.4.0", "cc3c5eb921a2ea5886953a2ea3beb0e36148d7d91dd9b628df74c3abb5ea7981", [:mix], [], "hexpm", "0441bf433f8fca4127b68c5985354cb772bba049057bca842f6e73df4f631c17"}, | ||
| "telemetry": {:hex, :telemetry, "1.2.1", "68fdfe8d8f05a8428483a97d7aab2f268aaff24b49e0f599faa091f1d4e7f61c", [:rebar3], [], "hexpm", "dad9ce9d8effc621708f99eac538ef1cbe05d6a874dd741de2e689c47feafed5"}, | ||
| "telemetry_poller": {:hex, :telemetry_poller, "1.0.0", "db91bb424e07f2bb6e73926fcafbfcbcb295f0193e0a00e825e589a0a47e8453", [:rebar3], [{:telemetry, "~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "b3a24eafd66c3f42da30fc3ca7dda1e9d546c12250a2d60d7b81d264fbec4f6e"}, | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| defmodule Poolex.Private.Metrics do | ||
| @moduledoc """ | ||
| Functions for dispatching metrics. | ||
| """ | ||
|
|
||
| @doc """ | ||
| Dispatches metrics with current count of idle workers. | ||
| """ | ||
| @spec dispatch_pool_size_metrics(Poolex.pool_id()) :: :ok | ||
| def dispatch_pool_size_metrics(pool_id) do | ||
| debug_info = Poolex.get_debug_info(pool_id) | ||
|
|
||
| :telemetry.execute( | ||
| [:poolex, :metrics, :pool_size], | ||
| %{ | ||
| idle_workers_count: debug_info.idle_workers_count, | ||
| busy_workers_count: debug_info.busy_workers_count, | ||
| overflowed: convert_overflow_to_number(debug_info.overflow > 0) | ||
| }, | ||
| %{pool_id: pool_id} | ||
| ) | ||
| end | ||
|
|
||
| @spec convert_overflow_to_number(boolean()) :: integer() | ||
| defp convert_overflow_to_number(true), do: 1 | ||
| defp convert_overflow_to_number(false), do: 0 | ||
|
|
||
| @doc """ | ||
| Starts a telemetry poller for dispatching metrics. | ||
| """ | ||
| @spec start_poller(list(Poolex.poolex_option())) :: GenServer.on_start() | ||
| def start_poller(opts) do | ||
| pool_id = Keyword.fetch!(opts, :pool_id) | ||
| measurements = collect_measurements(opts) | ||
|
|
||
| if measurements == [] do | ||
| :ok | ||
| else | ||
| :telemetry_poller.start_link( | ||
| measurements: measurements, | ||
| period: :timer.seconds(1), | ||
| name: :"#{pool_id}_metrics_poller" | ||
| ) | ||
| end | ||
| end | ||
|
|
||
| @spec collect_measurements(list(Poolex.poolex_option())) :: list() | ||
| defp collect_measurements(opts) do | ||
| pool_id = Keyword.fetch!(opts, :pool_id) | ||
|
|
||
| if Keyword.get(opts, :pool_size_metrics, false) do | ||
| [ | ||
| {Poolex.Private.Metrics, :dispatch_pool_size_metrics, [pool_id]} | ||
| ] | ||
| else | ||
| [] | ||
| end | ||
| end | ||
| end |
Oops, something went wrong.