octavia-cli [Telemetry]: High MTU usage on Segment

[alafanechere]

Current Behavior

The telemetry built-in the CLI relies on Segment.
It appears that it's consuming too many Monthly Tracked Users (MTU):
In June 2022 we consumed 18K MTUs on the CLI. Which is closely matching the number of CLI command run.

But in June we had on octavia-cli:

• 42 unique user_id
• 5233 unique anonymous_id
  Which should correspond to ~5275 MTUs.

Expected Behavior

A single MTU is consume for each unique user id / anymous id.

Resource

What's an MTU and how it is tracked

[nataliekwong]

@rileybrook We just received a warning from Segment that we're close to exceeding our monthly threshold again, after doubling it last month. Looking at the different sources, it appears to be correctly calculating MTUs from sources like our website, backend and frontend but the CLI remains high (24k) compared to the actual number of users that are using it.

Could we prioritize looking into this to avoid continued overages from Segment?

[rileybrook]

@nataliekwong We can look at this during the week of Dec 5-9. Until then we are heads-down working towards shipping the Cloud API alpha on Dec 5th.

[JonsSpaghetti]

@nataliekwong my working theory of why this is currently happening is we might be creating a new anonymous user per CLI action.

Was curious though how you pulled the June numbers:

42 unique user_id
5233 unique anonymous_id
Which should correspond to ~5275 MTUs.

When I look at the graphs etc. in segment currently I see this:

Looking at the breakdown, it does look like the vast majority of our CLI traffic is anonymous, and that things do roughly add up to the total MTUs per month but want to make sure I'm not missing any context before pursuing a fix.

[nataliekwong]

@alafanechere pulled the original numbers and would have more context on how it was pulled. I think it might have been from BigQuery. Augustin also suggested at the time that maybe the problem is due to not having an identify call.

[nataliekwong]

Hey @JonsSpaghetti and @alafanechere - is there any additional context I can help provide here to resolve the MTU overcounting?

We just received another Segment overage bill for $1600 for the month of October, and I imagine that means their November invoice will be about the same (& future invoices until we resolve).

[JonsSpaghetti]

@nataliekwong I have a fix that I'm PR'ing now so I am hopeful we'll have this resolved within the week.

[nataliekwong]

Thank you! I'll monitor MTUs in Segment from here.