Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

URGENT ACTION: Stop using az416426.vo.msecnd.net #2457

Open
MSNev opened this issue Dec 12, 2024 · 8 comments
Open

URGENT ACTION: Stop using az416426.vo.msecnd.net #2457

MSNev opened this issue Dec 12, 2024 · 8 comments
Labels
cdn issue keep Do not Mark as Stale and close p1

Comments

@MSNev
Copy link
Collaborator

MSNev commented Dec 12, 2024

ACTION REQUIRED: Stop using az416426.vo.msecnd.net

To avoid any global OUTAGE you MUST change ALL of your CDN usage from “https://az416426.vo.msecnd.net/scripts/..” to our primary CDN endpoint https://js.monitor.azure.com/scripts/...

  • CDN Endpoint Change: All references to “az416426.vo.msecnd.net” must be updated to “js.monitor.azure.com” to prevent service disruptions due to a possible issue related to our current CDN configuration, this change MUST be done by Jan 15th, 2025.
  • We have identified some migration challenges with the legacy domain “az416426.vo.msecnd.net” which may lead to a temporary or permanent outages.
  • Action Required: Organizations must update their references and deploy changes to production before the service is disabled, and ensure that their sites can handle the absence of the Application Insights JS SDK from the CDN.

Right now we are expecting that there WILL be either a temporary or permanent outage of this domain based on the currently known details around the legacy domain.

What do you need to do?

  • Change ALL references of “az416426.vo.msecnd.net” to “js.monitor.azure.com”
  • Deploy the change to production before this external upcoming change occurs (we don’t control the timeframe)
  • Ensure that your site can handle when Application Insights JS Sdk cannot be loaded from the CDN without breaking your functionality

Details

Both the az416426.vo.msecnd.net and js.monitor.azure.com source their files from the same location as such all files and content is identical regardless of which domain endpoint you use.

Workarounds

  • Stop using “az416426.vo.msecnd.net” and change to “js.monitor.azure.com”
  • There is currently NO other workaround

We are currently investigating the available options on how we can avoid / migrate / mitigate this situation, but at this point it is HIGHLY likely that there will be either a temporary or permanent outage of this domain. As we currently have no known way to “migrate” this domain to a different CDN.

Any updates will be added to this issue.

@MSNev
Copy link
Collaborator Author

MSNev commented Dec 20, 2024

Why is this occurring?

Please see this public announcement Azure CDN from Edgio retirement FAQ where this domain (az416426.vo.msecnd.net is currently an Edgio based CDN instance) and because of the historical nature of this domain it's more complicated that most situations.

@baxxos
Copy link

baxxos commented Jan 3, 2025

Just to clarify - this affects only users who include the Application Insights JS SDK in their app by referencing the legacy CDN endpoint (e.g. using a script tag)?

@MSNev
Copy link
Collaborator Author

MSNev commented Jan 6, 2025

Correct, this issue ONLY affects users that are loading the SDK via the legacy CDN, if you are using the newer SDK Loader (snippet) which defaults to the current CDN endpoint (js.monitor.azure.com) or are using the SDK via an npm package embedded in your own bundles / domain then you are unaffected by this issue.

@MSNev
Copy link
Collaborator Author

MSNev commented Jan 6, 2025

Update: We have just started the process of redirecting this domain (hopefully) to avoid any possible outage, this change "should" take affect within the next couple of hours.

Known Issue (with Zero workaround -- ie. full outage)

  • The new CDN endpoint only support TLS 1.2+, so for any clients that do NOT support TLS 1.2 or greater the SSL connection will fail, we currently still support http on this endpoint.
  • Note: all of the other endpoints also only support TLS 1.2+ (which is why there is zero workaround (for TLS/SSL traffic)

Even if this is successful, this domain is now on an aggressive migration path and you should still continue to migrate away from this domain.

@MSNev
Copy link
Collaborator Author

MSNev commented Jan 7, 2025

The DNS (CNAME) entry has been redirected, and we appear to be seeing successful traffic being redirected to the new CDN.

However, the DNS Zone delegation still needs to be migrated, so there is still a risk of an outage...

@MSNev
Copy link
Collaborator Author

MSNev commented Jan 7, 2025

I've just "re-enabled" http access on this legacy endpoint (it had been defaulted to redirect all http -> https).

It appears that referrers that had been requesting http versions where successfully fetching the https version (as a follow up request from the same referrer)

@MSNev
Copy link
Collaborator Author

MSNev commented Jan 15, 2025

DNS Delegation has just been migrated and appears to be resolving as expected.
One more final step scheduled for tomorrow (PST) to complete the migration process.

@MSNev
Copy link
Collaborator Author

MSNev commented Jan 15, 2025

The final mitigation step is now complete, significantly reducing the risk of an outage for this domain due to the Edgio retirement.

However, this domain is now entering a more orderly deprecation phase, with exact timeframes to be determined as we move beyond this unexpected situation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cdn issue keep Do not Mark as Stale and close p1
Projects
None yet
Development

No branches or pull requests

2 participants