Implement support for HTTP proxies to support corporate deployments

[jdegoes]

Is your feature request related to a problem? Please describe.

In corporate environments, the Coursier bootstrap script will fail, because the program will attempt to download the Coursier JAR without proxy settings, and there is no way to configure these.

The solution is not as simple as adding the deprecated http.proxyHost and http.proxyPort properties (and their SSL variations), because many proxies act as their own CA, and neither this CA nor the certificates they certify will be added to the key store, if indeed a key store is accessible in the environment in which Coursier is running.

Describe the solution you'd like

The ideal solution would be to fix Coursier deeply, so that it would use a real proxy (not rely on HttpURLConnection's flaky use of deprecated environment variables), and prompt user to accept certificates as necessary into a key store with write permissions, probably in the same place where metals is installed (as that's the only place you can guarantee has write access; the Java directory will be locked down in many corporate environments).

Describe alternatives you've considered

Metals could at least provide some configuration options for proxy information, which it would then forward to the bootstrap script (right now the JAR is started via the script, and there's no direct way to pass it options; one can try JAVA_TOOL_OPTIONS but it's unreliable and requires the ability to edit environment variables, which is disabled in many corporate environments). In particular, the fields should include http / https server and port, as well as username and password (many proxies require authentication), and whether or not to debug SSL connections; and the key store location.

Additional context

NDAs prevent me from including logs or further details.

Search terms:

HTTPS SSL proxies corporate firewalls IT proxy authentication.

[alexarchambault]

@jdegoes coursier is not tied to java.net.HttpURLConnection, it already can be swapped for okhttp in particular (although it's not loudly advertised yet).

How do you handle those proxies from sbt?

[alexarchambault]

People talk about a javax.net.ssl.trustStore Java property here or there, that seems to be made for that use case. Did you try setting it here?

[olafurpg]

Thanks for reporting! The current solution for HTTP proxies is that users can update the "server properties" setting in VS Code to include -Dhttps.proxyHost=… -Dhttps.proxyPort=…. The VS Code extension automatically picks up these settings if they are declared in .jvmopts.

If -Dhttps.proxyPort is not sufficient then we could add support for an "offline" mode where users could bring their own jars and list them in a JSON file .metals/offline.json that would look something like this

{
  "metals": "$CLASSPATH", // (required for VS Code)
  "bloop": "$CLASSPATH", // (optional)
  "scalafmt": "$CLASSPATH" // (optional)
}

If offline.json exists then Metals would not do any HTTP requests. We could provide scripts to generate offline.json either via sbt or coursier.

[SIkebe]

Any progress here? This issue is because coursier does not support proxy authentication?
coursier/coursier#930

[tgodzik]

@SIkebe The proxy authentication is supported as far as I know, a number of additional properties need to be setup as explained above. There are some improvements that can be made in the area like:

• swap java.net.HttpURLConnection for okhttp in coursier
• add offline mode for some more complex corporate setup

There is some more information about the current state here: https://scalameta.org/metals/docs/build-tools/proxy.html

We would be happy to help out however we can if there are still problems, however we do lack some information what exactly doesn't work currently and we do not have anyone able to spend time on the improvements.

[yelled1]

Actually, there are corporate machines that allows no connection to www. Would rather see just copying a cache or directory type of solution. And, I think I made this work to most part.
See above Running inside AWS Glue endpoint machine w/o internet #1802
I still get error when I start VS Code (who cares), but do get auto completion. Yet, I am not quite sure I am getting the full benefits of Metals. Pls let me know if above helps & do mention what I may have missed.

[olafurpg]

Metals already provides a custom main method to pre-populate caches that you can launch with Coursier like this

cs launch org.scalameta:metals_2.12:VERSION --main scala.meta.metals.DownloadDependencies

Users with restricted network access can for example use this command to build a docker image that can run Metals offline. Note that this command only populates the Metals caches, you would need to additionally cache outputs from the build (for example install sbt-bloop and run sbt bloopInstall).

[yelled1]

@olafurpg now I just got jolted to awaken to what @tgodzik said on the other post.
Thank you much both,

[olafurpg]

I'm gonna close this issue as resolved since there exists now

• better documentation for working behind proxies https://scalameta.org/metals/docs/build-tools/proxy.html
• a scala.meta.metals.DownloadDependencies main method that users can run to pre-populate Metals caches, which can be used as a step to build a docker image to run Metals in an offline mode

Please open a separate issue if the existing options don't work for you.

[jneira]

hi, https://scalameta.org/metals/docs/build-tools/proxy.html is broken now
i guess the new url would be https://scalameta.org/metals/docs/troubleshooting/proxy/#proxy-settings or https://scalameta.org/metals/docs/editors/vscode/#http-proxy