NT support

[mattklein123]

Get Envoy working on NT. This will require OS shim layer most/all POSIX operations, all Linux specific operations, and especially a new hot restart implementation.

[kavyako]

Are there any plans for adding Windows support? If yes, any timeline..

[mattklein123]

@kavyako no one is working on this. If we can find resources would love to do it. Happy to chat offline if MSFT is interested.

[vturecek]

@mattklein123 yeah we'd be interested to see what's possible. Send me your contact info and I'll set up some time to chat. My email is vturecek@microsoft.com. Cheers!

[mattklein123]

@vturecek sweet. Will send email. FYI all Lyft people working on Envoy are based in Seattle so we can meet in person if needed.

[mhoran]

Hey @vturecek, @mattklein123. Has anyone started work on Windows support? We're looking to use Envoy in Cloud Foundry to resolve the issue of Route Integrity, and my team in particular is concerned with how we'll do this on Windows. The Linux proposal involves Envoy, and we'd love to use the same solution on Windows, if possible.

We'd be interesting in collaborating in whatever way makes sense. If email communication is preferred, I can be reached at mhoran@pivotal.io.

[mattklein123]

@mhoran I will start a mail thread with MSFT folks to determine status.

[vturecek]

Thanks @mattklein123. Just in case there's more interest from others, yes Windows work is underway on our end. @mhoran - I'll follow up with more details over email.

[campbelldgunn]

@vturecek Any status from a MS point of view. We are running a mixed cluster (linux pool - linux nodes and windows pool - windows nodes) and would like holistic support for this environment.

[shalako]

@mhoran Just discovered this fork and thinks it might work
https://github.com/microsoft/envoy/tree/sridhar/win32_port

but it appears there is still work necessary to get it working with the required build system...
https://github.com/Microsoft/envoy/tree/sridhar/win32_port/win32_build

[kavyako]

@sridmad : for more details about the windows port.

[vturecek]

@campbelldgunn sorry for the late response - Windows support work is underway in our fork here: https://github.com/microsoft/envoy/

@sridmad has a branch with what I believe is a functional build at this point but I'll defer to him for the status on that: https://github.com/microsoft/envoy/tree/sridhar/win32_port

[vinayatgit]

Great to see work being done towards adding windows support to envoy. I am curious to understand how traffic redirection is handled on windows platform... basically windows equivalent for linux iptables.

[mhoran]

@vinayatgit we've come up with a proof of concept that leverages Windows proxy settings to force application egress through an Envoy proxy. Our investigation was successful, though we encountered issues with the documented APIs for configuring proxy settings in Windows Server 1803. More details here: https://www.pivotaltracker.com/story/show/158328675.

Of course, this only works for HTTP (and potentially HTTPS) traffic. There's still a question around how to handle TCP traffic, but that's something we can certainly add later.

[sesmith177]

We have been able to successfully compile envoy on Windows using Bazel. A branch with code changes + build system changes can be found here: https://github.com/greenhouse-org/envoy/tree/windows-linux-build

The necessary C++ changes were mostly pulled over from the work @sridmad has done in the Microsoft fork.

We discussed with @mattklein123 last week what the strategy should be to start getting these changes back upstream. The first step we will make is to PR just the build changes (i.e *.bzl, BUILD files etc.) and go from there to get the correct C++ changes in

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[mattklein123]

FYI I have reached out to MSFT also to see if we can get some help with this. cc @brendandburns

[mhoran]

Thanks @mattklein123. I spoke with the Microsoft team back at MS Build in May and at the time there were no Microsoft developers allocated to the effort. We had discussed next steps but that is stalled until we have someone to work with on that side. Meanwhile, we're still moving forward with the port. The goal was to get Microsoft folks to help with the heavy lifting in the sockets implementation. cc @vturecek @sridmad

[tapika]

Do you have any even draft specs how envoy could be built for windows ?

Trying at the moment with "choco install bazel" +
"C:\tools\msys64\msys2_shell.cmd" > "pacman -S base-devel --needed"

Last time failed without base-devel, but sense there will be more problems after this.

Is it also possible to generate Visual Studio projects / solutions for envoy ?

Would be interested to check if it's possible to strip off HTTP (as a server) to GPRC transcoder code without rest of libraries.

[tapika]

command like this:

bazel build --cxxopt=/wd4819 --cxxopt=/W2 --cxxopt=/wd4624 --cxxopt=/wd4244 --cxxopt=/wd4200 --cxxopt=/wd4267 --cxxopt=/wd4309 --cxxopt=/wd4838 //source/exe:envoy-static

fails with error:

ERROR: Skipping '//source/exe:envoy-static': no such package '@com_google_protobuf//': Traceback (most recent call last): File "C:/users/user/_bazel_user/3jojcmei/external/bazel_tools/tools/build_defs/repo/http.bzl", line 56 patch(ctx) File "C:/users/user/_bazel_user/3jojcmei/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 91, in patch fail(("Error applying patch %s:\n%s%s...))) Error applying patch @envoy//bazel:protobuf.patch: /bin/bash: C:/users/user/_bazel_user/3jojcmei/external/envoy/bazel/protobuf.patch: No such file or directory WARNING: Target pattern parsing failed. ERROR: no such package '@com_google_protobuf//': Traceback (most recent call last): File "C:/users/user/_bazel_user/3jojcmei/external/bazel_tools/tools/build_defs/repo/http.bzl", line 56 patch(ctx) File "C:/users/user/_bazel_user/3jojcmei/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 91, in patch fail(("Error applying patch %s:\n%s%s...))) Error applying patch @envoy//bazel:protobuf.patch: /bin/bash: C:/users/user/_bazel_user/3jojcmei/external/envoy/bazel/protobuf.patch: No such file or directory INFO: Elapsed time: 767.533s INFO: 0 processes. FAILED: Build did NOT complete successfully (0 packages loaded) currently loading: source/exe Fetching @com_envoyproxy_protoc_gen_validate; fetching 434s

[achasveachas]

@tapika our latest work is in this branch, but it's pretty hacky and we are nowhere near a working build.

[wrowe]

@tapika your error is consistent with using a windows cr/lf checkout, we are checking out envoy
in unix file mode ('\n' line endings) and it doesn't have an issue applying patches. You need to be
consistent about the fixed line ending tags checked out from git vs the flavor of the patch utility in use.

[coreprocess]

I just wanted to document my recent effort to produce a Windows build unsuccessfully. I tried two approaches:

1. Building with MSVC (compiler) and MSys (autoconf) fails. The code contains references to Unix specific socket headers, which are not available on Windows.
2. Building with Cygwin fails, as Abseil prevents builds on Cygwin deliberately.

During my trial and error phase, I identified two shortcomings within Bazel concerning Cygwin. I addressed both with pull requests.

1. https://github.com/core-process/rules_foreign_cc/commit/1f4fadbd866106a4b6ed9403205f7ea345f813e9 (already merged by maintainer, see Improve cygwin compatibility bazel-contrib/rules_foreign_cc#323)
2. https://github.com/core-process/rules_cc/commit/5a1f5f7d5e051082fa1f5f7d8dbb2af6cff80dd1

The second issue can be addressed via a toolchain configuration alternatively in the envoy project itself. Here is the source code:

package(default_visibility = ['//visibility:public'])

load(":windows_cc_toolchain_config.bzl", "cc_toolchain_config")

filegroup(
    name = "empty",
    srcs = [],
)

filegroup(
    name = "cygwin_compiler_files",
    srcs = [":builtin_include_directory_paths_mingw"]
)

cc_toolchain_suite(
    name = "cygwin",
    toolchains = {
        "x64_windows|cygwin-gcc": ":cc-compiler-x64_windows_cygwin",
        "x64_windows": ":cc-compiler-x64_windows_cygwin",
    },
)

cc_toolchain(
    name = "cc-compiler-x64_windows_cygwin",
    toolchain_identifier = "cygwin_x64",
    toolchain_config = ":cygwin_x64",
    all_files = ":empty",
    ar_files = ":empty",
    as_files = ":cygwin_compiler_files",
    compiler_files = ":cygwin_compiler_files",
    dwp_files = ":empty",
    linker_files = ":empty",
    objcopy_files = ":empty",
    strip_files = ":empty",
    supports_param_files = 1,
)

cc_toolchain_config(
    name = "cygwin_x64",
    cpu = "x64_windows",
    compiler = "cygwin-gcc",
    host_system_name = "local",
    target_system_name = "local",
    target_libc = "cygwin",
    abi_version = "local",
    abi_libc_version = "local",
    cxx_builtin_include_directories = [
        "C:/tools/cygwin/",
    ],
    tool_paths = {
        "ar": "C:/tools/cygwin/bin/ar.exe",
        "compat-ld": "C:/tools/cygwin/bin/compat-ld.exe",
        "cpp": "C:/tools/cygwin/bin/cpp.exe",
        "dwp": "C:/tools/cygwin/bin/dwp.exe",
        "gcc": "C:/tools/cygwin/bin/gcc.exe",
        "gcov": "C:/tools/cygwin/bin/gcov.exe",
        "ld": "C:/tools/cygwin/bin/ld.exe",
        "nm": "C:/tools/cygwin/bin/nm.exe",
        "objcopy": "C:/tools/cygwin/bin/objcopy.exe",
        "objdump": "C:/tools/cygwin/bin/objdump.exe",
        "strip": "C:/tools/cygwin/bin/strip.exe"
    },
    tool_bin_path = "C:/tools/cygwin/bin",
    dbg_mode_debug_flag = "/DEBUG:FULL",
    fastbuild_mode_debug_flag = "/DEBUG:FASTLINK",
)

toolchain(
    name = "cc-toolchain-x64_windows_cygwin",
    exec_compatible_with = [
        "@platforms//cpu:x86_64",
        "@platforms//os:windows",
        "@bazel_tools//tools/cpp:cygwin",
    ],
    target_compatible_with = [
        "@platforms//cpu:x86_64",
        "@platforms//os:windows",
    ],
    toolchain = ":cc-compiler-x64_windows_cygwin",
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
)

The referenced file windows_cc_toolchain_config.bzl can be copied from bazelbuild/rules_cc.

Here is my build pipeline regarding Cygwin: https://github.com/core-process/envoy-build/blob/master/.github/workflows/main.yaml#L65-L109

You will find an environment setup for MSVC and MSys commented out in the same file.

[mattklein123]

For everyone watching this issue, FYI that Microsoft has committed resources to seeing this port complete. They are going to reach out to the Pivotal folks to help out. cc @mhoran @achasveachas @wrowe. I'm also going to meet with Microsoft in a couple of weeks in-person about this as well.

[aktxyz]

any new info/status from recent microsoft/pivotal meetings?

[wrowe]

There will be more news to share in coming weeks and months from different teams of engineers, nothing I can helpfully share. I can update you that my team at Pivotal has resolved PR 8280, proposed PR 8572, and we are preparing the necessary patches in the near-term for filesystem/socket objects on Windows so that the code will compile for anyone interested in participating in development.

[achasveachas]

@aktxyz We did have two productive meetings with the folks from Microsoft and they committed to helping us push the Windows port through.

We are still hammering out the details, we will probably have more solid information to share at EnvoyCon/KubeCon in 2 weeks.

[RamjotSingh]

As part of Windows port are there plans to support the FIPS 140-2 mode? https://www.envoyproxy.io/docs/envoy/v1.10.0/intro/arch_overview/ssl#fips-140-2

[aktxyz]

Is this still the best place to check for status ?

[mhoran]

Yes! There was also an update given at KubeCon. There's also now an event on the CNCF calendar for our bi-weekly sync but it's not showing up publicly. I'll look into what's happening there.

[sunjayBhatia]

12/20/2019 Update:

Recent PRs merged:

• Fix win32 breakage 18 dec #9399
• Bump PGV to pick up utf-32 unicode len counting fix #9314
• Correct trie test by comparing string contents #9303
• Correct test of OptionsImpl argc type (Was: Correct type for std::array size() result) #9290
• Drop Chromium UTF8 references we have not imported #9281
• Correct stdc++ language and size issues for MSVC #8572

PRs in flight:

• Google Test simple_re doesn't support [0-9] patterns #9398
• Resolve most remaining Windows compilation quirks #9331

We have generally finished up the simple compilation issues in the current codebase. We will still have to fix any new breakages committed to the master branch and are considering additions to CI to prevent this occurrence. Our next steps are around upstreaming our changes to the OS sys calls package, file watcher, and networking package that we have on our compiling fork. We also will continue to attempt to enable more integration tests that rely on extensions.

[sunjayBhatia]

2/28/2020 Update:

Recent PRs merged:

• Fix assorted breakages and Windows compilation changes #9966
• Introduce os_fd_t and windows port of os_sys_calls_impl #10036
• Fix various win32 compile issues #9915
• Coalesce directory management to Envoy::TestEnvironment #9721
• Google Test simple_re doesn't support [0-9] patterns #9398
• Resolve most remaining Windows compilation quirks #9331

PRs in flight:

• Use os_sys_calls in all code paths to replace raw syscalls #10107

We have upstreamed our cross-platform changes to the OS sys calls package, file watcher, and many changes to the networking package (excluding things that touch Unix sockets/pipes). Most of our remaining changes are Windows specific as well as changes to build scripts/rules to enable Windows to compile. We are planning on submitting a Draft PR with our current work in progress so the community can start to work on top of that to help out with Window support. This Draft PR branch will be periodically revised/rebased on top of master as changes are accepted and revisions are made.

The next tracks of work we will tackle are enabling a static exe to be built in CI (though no tests yet, this involves build script/rule changes and Windows specific code changes) as well as improving error handling in our Windows implementation to better handle the differences and quirks between WSAE* socket errors and Posix socket errors.

[wrowe]

If you have forked from github.com/greenhouse-org/envoy as mentioned by Sam above; please note that we have archived the historical efforts to the greenhouse-org/envoy-archive static repo, and started a fresh fork from envoyproxy/envoy at that original repo name, with the vmware-windows-build and vmware-fix-upstream branches in place of pivotal-windows-build and pivotal-fix-upstream. This allows everyone to observe the source deltas to envoyproxy master, rather than the very stale microsoft/envoy fork.

Last week, we also went to the effort to include every extension which compiles on Windows. We next need to modify the test targets to look at the correct extension list, but this resulted in some dozen excluded extensions which do not compile, and some 60 failing tests. A handful of bug fixes are expected to pare that list back of failing tests down to a dozen or so. With much of the code already on master, and several more PR's to follow in the coming weeks, we will be seeking out individuals interested in helping with the heavy lift of hacking and debugging the problematic extensions for this port.

Developers can track activity and raise discussion on the envoyproxy.slack.com #envoy-windows-dev channel. When a generally available preview is ready for non-developer participation, we will then create a corresponding -users channel.

[sunjayBhatia]

We have an open PR here: #10293 that once merged will enable CI to build a static version of envoy on each PR

[sunjayBhatia]

4/24/2020 Update:

Merged PRs:

• Enable Windows CI //source/exe:envoy-static build #10293 - CI builds envoy-static.exe on Windows for each PR/commit
• Correct Windows build defects noted with recent version updates #10469
• Windows compilation: enable compiling expanded list of extensions in envoy-static #10542 - All except 4 extensions are now compiled into Envoy on Windows, windows/.bazelrc merged into .bazelrc
• Shorten adaptive_concurrency/concurrency_controller paths #10560
• Windows compilation: ensure statusCodeToString has no control paths that do not return #10777
• Windows compilation: Ensure thrift_proxy tests/mocks do not fail to compile with missing Validate method on google::protobuf::Struct #10780
• docs: Required dependencies for building on Windows #10790 - Docs for building Envoy on Windows, feedback/PRs to update are appreciated
• Fix Windows compilation of test sources #10822 - Test sources compiling, not yet enabled in CI due to high resource utilization of test compilation (means we still have to intermittently fix compilation issues in test sources until we can get CI enabled)
• ci: Enable remote cache in Windows CI #10831 - Remote cache enabled in CI for faster builds
• Windows compilation: Remove inclusion of pthread.h #10895

Open Issues:

• Windows RBE Tracking Issue #10619 - Tracking work to enable RBE on Windows, we've gotten good feedback from the maintainers of the RBE toolchain generation tools for bazel that our changes to enable toolchain generation on Windows will be accepted
• Replace pipe() with os_sys_calls_impl::socketpair(AF_UNIX,...) in buffer tests #10871

Current Work:

• Fixing failing tests, tagging failing tests with fails_on_windows tag so we can get community help
• A few parallel ideas/tracks to get Windows tests compiling/running in CI:
  • Enabling remote execution with bazel + Google RBE service
  • Provision AZP CI workers with more resources, the default AZP provided 2 CPU 7 GB RAM is not enough
  • Dockerize our CI scripts to ensure we have a consistent/controlled CI environment to execute in

[sunjayBhatia]

PR that tags failing tests so we can get more community visibility: #10940

[daschott]

As @wrowe already mentioned above, I'd like to reiterate that the best way to track the effort of Envoy support on Windows OS and participate is to join the #envoy-windows-dev Slack channel in https://envoyslack.cncf.io/

There are weekly meetings with the working group giving updates on the incremental progress and development activities. Meeting notes + details are captured here.

[sunjayBhatia]

Also feel free to follow on any specific issues with the area/windows tag and PRs with Windows: in the title

[mattklein123]

Given that alpha was declared today, I'm going to call this issue closed. Let's track follow ups with area/windows as the team has indicated! Go team!

[wrowe]

For those following this activity moving forwards, watch

https://github.com/envoyproxy/envoy/milestone/16 Windows Beta milestone.

[davinci26]

Hello all,

We would like to declare 1.17+ a Beta release for Envoy on Windows. Since the Alpha release, we have made major advancements in Windows platform support.

Features in 1.17:

• Synthetic edge events, which makes our asynchronous I/O model way more efficient. Envoy is spinning CPU on Windows #13189
• Support for stop signals (Ctrl+C/Ctrl+Break). Process Control on Windows #13188
• Improve the reliability of Envoy on Windows. Windows is a required CI step and we have eliminated flaky Windows tests.

Features since 1.17:

• Added support for Clang compiler. Windows builds with clang-cl.exe #11974
• Added support for the original.dst filter. Support Original Destination Redirect support on Windows for service mesh scenario  #13955
• Experimental support for running Envoy under SCM. Process Control on Windows #13188

We encourage you to start testing your scenarios with Envoy on Windows based on release 1.17 or the head of the main branch.

We are always looking to improve so your feedback is highly appreciated. Feel free to reach out on the EnvoyProxy Slack channel #envoy-windows-dev or open a Github issue with suggestions or feedback.

cc @envoyproxy/windows-dev @pravb