Difficulty deploying golem with shinyproxy on AWS EC2

[yogat3ch]

Hi @ColinFay & @VincentGuyader,
We're currently in the process of deploying a golem to an existing working shinyproxy server on an EC2 instance. The server currently has other iterations of this application, pre-golem working on it.
We've built the docker image for the app and tested it. It's functioning as intended locally. I've included the Dockerfile contents below.

Dockerfile

``` FROM rocker/r-ver:4.1.3 ARG GITHUB_PAT # Only once before non CRAN remotes RUN GITHUB_PAT=$GITHUB_PAT RUN apt-get update && apt-get install -y g++ curl cmake libzmq3-dev git-core libcurl4-openssl-dev libgit2-dev libicu-dev libssl-dev libxml2-dev make pandoc pandoc-citeproc zlib1g-dev && rm -rf /var/lib/apt/lists/* RUN echo "options(repos = c(CRAN = 'https://cran.rstudio.com/'), download.file.method = 'libcurl', Ncpus = 4)" >> /usr/local/lib/R/etc/Rprofile.site RUN R -e 'install.packages("remotes")' RUN Rscript -e 'remotes::install_version("cli",upgrade="never", version = "3.3.0")' RUN Rscript -e 'remotes::install_version("rlang",upgrade="never", version = "1.0.3")' RUN Rscript -e 'remotes::install_version("magrittr",upgrade="never", version = "2.0.3")' RUN Rscript -e 'remotes::install_version("tibble",upgrade="never", version = "3.1.7")' RUN Rscript -e 'remotes::install_version("curl",upgrade="never", version = "4.3.2")' RUN Rscript -e 'remotes::install_version("rstudioapi",upgrade="never", version = "0.13")' RUN Rscript -e 'remotes::install_version("fs",upgrade="never", version = "1.5.2")' RUN Rscript -e 'remotes::install_version("waldo",upgrade="never", version = "0.3.1")' RUN Rscript -e 'remotes::install_version("stringr",upgrade="never", version = "1.4.0")' RUN Rscript -e 'remotes::install_version("purrr",upgrade="never", version = "0.3.4")' RUN Rscript -e 'remotes::install_version("zip",upgrade="never", version = "2.2.0")' RUN Rscript -e 'remotes::install_version("htmltools",upgrade="never", version = "0.5.2")' RUN Rscript -e 'remotes::install_version("sass",upgrade="never", version = "0.4.0")' RUN Rscript -e 'remotes::install_version("shiny",upgrade="never", version = "1.7.1")' RUN Rscript -e 'remotes::install_version("tidyselect",upgrade="never", version = "1.1.2")' RUN Rscript -e 'remotes::install_version("dplyr",upgrade="never", version = "1.0.9")' RUN Rscript -e 'remotes::install_version("scales",upgrade="never", version = "1.1.1")' RUN Rscript -e 'remotes::install_version("data.table",upgrade="never", version = "1.14.2")' RUN Rscript -e 'remotes::install_version("tidyr",upgrade="never", version = "1.2.0")' RUN Rscript -e 'remotes::install_version("googledrive",upgrade="never", version = "2.0.0")' RUN Rscript -e 'install.packages(c("golem"))' RUN Rscript -e 'remotes::install_version("devtools",upgrade="never", version = "2.4.3")' RUN Rscript -e 'remotes::install_version("tippy",upgrade="never", version = "0.1.0")' RUN Rscript -e 'remotes::install_version("snakecase",upgrade="never", version = "0.11.0")' RUN Rscript -e 'remotes::install_version("shinyWidgets",upgrade="never", version = "0.6.3")' RUN Rscript -e 'remotes::install_version("shinydashboardPlus",upgrade="never", version = "2.0.3")' RUN Rscript -e 'remotes::install_version("shinycssloaders",upgrade="never", version = "1.0.0")' RUN Rscript -e 'remotes::install_version("readxl",upgrade="never", version = "1.3.1")' RUN Rscript -e 'remotes::install_version("readr",upgrade="never", version = "2.1.2")' RUN Rscript -e 'remotes::install_version("lubridate",upgrade="never", version = "1.8.0")' RUN Rscript -e 'remotes::install_version("DT",upgrade="never", version = "0.21")' RUN Rscript -e 'remotes::install_version("config",upgrade="never", version = "0.3.1")' RUN Rscript -e 'remotes::install_version("aws.s3",upgrade="never", version = "0.3.21")' RUN GITHUB_PAT=$GITHUB_PAT Rscript -e 'install.packages(c("arrow"))' RUN Rscript -e 'remotes::install_github("tidyverse/ggplot2@82c5459b880022dc39b4abb25c924c96e28f7e42")' RUN GITHUB_PAT=$GITHUB_PAT Rscript -e 'remotes::install_github("Martin-McCoy/virgaUtils@fbcf27ba90efd9786af3911886bf555e8b48a653")' RUN Rscript -e 'remotes::install_github("yogat3ch/UU@d285214ea61d95230118c735b5535e60e915d725")' RUN Rscript -e 'remotes::install_github("plotly/plotly@bb98fefcf5bd4ef5deb1b302e1e5412558fa8ad7")' RUN mkdir /build_zone ADD . /build_zone WORKDIR /build_zone RUN echo "options(repos = c(CRAN = 'https://cran.rstudio.com/'), download.file.method = 'libcurl', Ncpus = 4)" >> /usr/local/lib/R/etc/Rprofile.site RUN R -e 'install.packages("remotes")' RUN GITHUB_PAT=$GITHUB_PAT R -e 'remotes::install_local(upgrade="never")' RUN rm -rf /build_zone COPY Rprofile.site /usr/lib/R/etc/ EXPOSE 3838 CMD R -e "options(shiny.port=3838,shiny.host='0.0.0.0', golem.app.prod = TRUE);crssprViz::run_app()" # system(glue::glue("docker build --build-arg GITHUB_PAT={remotes:::github_pat()} -t crsspr_viz . --progress=plain 2> build.log")) # docker run -d -it --name crsspr_viz --rm -p 3838:3838 crsspr_viz ```

We uploaded our image to docker hub, albeit during a time when I was experiencing internet outages. Due to this, we just downloaded and tested the image that was uploaded to docker hub and it's verified working on a co-workers machine.

We pulled the docker image onto the shinyproxy instance and to each of the worker EC2 instances.
We configured the shinyproxy.yml file by adding another service in the Services section: that looks like the following:

 crssprVizTest:
    image: keatonwilson/crsspr_viz
    networks:
      - sp-net

We configured the application.yml file with the details for this image:

    - id: crssprVizTest
      display-name: CRSSPRVIZ Test
      container-cmd: [“R”, “-e”, “options(shiny.port=3838,shiny.host='0.0.0.0',golem.app.prod=TRUE);crssprViz::run_app()“]
      container-image: keatonwilson/crsspr_viz
      access-groups: [hyper_admins]
      container-network: sp-net

We pushed these updates to the docker service with sudo docker service update shinyproxy_shinyproxy --force.
We've added logging inside the containers with

proxy:
  container-log-path: ./container-logs

However, go by way of the shinyproxy and attempt to load the golem it times out and thus doesn't create any logs. Error code and stack trace below.

Error

Error
Status code: 500

Message: Container did not respond in time

Stack Trace:
eu.openanalytics.containerproxy.ContainerProxyException: Container did not respond in time
at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.startProxy(AbstractContainerBackend.java:118)
at eu.openanalytics.containerproxy.service.ProxyService.startProxy(ProxyService.java:213)
at eu.openanalytics.shinyproxy.controllers.AppController.getOrStart(AppController.java:109)
at eu.openanalytics.shinyproxy.controllers.AppController.startApp(AppController.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:870)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:776)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:881)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:855)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter.doFilterInternal(OAuth2AuthorizationRequestRedirectFilter.java:128)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:64)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:94)
at eu.openanalytics.containerproxy.util.ProxyMappingManager$ProxyPathHandler.handleRequest(ProxyMappingManager.java:159)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:336)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

We're running out of ideas as to what is causing the issue but we have a hunch that it has something to do with how shinyproxy is communicating with the container, possibly port mapping related, though we ensured that the port mapping was to 3838 for shinyproxy and that this port was exposed in the Dockerfile.

Do y'all have any deployments where you've encountered this? Any ideas as to what might be happening? We'll be crossposting this to shinyproxy shortly.

[yogat3ch]

@keatonwilson

[yogat3ch]

Update on this - the same error is manifesting on a local version of shinyproxy, ruling out any AWS EC2 related issues.

[yogat3ch]

Update: 2022-07-18 @keatonwilson determined that an update of shinyproxy for golem compatibility, as well as ensuring the architecture for our dependencies matched our EC2 instance architecture was needed to resolve this issue!