Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fcrepo container not listening to port 80? #245

Closed
rbos opened this issue Apr 5, 2022 · 10 comments
Closed

fcrepo container not listening to port 80? #245

rbos opened this issue Apr 5, 2022 · 10 comments

Comments

@rbos
Copy link

rbos commented Apr 5, 2022

I updated from quite an old build, to 1.0.0-alpha-14, and then to -15 using the "make pull" method, and it went more or less smoothly, and I'm stuck on one issue: PDFs won't display. I also did a "git pull" against the latest isle-dc development branch to get the most recent docker-compose files.

  • A test item shows a blank space and "Missing PDF file[sic]". Expanding 'more information' shows Message: Missing PDF "https://[redacted]/_flysystem/fedora/2022-03/drupal-content-entity-8.0%20%281%29.pdf"
  • On the 'traefik' container, it shows time="2022-03-30T15:11:14Z" level=debug msg="'502 Bad Gateway' caused by: dial tcp 192.168.240.17:80: connect: connection refused"' every time I try to load the item. So I guess it wants to load the pdf from there via http.
  • 192.168.240.17 seems to be isle-dc_fcrepo_1. Going into that container and running netstat shows nothing listening on :80; just tomcat on :8080.
  • I modified a few .ymls as discussed in isle-dc issue #222 and then ran docker-compose down and docker-compose up on the theory that the fcrepo container no longer contains an nginx proxy, but that doesn't seem to fix it.

Any suggestions?

@mjordan
Copy link
Contributor

mjordan commented Apr 5, 2022

Adding to @rbos' findings, on this server it is also not possible to upload any "original files". Just tried with a PNG which resulted in the following message in the Drupal log:

The upload directory fedora://2022-04 for the file field field_media_image could not be created or is not accessible. A newly uploaded file could not be saved in this directory as a consequence, and the upload was canceled.

So, additional indication this is an issue with fcrepo's availability.

@noahwsmith
Copy link
Contributor

Oh, I wonder if this is Fedora needing to do the internal upgrade from F5 to F6, or similar. Could you provide the output from boot to crash (I'm assuming these symptoms mean the fedora container has crashed) for Fedora, via docker-compose logs -f fcrepo?

@rbos
Copy link
Author

rbos commented Apr 6, 2022

The fcrepo container is running, and there's a tomcat instance in it. No obvious errors that I saw.

fcrepo_1      | [s6-init] ensuring user provided files have correct perms...exited 0.
fcrepo_1      | [fix-attrs.d] applying ownership & permissions fixes...
fcrepo_1      | [fix-attrs.d] done.
fcrepo_1      | [cont-init.d] executing container initialization scripts...
fcrepo_1      | [cont-init.d] 00-container-environment-00-init.sh: executing... 
fcrepo_1      | [cont-init.d] 00-container-environment-00-init.sh: exited 0.
fcrepo_1      | [cont-init.d] 00-container-environment-01-override-tomcat.sh: executing... 
fcrepo_1      | [cont-init.d] 00-container-environment-01-override-tomcat.sh: exited 0.
fcrepo_1      | [cont-init.d] 00-container-environment-01-set-db-driver.sh: executing... 
fcrepo_1      | [cont-init.d] 00-container-environment-01-set-db-driver.sh: exited 0.
fcrepo_1      | [cont-init.d] 00-container-environment-02-database-defaults.sh: executing... 
fcrepo_1      | [cont-init.d] 00-container-environment-02-database-defaults.sh: exited 0.
fcrepo_1      | [cont-init.d] 01-confd-render-templates.sh: executing... 
fcrepo_1      | [cont-init.d] 01-confd-render-templates.sh: exited 0.
fcrepo_1      | [cont-init.d] 03-fcrepo-setup.sh: executing... 
fcrepo_1      | Waiting for up to 300 seconds to connect to Database mariadb:3306
fcrepo_1      | Waiting for 3306 on mariadb to open.
fcrepo_1      | Database found
fcrepo_1      | Validating Database credentials
fcrepo_1      | mysqld is alive
fcrepo_1      | Credentials are valid
fcrepo_1      | [cont-init.d] 03-fcrepo-setup.sh: exited 0.
fcrepo_1      | [cont-init.d] done.
fcrepo_1      | [services.d] starting services
fcrepo_1      | [services.d] done.
fcrepo_1      | NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
fcrepo_1      | NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
fcrepo_1      | 05-Apr-2022 21:57:10.155 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/9.0.58
fcrepo_1      | 05-Apr-2022 21:57:10.245 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Jan 15 2022 14:37:38 UTC
fcrepo_1      | 05-Apr-2022 21:57:10.246 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.58.0
fcrepo_1      | 05-Apr-2022 21:57:10.246 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
fcrepo_1      | 05-Apr-2022 21:57:10.246 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            5.4.181-1.el7.elrepo.x86_64
fcrepo_1      | 05-Apr-2022 21:57:10.246 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
fcrepo_1      | 05-Apr-2022 21:57:10.246 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/lib/jvm/java-11-openjdk
fcrepo_1      | 05-Apr-2022 21:57:10.247 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           11.0.14+9-alpine-r0
fcrepo_1      | 05-Apr-2022 21:57:10.247 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Alpine
fcrepo_1      | 05-Apr-2022 21:57:10.247 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /opt/tomcat
fcrepo_1      | 05-Apr-2022 21:57:10.247 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /opt/tomcat
fcrepo_1      | 05-Apr-2022 21:57:10.490 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
fcrepo_1      | 05-Apr-2022 21:57:10.490 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
fcrepo_1      | 05-Apr-2022 21:57:10.490 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
fcrepo_1      | 05-Apr-2022 21:57:10.490 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
fcrepo_1      | 05-Apr-2022 21:57:10.491 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
fcrepo_1      | 05-Apr-2022 21:57:10.491 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
fcrepo_1      | 05-Apr-2022 21:57:10.491 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
fcrepo_1      | 05-Apr-2022 21:57:10.504 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfcrepo.home=/data/home
fcrepo_1      | 05-Apr-2022 21:57:10.504 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfcrepo.velocity.runtime.log=/opt/tomcat/logs/velocity.log
fcrepo_1      | 05-Apr-2022 21:57:10.504 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfcrepo.jms.baseUrl=http://8736dedbef58/fcrepo/rest
fcrepo_1      | 05-Apr-2022 21:57:10.504 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfcrepo.external.content.allowed=/opt/tomcat/conf/allowed-external-content.txt
fcrepo_1      | 05-Apr-2022 21:57:10.504 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfcrepo.autoversioning.enabled=false
fcrepo_1      | 05-Apr-2022 21:57:10.504 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfcrepo.activemq.directory=file:///data/home/data/Activemq
fcrepo_1      | 05-Apr-2022 21:57:10.504 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfcrepo.activemq.configuration=file:///opt/tomcat/conf/activemq.xml
fcrepo_1      | 05-Apr-2022 21:57:10.505 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -DconnectionTimeout=-1
fcrepo_1      | 05-Apr-2022 21:57:10.514 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfcrepo.db.url=jdbc:mysql://mariadb:3306/fcrepo
fcrepo_1      | 05-Apr-2022 21:57:10.514 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfcrepo.db.user=fcrepo
fcrepo_1      | 05-Apr-2022 21:57:10.515 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfcrepo.db.password=PotatoSoupWithBaconAndLeeks
fcrepo_1      | 05-Apr-2022 21:57:10.515 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
fcrepo_1      | 05-Apr-2022 21:57:10.515 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/opt/tomcat
fcrepo_1      | 05-Apr-2022 21:57:10.515 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/opt/tomcat
fcrepo_1      | 05-Apr-2022 21:57:10.515 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/opt/tomcat/temp
fcrepo_1      | 05-Apr-2022 21:57:10.527 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/lib/jvm/java-11-openjdk/lib/server:/usr/lib/jvm/java-11-openjdk/lib:/usr/lib/jvm/java-11-openjdk/../lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
fcrepo_1      | 05-Apr-2022 21:57:13.770 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
fcrepo_1      | 05-Apr-2022 21:57:14.068 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [5829] milliseconds
fcrepo_1      | 05-Apr-2022 21:57:14.634 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
fcrepo_1      | 05-Apr-2022 21:57:14.634 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.58]
fcrepo_1      | 05-Apr-2022 21:57:14.673 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/opt/tomcat/webapps/ROOT]
fcrepo_1      | 05-Apr-2022 21:57:18.350 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
fcrepo_1      | 05-Apr-2022 21:57:18.594 WARNING [main] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [125] milliseconds.
fcrepo_1      | 05-Apr-2022 21:57:18.835 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/tomcat/webapps/ROOT] has finished in [4,162] ms
fcrepo_1      | 05-Apr-2022 21:57:18.835 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/opt/tomcat/webapps/host-manager]
fcrepo_1      | 05-Apr-2022 21:57:20.502 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
fcrepo_1      | 05-Apr-2022 21:57:20.544 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/tomcat/webapps/host-manager] has finished in [1,708] ms
fcrepo_1      | 05-Apr-2022 21:57:20.544 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/opt/tomcat/webapps/manager]
fcrepo_1      | 05-Apr-2022 21:57:21.770 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
fcrepo_1      | 05-Apr-2022 21:57:21.782 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/tomcat/webapps/manager] has finished in [1,238] ms
fcrepo_1      | 05-Apr-2022 21:57:21.783 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/opt/tomcat/webapps/fcrepo]
fcrepo_1      | 05-Apr-2022 21:57:45.512 WARNING [main] org.apache.catalina.startup.ContextConfig.validateSecurityRoles Security role name [fedoraUser] used in an <auth-constraint> without being defined in a <security-role>
fcrepo_1      | 05-Apr-2022 21:57:45.512 WARNING [main] org.apache.catalina.startup.ContextConfig.validateSecurityRoles Security role name [fedoraAdmin] used in an <auth-constraint> without being defined in a <security-role>
fcrepo_1      | 05-Apr-2022 21:57:45.571 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
fcrepo_1      | 05-Apr-2022 21:57:46.038 INFO [main] org.apache.catalina.core.ApplicationContext.log 1 Spring WebApplicationInitializers detected on classpath
fcrepo_1      | 05-Apr-2022 21:57:46.065 INFO [main] org.apache.catalina.core.ApplicationContext.log Initializing Shiro environment
fcrepo_1      | 05-Apr-2022 21:57:46.084 INFO [main] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext
fcrepo_1      | INFO 21:57:47.976 [main] (FedoraPropsConfig) Fedora home: /data/home
fcrepo_1      | INFO 21:57:48.632 [main] (OcflPropsConfig) Fedora storage type: ocfl-fs
fcrepo_1      | INFO 21:57:48.632 [main] (OcflPropsConfig) Fedora staging: /data/home/data/staging
fcrepo_1      | INFO 21:57:48.632 [main] (OcflPropsConfig) Fedora OCFL temp: /data/home/data/ocfl-temp
fcrepo_1      | INFO 21:57:48.632 [main] (OcflPropsConfig) Fedora OCFL reindexing threads: 3
fcrepo_1      | INFO 21:57:48.632 [main] (OcflPropsConfig) Fedora OCFL reindexing batch size: 100
fcrepo_1      | INFO 21:57:48.632 [main] (OcflPropsConfig) Fedora OCFL reindexing fail on error: true
fcrepo_1      | INFO 21:57:48.633 [main] (OcflPropsConfig) Fedora OCFL root: /data/home/data/ocfl-root
fcrepo_1      | INFO 21:57:48.633 [main] (OcflPropsConfig) Fedora OCFL digest algorithm: SHA-512
fcrepo_1      | INFO 21:57:48.695 [main] (DatabaseConfig) JDBC URL: jdbc:mysql://mariadb:3306/fcrepo
fcrepo_1      | INFO 21:57:48.695 [main] (DatabaseConfig) JDBC User: fcrepo
fcrepo_1      | INFO 21:57:48.695 [main] (DatabaseConfig) JDBC Password length: 27
fcrepo_1      | INFO 21:57:48.695 [main] (DatabaseConfig) Using database driver: com.mysql.cj.jdbc.Driver
fcrepo_1      | WARN 21:57:49.184 [main] (Database) Flyway upgrade recommended: MariaDB 10.6 is newer than this version of Flyway and support has not been tested. The latest supported version of MariaDB is 10.5.
fcrepo_1      | INFO 21:57:50.220 [main] (JmsConfig) JMS messaging enabled
fcrepo_1      | INFO 21:57:50.309 [main] (AuthConfig) Auth delegate principal provider enabled
fcrepo_1      | INFO 21:57:52.346 [main] (ExternalContentPathValidator) Loading list of allowed external content locations from /opt/tomcat/conf/allowed-external-content.txt
fcrepo_1      | INFO 21:57:52.386 [main] (RepositoryInitializer) Initializing repository
fcrepo_1      | 05-Apr-2022 21:57:52.530 SEVERE [main] org.apache.tomcat.util.descriptor.web.SecurityConstraint.findUncoveredHttpMethods For security constraints with URL pattern [/*] only the HTTP methods [HEAD DELETE POST GET OPTIONS PUT PATCH] are covered. All other methods are uncovered.
fcrepo_1      | 05-Apr-2022 21:57:54.240 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/tomcat/webapps/fcrepo] has finished in [32,457] ms
fcrepo_1      | 05-Apr-2022 21:57:54.245 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
fcrepo_1      | 05-Apr-2022 21:57:54.265 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [40196] milliseconds
fcrepo_1      | INFO 22:03:52.388 [ScheduledTask1] (TransactionImpl) Rolling back transaction a5218e03-3197-42d8-9a93-c05a5853c894

@noahwsmith
Copy link
Contributor

Hmm, so is it a permissions problem?

My advice is to spin up a new instance of ISLE on a dev box and compare the resulting docker-compose.yml file as well as the permissions within the Fedora datastore to the one in the older app.

@rbos
Copy link
Author

rbos commented Apr 6, 2022

I guess it could be? But the error message makes me think it's trying to access :80 on fcrepo, but there's no service on there listening to :80. So either something is misconfigured to point at the incorrect port, or there should be a service on fcrepo that proxies to tomcat.

I'll build a fresh .env from the latest sample.env on our dev server and see what the docker-compose looks like, I guess.

@DonRichards
Copy link
Member

Here's a list of things to check to save some time and effort.

Fedora needs to be exposed (EXPOSE_FEDORA=true) and this is where you set the port. Make this change in your .env file then make down/pull/up

/admin/config/media/file-system
change "Default download method" to fedora

If you're checking via pdf viewer you might need to update
/admin/config/media/pdfjs
Add path as "/libraries/pdf.js/web/viewer.html"

@mjordan
Copy link
Contributor

mjordan commented Apr 6, 2022

Thanks Don. The PDF viewer is rendering so it's probably OK, but it will be good to check. The likely problem is that the PDF files aren't accessible to the viewer.

@rbos
Copy link
Author

rbos commented Apr 6, 2022

Okay; I had EXPOSE_FEDORA already set, so that wasn't it, and changing "default download method" and restarting didn't change anything, but I did a fresh make down/pull/up, and re-applied our SSL settings to docker-compose.yml, and it seems to be rendering the PDFs now.

I think this ticket can be closed, if @mjordan will verify that uploads are working again as well.

@librarychik
Copy link

@rbos I was able to upload an image file and docx file. (yay)

@rbos
Copy link
Author

rbos commented Apr 6, 2022

Tentatively closed, then.

@rbos rbos closed this as completed Apr 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants