Improve resource resolution #161

jmehrens · 2024-03-07T21:41:25Z

Jenkins Dependencies and Class Loading shows a classloader tree where for example JAF-API is not able to locate JakartaMail (plug-in C) mailcap.file resolved from JAF-API (plugin A) because the context classloader is Jenkins core. See: #145 (comment)

When I removed the workaround, your new code correctly kicked in and used the calling class loader from Jenkins Jakarta Activation plugin, but since that class loader could not see the Jenkins Jakarta Mail plugin, the mailcap was empty.

Mail uses JAF but Mail jar contains the mailcap resource file.

More thought needs to be put in to this but there are some options:

Add constructors for MailcapCommandMap and MimetypesFileTypeMap that take a ClassLoader. Then modify mail to use the newer constructor. Need to examine call sites.
Use StackWalker to find the calling class loader and probe that class loader after the context class loader and before that application classloader.
Have mail locate and parse the mailcap file and add the entries manually.
Force the context classloader to the mail classloader when interacting with JAF. This would work but, I think this is the wrong approach as in some cases user might want the CCL to load the resource.

The text was updated successfully, but these errors were encountered:

jmehrens · 2024-03-07T21:48:42Z

@basil If you are willing and able to:

List the byte size of angus-activation-2.0.2.jar from your machine.
Attach debug output from with system property jakarta.activation.debug=true. This will send output to standard out file.

basil · 2024-03-07T21:52:37Z

How is the byte size of angus-activation-2.0.2.jar relevant?

jmehrens · 2024-03-07T22:24:15Z

It is a way to see if that jar was deployed without asking if that jar was deployed :) The idea is force examination of the system instead of examination of ones memory of what was deployed. I trust you did it correctly but want to verify it. We already have 2 tickets of users falling in to this trap (which I need to fix) and I want to rule this out as a problem.

If that jar is missing there are some silent failures that are only shown with debugging on. As an example:

MimetypesFileTypeMap: load HOME
Cannot find or load an implementation for MimeTypeRegistryProvider.MimeTypeRegistry: can't load /home/jmehrens/.mime.types; Exception: java.lang.IllegalStateException: Provider for jakarta.activation.spi.MimeTypeRegistryProvider cannot be found
MimetypesFileTypeMap: load SYS
Cannot find or load an implementation for MimeTypeRegistryProvider.MimeTypeRegistry: can't load /usr/lib/jvm/jdk-21-oracle-x64/conf/mime.types; Exception: java.lang.IllegalStateException: Provider for jakarta.activation.spi.MimeTypeRegistryProvider cannot be found
MimetypesFileTypeMap: load JAR
MimetypesFileTypeMap: !anyLoaded
MimetypesFileTypeMap: not loading mime types file: /META-INF/mime.types
MimetypesFileTypeMap: load DEF
Cannot find or load an implementation for MimeTypeRegistryProvider.MimeTypeRegistry: can't load /META-INF/mimetypes.default; Exception: java.lang.IllegalStateException: Provider for jakarta.activation.spi.MimeTypeRegistryProvider cannot be found

basil · 2024-03-07T23:16:08Z

The idea is force examination of the system instead of examination of ones memory of what was deployed.

I support users in an open-source project, so I can understand this perspective. This also means we can skip the indirection and I can explain to you how to reproduce the problem from scratch.

Download the latest Jenkins weekly release, then start it with Java 11, 17, or 21 with java -jar jenkins.war. The first time you start it, complete the setup wizard following the defaults and install the suggested plugins. Among other things, this will install three interesting plugins: Email Extension, Jakarta Mail API, and Jakarta Activation API.

Once Jenkins is up and running, go to Manage Jenkins, System. Under "System Admin e-mail address", set an address like Basil Crow <basilcrow@example.com>. Under "Extended E-mail Notification", configure your SMTP server. If your SMTP server requires authentication, configure that under Advanced. Also check Enable Debug Mode as that can be quite handy.

Once Jenkins is configured, create a new Freestyle project. As an example, I often create a test project that clones https://github.com/basil/simple-maven-project-with-tests.git and runs mvn clean verify. Under Post-build Actions add Editable Email Notification. Under Project Recipient List add your email address. Under Advanced Settings, Triggers remove any existing triggers and add an Always trigger that sends to Recipient List. This means every time you run the job, it will send an email to the Recipient List.

Now run the job in Jenkins, and at the end it should send you an email. This should work before proceeding to the next step, so if it fails, do whatever it takes to get it to work, or contact me for help. Now, once it is working, you can break it by removing my workaround as follows.

Clone https://github.com/jenkinsci/jakarta-mail-api-plugin and remove the workaround with rm -rf src/main/java. Build it with Java 11, 17, or 21 with mvn clean verify -Pquick-build, which will generate a target/jakarta-mail-api.hpi file. Replace ${JENKINS_HOME}/plugins/jakarta-mail-api.jpi with this file and restart Jenkins. Re-run the job. An email should no longer be sent and in the console log you should see a stack trace about how some mime type is missing because the mailcap is empty. Attach a debugger to the Java process and you can verify the gory details of my assessment. The stack trace:

12:23:33 DEBUG SMTP: IOException while sending, closing, THROW: 
12:23:33 jakarta.activation.UnsupportedDataTypeException: no object DCH for MIME type multipart/mixed; 
12:23:33 	boundary="----=_Part_0_1965321476.1709583812613"
12:23:33 	at jakarta.activation.ObjectDataContentHandler.writeTo(DataHandler.java:872)
12:23:33 	at jakarta.activation.DataHandler.writeTo(DataHandler.java:291)
12:23:33 	at jakarta.mail.internet.MimeBodyPart.writeTo(MimeBodyPart.java:1683)
12:23:33 	at jakarta.mail.internet.MimeMessage.writeTo(MimeMessage.java:1913)
12:23:33 	at org.eclipse.angus.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:1305)
12:23:33 	at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:554)
12:23:33 	at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:484)
12:23:33 	at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:385)
12:23:33 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
12:23:33 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
12:23:33 	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)
12:23:33 	at hudson.model.Build$BuildExecution.cleanUp(Build.java:189)
12:23:33 	at hudson.model.Run.execute(Run.java:1936)
12:23:33 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
12:23:33 	at hudson.model.ResourceController.execute(ResourceController.java:101)
12:23:33 	at hudson.model.Executor.run(Executor.java:442)
12:23:33 MessagingException message: IOException while sending message
12:23:33 Next UnsupportedDataTypeException message: no object DCH for MIME type multipart/mixed; 
12:23:33 	boundary="----=_Part_0_1965321476.1709583812613"

jmehrens · 2024-03-09T17:57:39Z

@basil Thanks for the reproducer! This will save me a ton of time. You are a gentleman and a scholar.

For my own notes this is the project console output of the basic test to get a weekly build working with no changes.

Looking for: jmehrens@localhost
	starting at: 0
	firstFoundIdx: 0
	firstFoundIdx-substring: jmehrens@localhost
	=> found type: 0
Successfully created MimeMessage
Sending email to: jmehrens@localhost
DEBUG: getProvider() returning jakarta.mail.Provider[TRANSPORT,smtp,org.eclipse.angus.mail.smtp.SMTPTransport,Oracle]
DEBUG SMTP: useEhlo true, useAuth false
DEBUG SMTP: trying to connect to host "localhost", port 2525, isSSL false
220 jmehrens-VirtualBox ESMTP SubEthaSMTP null
DEBUG SMTP: connected to host "localhost", port: 2525
EHLO jmehrens-VirtualBox
250-jmehrens-VirtualBox
250-8BITMIME
250-AUTH LOGIN
250 Ok
DEBUG SMTP: Found extension "8BITMIME", arg ""
DEBUG SMTP: Found extension "AUTH", arg "LOGIN"
DEBUG SMTP: Found extension "Ok", arg ""
DEBUG SMTP: use8bit false
MAIL FROM:<nobody@nowhere>
250 Ok
DEBUG SMTP: sendPartial set
RCPT TO:<jmehrens@localhost>
250 Ok
DEBUG SMTP: Verified Addresses
DEBUG SMTP:   jmehrens@localhost
DATA
354 End data with <CR><LF>.<CR><LF>
Date: Sat, 9 Mar 2024 21:58:24 -0600 (CST)
From: address not configured yet <nobody@nowhere>
To: jmehrens@localhost
Message-ID: <1521760439.1.1710043105326@jmehrens-VirtualBox>
Subject: simple-maven-project - Build # 7 - Successful!
MIME-Version: 1.0
Content-Type: multipart/mixed; 
	boundary="----=_Part_0_26769737.1710043105033"
X-Jenkins-Job: simple-maven-project
X-Jenkins-Result: SUCCESS
List-ID: 

------=_Part_0_26769737.1710043105033
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

simple-maven-project - Build # 7 - Successful:

Check console output at http://localhost:8080/job/simple-maven-project/7/ to view the results.
------=_Part_0_26769737.1710043105033--
.
250 Ok
DEBUG SMTP: message successfully delivered to mail server
QUIT
221 Bye
Finished: SUCCESS

Here is the launch console output for jenkins.war running with activation debugging for the above console log:

jmehrens@jmehrens-VirtualBox:~/jenkins$ java -Djakarta.activation.debug=true -Dangus.activation.debug=true -jar jenkins.war
Running from: /home/jmehrens/jenkins/jenkins.war
webroot: /home/jmehrens/.jenkins/war
2024-03-10 03:55:40.405+0000 [id=1]	INFO	winstone.Logger#logInternal: Beginning extraction from war file
2024-03-10 03:55:40.733+0000 [id=1]	WARNING	o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2024-03-10 03:55:40.941+0000 [id=1]	INFO	org.eclipse.jetty.server.Server#doStart: jetty-10.0.20; built: 2024-01-29T20:46:45.278Z; git: 3a745c71c23682146f262b99f4ddc4c1bc41630c; jvm 21.0.1+12-LTS-29
2024-03-10 03:55:41.716+0000 [id=1]	INFO	o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2024-03-10 03:55:41.874+0000 [id=1]	INFO	o.e.j.s.s.DefaultSessionIdManager#doStart: Session workerName=node0
2024-03-10 03:55:43.204+0000 [id=1]	INFO	hudson.WebAppMain#contextInitialized: Jenkins home directory: /home/jmehrens/.jenkins found at: $user.home/.jenkins
2024-03-10 03:55:43.553+0000 [id=1]	INFO	o.e.j.s.handler.ContextHandler#doStart: Started w.@7adbd080{Jenkins v2.448,/,file:///home/jmehrens/.jenkins/war/,AVAILABLE}{/home/jmehrens/.jenkins/war}
2024-03-10 03:55:43.585+0000 [id=1]	INFO	o.e.j.server.AbstractConnector#doStart: Started ServerConnector@45afc369{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2024-03-10 03:55:43.631+0000 [id=1]	INFO	org.eclipse.jetty.server.Server#doStart: Started Server@f0f2775{STARTING}[10.0.20,sto=0] @4472ms
2024-03-10 03:55:43.634+0000 [id=40]	INFO	winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2024-03-10 03:55:44.275+0000 [id=46]	INFO	jenkins.InitReactorRunner$1#onAttained: Started initialization
2024-03-10 03:55:44.717+0000 [id=53]	INFO	jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2024-03-10 03:55:54.000+0000 [id=58]	INFO	jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
MimetypesFileTypeMap: load HOME
MailcapCommandMap: load HOME
MimeTypeRegistry: can't load from file - /home/jmehrens/.mime.types; Exception: java.io.FileNotFoundException: /home/jmehrens/.mime.types (No such file or directory)
MimetypesFileTypeMap: load SYS
MimeTypeRegistry: can't load from file - /usr/lib/jvm/jdk-21-oracle-x64/conf/mime.types; Exception: java.io.FileNotFoundException: /usr/lib/jvm/jdk-21-oracle-x64/conf/mime.types (No such file or directory)
MimetypesFileTypeMap: load JAR
MimetypesFileTypeMap: !anyLoaded
new MailcapFile: file /home/jmehrens/.mailcap
MailcapRegistry: can't load from file - /home/jmehrens/.mailcap; Exception: java.io.FileNotFoundException: /home/jmehrens/.mailcap (No such file or directory)
MailcapCommandMap: load SYS
MimetypesFileTypeMap: not loading mime types file: /META-INF/mime.types
MimetypesFileTypeMap: load DEF
new MailcapFile: file /usr/lib/jvm/jdk-21-oracle-x64/conf/mailcap
MailcapRegistry: can't load from file - /usr/lib/jvm/jdk-21-oracle-x64/conf/mailcap; Exception: java.io.FileNotFoundException: /usr/lib/jvm/jdk-21-oracle-x64/conf/mailcap (No such file or directory)
MailcapCommandMap: load JAR
MailcapCommandMap: getResources
MailcapCommandMap: URL jar:file:/home/jmehrens/.jenkins/plugins/jakarta-mail-api/WEB-INF/lib/angus-mail-2.0.3.jar!/META-INF/mailcap
Added: MIMETypeEntry: text/html, html
Added: MIMETypeEntry: text/html, htm
Added: MIMETypeEntry: text/html, HTML
Added: MIMETypeEntry: text/html, HTM
Added: MIMETypeEntry: text/plain, txt
Added: MIMETypeEntry: text/plain, text
Added: MIMETypeEntry: text/plain, TXT
Added: MIMETypeEntry: text/plain, TEXT
Added: MIMETypeEntry: image/gif, gif
Added: MIMETypeEntry: image/gif, GIF
Added: MIMETypeEntry: image/ief, ief
Added: MIMETypeEntry: image/jpeg, jpeg
Added: MIMETypeEntry: image/jpeg, jpg
Added: MIMETypeEntry: image/jpeg, jpe
Added: MIMETypeEntry: image/jpeg, JPG
Added: MIMETypeEntry: image/tiff, tiff
Added: MIMETypeEntry: image/tiff, tif
Added: MIMETypeEntry: image/png, png
Added: MIMETypeEntry: image/png, PNG
Added: MIMETypeEntry: image/x-xwindowdump, xwd
Added: MIMETypeEntry: application/postscript, ai
Added: MIMETypeEntry: application/postscript, eps
Added: MIMETypeEntry: application/postscript, ps
Added: MIMETypeEntry: application/rtf, rtf
Added: MIMETypeEntry: application/x-tex, tex
Added: MIMETypeEntry: application/x-texinfo, texinfo
Added: MIMETypeEntry: application/x-texinfo, texi
Added: MIMETypeEntry: application/x-troff, t
Added: MIMETypeEntry: application/x-troff, tr
Added: MIMETypeEntry: application/x-troff, roff
Added: MIMETypeEntry: audio/basic, au
Added: MIMETypeEntry: audio/midi, midi
Added: MIMETypeEntry: audio/midi, mid
new MailcapFile: InputStream
Added: MIMETypeEntry: audio/x-aifc, aifc
Added: MIMETypeEntry: audio/x-aiff, aif
Added: MIMETypeEntry: audio/x-aiff, aiff
Added: MIMETypeEntry: audio/x-mpeg, mpeg
Added: MIMETypeEntry: audio/x-mpeg, mpg
Added: MIMETypeEntry: audio/x-wav, wav
Added: MIMETypeEntry: video/mpeg, mpeg
Added: MIMETypeEntry: video/mpeg, mpg
Added: MIMETypeEntry: video/mpeg, mpe
Added: MIMETypeEntry: video/quicktime, qt
Added: MIMETypeEntry: video/quicktime, mov
Added: MIMETypeEntry: video/x-msvideo, avi
parse: text/plain;;		x-java-content-handler=org.eclipse.angus.mail.handlers.text_plain
  Type: text/plain
    Command: content-handler, Class: org.eclipse.angus.mail.handlers.text_plain
MimetypesFileTypeMap: successfully loaded mime types file: /META-INF/mimetypes.default
parse: text/html;;		x-java-content-handler=org.eclipse.angus.mail.handlers.text_html
  Type: text/html
    Command: content-handler, Class: org.eclipse.angus.mail.handlers.text_html
parse: text/xml;;		x-java-content-handler=org.eclipse.angus.mail.handlers.text_xml
  Type: text/xml
    Command: content-handler, Class: org.eclipse.angus.mail.handlers.text_xml
parse: multipart/*;;		x-java-content-handler=org.eclipse.angus.mail.handlers.multipart_mixed; x-java-fallback-entry=true
  Type: multipart/*
    Command: content-handler, Class: org.eclipse.angus.mail.handlers.multipart_mixed
parse: message/rfc822;;	x-java-content-handler=org.eclipse.angus.mail.handlers.message_rfc822
  Type: message/rfc822
    Command: content-handler, Class: org.eclipse.angus.mail.handlers.message_rfc822
MailcapCommandMap: successfully loaded mailcap file from URL: jar:file:/home/jmehrens/.jenkins/plugins/jakarta-mail-api/WEB-INF/lib/angus-mail-2.0.3.jar!/META-INF/mailcap
MailcapCommandMap: load DEF
new MailcapFile: InputStream
MailcapCommandMap: successfully loaded mailcap file: /META-INF/mailcap.default
2024-03-10 03:55:54.064+0000 [id=55]	INFO	jenkins.InitReactorRunner$1#onAttained: Started all plugins
2024-03-10 03:55:54.097+0000 [id=58]	INFO	jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2024-03-10 03:55:54.761+0000 [id=50]	INFO	h.p.b.g.GlobalTimeOutConfiguration#load: global timeout not set
2024-03-10 03:55:56.174+0000 [id=58]	INFO	jenkins.InitReactorRunner$1#onAttained: System config loaded
2024-03-10 03:55:56.175+0000 [id=58]	INFO	jenkins.InitReactorRunner$1#onAttained: System config adapted
2024-03-10 03:55:57.028+0000 [id=51]	INFO	jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2024-03-10 03:55:57.057+0000 [id=49]	INFO	jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2024-03-10 03:55:57.162+0000 [id=59]	INFO	jenkins.InitReactorRunner$1#onAttained: Completed initialization
2024-03-10 03:55:57.444+0000 [id=39]	INFO	hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running
DEBUG: setDebug: Jakarta Mail version 2.1.3
MailcapCommandMap: createDataContentHandler for text/plain
  search DB #1
    got content-handler
      class org.eclipse.angus.mail.handlers.text_plain
MailcapCommandMap: createDataContentHandler for multipart/mixed
  search DB #1
  search DB #2
  search fallback DB #1
    got content-handler
      class org.eclipse.angus.mail.handlers.multipart_mixed

TODO notes for myself:

jakarta.activation.MailcapCommandMap has multiple constructors and multiple search DBs. Mail knows it bundled with a mailcap file so one would think it would use the inputstream constructor instead of the default. The problem is the stream constructor doesn't query the context classloader too in the case an application has a custom command map.
Any fix has to align with spec or the spec needs to be modified in this release. TBD.
Need to look into a minor ticket to improve the UnsupportedDataTypeException: no object DCH for MIME type multipart/mixed; message. Needs additional details in the message to aid debugging.
Activation debugging doesn't print the version when debugging is turned on like Mail does. Both Mail and Activation could also debug print the implementation by looking at the StreamProvider and or FactoryFinder.
Unpack jenkins.war to verify the 4 mail/activation jars are current and present.
Spend time on completing the Jenkins test with workaround removed.

jmehrens · 2024-03-12T20:24:09Z

Notes to myself:

Further review of MailcapCommandMap shows all constructors search context classloader. Initial review of that point was incorrect.
I think part of the problem here is thar Mail Api can create a datahandler but the mailcap file was in JakartaMail but is now in Angus Mail. Meaning a stack walk searching for classloaders won't fix the problem. When JAF objects are passed to or set in JakartaMail API, JAF needs to be able see classloader of the mail implementation to get mailcap and load data handlers from implementation.
Need to prototype before recommending changes.

jmehrens self-assigned this Mar 11, 2024

jmehrens mentioned this issue May 20, 2024

MailcapCommandMap can not load Mime Handlers in OSGI Runtime eclipse-ee4j/angus-mail#148

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve resource resolution #161

Improve resource resolution #161

jmehrens commented Mar 7, 2024

jmehrens commented Mar 7, 2024 •

edited

Loading

basil commented Mar 7, 2024

jmehrens commented Mar 7, 2024

basil commented Mar 7, 2024 •

edited

Loading

jmehrens commented Mar 9, 2024 •

edited

Loading

jmehrens commented Mar 12, 2024

Improve resource resolution #161

Improve resource resolution #161

Comments

jmehrens commented Mar 7, 2024

jmehrens commented Mar 7, 2024 • edited Loading

basil commented Mar 7, 2024

jmehrens commented Mar 7, 2024

basil commented Mar 7, 2024 • edited Loading

jmehrens commented Mar 9, 2024 • edited Loading

jmehrens commented Mar 12, 2024

jmehrens commented Mar 7, 2024 •

edited

Loading

basil commented Mar 7, 2024 •

edited

Loading

jmehrens commented Mar 9, 2024 •

edited

Loading