[JENKINS-48344] Lightweight timestamp annotations #25

jglick · 2018-09-28T22:55:55Z

Uses jenkinsci/workflow-job-plugin#105 (and its upstreams) to implement much cheaper timestamping for Pipeline builds. Produces log file output like

…
[2018-09-28T22:49:03.570Z] ^[[8mha:////etc.etc.^[[0m[Pipeline] sh
[2018-09-28T22:49:03.895Z] [p] Running shell script
[2018-09-28T22:49:04.427Z] + sleep 1
[2018-09-28T22:49:05.249Z] + sleep 1
[2018-09-28T22:49:06.262Z] + sleep 1
[2018-09-28T22:49:07.281Z] + sleep 1
[2018-09-28T22:49:08.298Z] + sleep 1
…

which gets rendered like

No ConsoleNotes needed.

jglick · 2018-09-28T22:58:30Z

BTW this does not need #24. It could benefit from jenkinsci/jenkins#3662.

jglick · 2018-09-28T22:59:17Z

~~Some LogFileReaderTest regressions, probably just caused by updating some dep. Can look at it later.~~

jglick

Oh and have not yet checked impact on

per-step logs
Blue Ocean

jglick · 2018-09-28T23:01:07Z

src/main/java/hudson/plugins/timestamper/pipeline/GlobalAnnotator.java

+        long buildStartTime = build.getStartTimeInMillis();
+        String html = text.toString(true);
+        int start;
+        if (html.startsWith("<span class=\"pipeline-new-node\" ")) { // cf. LogStorage.startStep


Unpleasant that this is necessary. Not sure if it is solvable upstream. You would probably expect that the MarkupText is what is produced before this span is added by FileLogStorage.

…th other tests.

jglick · 2018-10-16T15:19:34Z

Per-step display in BO shows timestamps, since BO completely ignores ConsoleAnnotator and ConsoleNotes, but at least they are sort of human-readable:

dwnusbaum

Overall approach looks good to me, just added some java.time suggestions. The Blue Ocean view seems kind of awkward, do you think there is any easy way to clean it up? CC @vivek to see what he thinks of how it looks in Blue Ocean.

src/main/java/hudson/plugins/timestamper/pipeline/GlobalDecorator.java

src/main/java/hudson/plugins/timestamper/pipeline/GlobalAnnotator.java

dwnusbaum · 2018-10-22T16:57:08Z

pom.xml

-    <java.level>7</java.level>
+    <revision>1.8.11</revision>
+    <changelist>-SNAPSHOT</changelist>
+    <jenkins.version>2.121.1</jenkins.version>


Quite a jump, although required for the JEP-210 updates. Here are stats for the plugin's usage by Jenkins core version: http://stats.jenkins.io/pluginversions/timestamper.html. According to those stats, 89.63% of users of timestamper-1.8.10 are on Jenkins 2.121.1 or newer, so this looks reasonable to me.

jglick · 2018-10-24T17:33:50Z

The Blue Ocean view seems kind of awkward, do you think there is any easy way to clean it up?

Only by having Blue Ocean implement a custom display hook, like it did for ANSI coloration. Since BO ignores ConsoleNote and ConsoleAnnotatorFactory, without this patch there is no display of timestamps—so presumably BO-focused installations would not have been using this plugin to begin with. To my knowledge the BO rendering system is simply not extensible via plugins the way the classic UI is.

…he middle of a line.

timja

@StevenGBrown Would you be able to take a look at this please?

olamy

LGTM

jetersen

LGTM

jglick · 2019-01-28T20:57:27Z

I have requested comaintainership of this plugin. According to policy, there is a two-week waiting period.

jglick · 2019-02-07T20:43:00Z

Possible casualty (untested): TimestamperAPI.read, used by at least aws-cloudwatch-logs-publisher (though that now has an alternative in pipeline-cloudwatch-logs anyway).

dankirkd · 2019-02-08T00:15:47Z

@jglick Can you indicate how to use the plugin in a Jenkinsfile via the "global option", as opposed to having to wrap blocks within each of the "steps"? I can't find any example of that, and have tried to get an answer from @StevenGBrown on https://wiki.jenkins.io/pages/viewpage.action?pageId=46334448 with no success.

jglick · 2019-02-08T17:04:23Z

how to use the plugin in a Jenkinsfile via the "global option"

You do not—it is global configuration (whether specified in the UI or configuration-as-code or whatever).

dankirkd · 2019-02-08T18:50:11Z

What is the default setting on the "all pipelines" config? Is it set by default?

jglick · 2019-02-08T20:02:56Z

What is the default setting on the "all pipelines" config? Is it set by default?

No, it is off by default, to minimize the behavioral change caused by merely updating the plugin.

alde · 2019-02-18T12:04:01Z

It seems to me that hitting /timestamps/?time=%27HH:mm:ss%27&appendLog no longer has any effect, as the result is identical to hitting /consoleText.

Could this be a side-effect of this change?

jglick · 2019-02-18T22:02:01Z

@alde quite possibly. I am unaware of that URL pattern or what it is supposed to do.

dankirkd · 2019-02-18T22:36:41Z

@alde Try /timestamps/?time=HH:mm:ss&appendLog as that seems to work fine for me with version 1.8

@jglick It is just outputs the timestamp and with appendLog appends the text that goes with it - effectively looks like consoleText with a timestamp.

alde · 2019-02-19T07:07:46Z

With version 1.9 nothing I put in time={} has any effect anymore. I'll look at whether downgrading to 1.8 will have an effect. Anyhow, this may be an indication of a regression.
In 1.9 both /timestamps/?time=HH:mm:ss&appendLog and /consoleText output lines like

[2019-02-18T11:40:12.626Z] [WS-CLEANUP] Deleting project workspace...

jglick · 2019-02-19T13:26:30Z

I see, there are some scripting URLs which apparently have no test coverage. As noted in #25 (comment) there is also a Java API which I think has no test coverage. Probably needs some follow-up work to write proper test cases for all these things and make any adjustments needed for them to work with the new architecture.

j3p0uk · 2020-04-29T14:53:34Z

@jglick I think this has had a pretty serious side-effect on folks using timestamped, where the consoleText url now has timestamps, whereas before the raw text api url was timestamps.

I was using the consoleText for multi-line parsing for numerous elements, which has been broken by this change.

Is there a way to get the raw console text without the timestamps from the API?

jglick · 2020-04-29T15:05:54Z

Is there a way to get the raw console text without the timestamps from the API?

Not currently. IIRC there is a filed RFE for it.

basil · 2020-04-29T15:17:52Z

A cheap workaround could be to set the format to the empty string (id est, /timestamps/?time=&appendLog). That will still print the separator at the beginning of each line, but it might be better than nothing.

j3p0uk · 2020-04-29T15:21:04Z

A cheap workaround could be to set the format to the empty string (id est, /timestamps/?time=&appendLog). That will still print the separator at the beginning of each line, but it might be better than nothing.

Unfortunately that doesn't work, as the timestamps are now persisted in the on-disk log:

  [Pipeline] { (Checkout Build Sources)
  [Pipeline] checkout
  [2020-04-29T13:03:20.725Z] using credential ---
  [2020-04-29T13:03:20.738Z] Cloning the remote Git repository

basil · 2020-04-29T15:23:27Z

That should be fixed as of 1.11.1 (see JENKINS-60007).

[JENKINS-48344] Lightweight timestamp annotations.

077e56b

jglick requested review from olamy, abayer, rsandell, StevenGBrown, dwnusbaum and svanoort September 28, 2018 22:55

jglick commented Sep 28, 2018

View reviewed changes

jglick changed the title ~~[JENKINS-48344] Lightweight timestamp annotations~~ [WiP] [JENKINS-48344] Lightweight timestamp annotations Sep 28, 2018

jglick added 7 commits October 4, 2018 18:32

Updates to released versions.

04b31fc

NPE when encountering some other formats.

cad91e5

Updates to official releases.

8d45f11

[JENKINS-54081] Fixing interaction with timestamps step.

1afb7d4

Allow test to pass even after SECURITY-382 fix.

ce79838

Fixing TimestamperConfigTest to not use mocks.

d5abfa2

Amended fix of LogFileReaderTest to work even when sharing the JVM wi…

ec44eab

…th other tests.

jglick changed the title ~~[WiP] [JENKINS-48344] Lightweight timestamp annotations~~ [JENKINS-48344] Lightweight timestamp annotations Oct 16, 2018

jglick changed the title ~~[JENKINS-48344] Lightweight timestamp annotations~~ [WiP] [JENKINS-48344] Lightweight timestamp annotations Oct 16, 2018

Made per-step annotation work on Jenkins 2.145+.

2181e53

jglick changed the title ~~[WiP] [JENKINS-48344] Lightweight timestamp annotations~~ [JENKINS-48344] Lightweight timestamp annotations Oct 16, 2018

Merge branch 'master' into lightweight-annotator-JENKINS-48344

9b19c6e

dwnusbaum reviewed Oct 22, 2018

View reviewed changes

dwnusbaum approved these changes Oct 25, 2018

View reviewed changes

Properly synchronize access so we do not risk having a timestamp in t…

bf1e61c

…he middle of a line.

jglick mentioned this pull request Oct 26, 2018

[JENKINS-54073] Buffering for FileLogStorage jenkinsci/workflow-api-plugin#81

Merged

Forgot to implement flush & close as in jenkinsci/jenkins#3703.

dc6e9a5

jglick mentioned this pull request Jan 16, 2019

[JENKINS-55257] If ConsoleNote.encodeToBytes is called at all from an agent, it should not use AnonymousClassWarnings jenkinsci/jenkins#3851

Merged

timja approved these changes Jan 19, 2019

View reviewed changes

olamy approved these changes Jan 19, 2019

View reviewed changes

jetersen approved these changes Jan 25, 2019

View reviewed changes

jglick mentioned this pull request Feb 7, 2019

Adding myself to timestamper-plugin jenkins-infra/repository-permissions-updater#1022

Merged

jglick merged commit 0e2fe42 into jenkinsci:master Feb 7, 2019

jglick deleted the lightweight-annotator-JENKINS-48344 branch February 7, 2019 20:26

jglick added a commit to jglick/workflow-job-plugin that referenced this pull request Feb 7, 2019

jenkinsci/timestamper-plugin#25 released.

2929631

jglick mentioned this pull request Feb 7, 2019

Timestamper 1.9 released jenkinsci/workflow-job-plugin#124

Merged

daniel-beck mentioned this pull request Feb 9, 2019

Do not retain as many artifacts jenkinsci/jenkins#3893

Merged

basil mentioned this pull request Jun 23, 2019

[JENKINS-58102] GlobalAnnotator does not annotate lines that are part of a multi-line color sequence #28

Merged

jglick mentioned this pull request Jul 3, 2019

[JENKINS-45693] Remove beta mark from TaskListenerDecorator jenkinsci/workflow-api-plugin#97

Merged

This was referenced Feb 19, 2020

Add tests for JENKINS-60007 #44

Merged

[JENKINS-60007] Timestamper scripting API doesn't work for Pipeline jobs #47

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JENKINS-48344] Lightweight timestamp annotations #25

[JENKINS-48344] Lightweight timestamp annotations #25

jglick commented Sep 28, 2018 •

edited

Loading

jglick commented Sep 28, 2018

jglick commented Sep 28, 2018 •

edited

Loading

jglick left a comment

jglick Sep 28, 2018

jglick commented Oct 16, 2018

dwnusbaum left a comment

dwnusbaum Oct 22, 2018 •

edited

Loading

jglick commented Oct 24, 2018

timja left a comment

olamy left a comment

jetersen left a comment

jglick commented Jan 28, 2019

jglick commented Feb 7, 2019

dankirkd commented Feb 8, 2019

jglick commented Feb 8, 2019

dankirkd commented Feb 8, 2019

jglick commented Feb 8, 2019

alde commented Feb 18, 2019

jglick commented Feb 18, 2019

dankirkd commented Feb 18, 2019

alde commented Feb 19, 2019 •

edited

Loading

jglick commented Feb 19, 2019

j3p0uk commented Apr 29, 2020

jglick commented Apr 29, 2020

basil commented Apr 29, 2020

j3p0uk commented Apr 29, 2020

basil commented Apr 29, 2020

[JENKINS-48344] Lightweight timestamp annotations #25

[JENKINS-48344] Lightweight timestamp annotations #25

Conversation

jglick commented Sep 28, 2018 • edited Loading

jglick commented Sep 28, 2018

jglick commented Sep 28, 2018 • edited Loading

jglick left a comment

Choose a reason for hiding this comment

jglick Sep 28, 2018

Choose a reason for hiding this comment

jglick commented Oct 16, 2018

dwnusbaum left a comment

Choose a reason for hiding this comment

dwnusbaum Oct 22, 2018 • edited Loading

Choose a reason for hiding this comment

jglick commented Oct 24, 2018

timja left a comment

Choose a reason for hiding this comment

olamy left a comment

Choose a reason for hiding this comment

jetersen left a comment

Choose a reason for hiding this comment

jglick commented Jan 28, 2019

jglick commented Feb 7, 2019

dankirkd commented Feb 8, 2019

jglick commented Feb 8, 2019

dankirkd commented Feb 8, 2019

jglick commented Feb 8, 2019

alde commented Feb 18, 2019

jglick commented Feb 18, 2019

dankirkd commented Feb 18, 2019

alde commented Feb 19, 2019 • edited Loading

jglick commented Feb 19, 2019

j3p0uk commented Apr 29, 2020

jglick commented Apr 29, 2020

basil commented Apr 29, 2020

j3p0uk commented Apr 29, 2020

basil commented Apr 29, 2020

jglick commented Sep 28, 2018 •

edited

Loading

jglick commented Sep 28, 2018 •

edited

Loading

dwnusbaum Oct 22, 2018 •

edited

Loading

alde commented Feb 19, 2019 •

edited

Loading