webMethods Suite Log File Archiver

The job of the Log File Archiver is to archive and (after a customizable retention period) delete log files from webMethods Suite products. This prevents file systems from filling up and helps to cater for compliance requirements.

Log File Archiver has been implemented as an Ant script, because that makes it very easy to work with files. Also, changes can be done quickly and easily, when a certain environment needs different handling.

To make installation and use easier for standard environments, it comes with convenience scripts (batch and shell) to make its use as easy as possible.

Currently supported webMethods components are

• Integration Server and Microservices Runtime
• Universal Messaging
• MWS
• Optimize
  • Analytic Engine
  • Web Service Data Collector
• Command Central
  • CCE
  • SPM

If you need support for other components, please create a GitHub issue or simply implement it yourself and feed back your improvement as a pull request.

How It Works

The overall approach is that the script loops over all components and does the following things:

• Change into the log file directory of the component.
• Identify all files that are to be processed on that day. Those are effectively the files with a date-time-stamp, which points to a day in the past. In other words, all files that have accumulated since the last run of Log File Archiver will be processed.
  • On the very first run all files that have been created, perhaps over months or even years, will be processed. This can be a lot of files and a significant amount of space.
  • When run on a daily basis, usually via something like a Cron job, this means that files for only the last day will be processed.
  • As an exception to the last point, there are cases when the logging sub-system still holds a lock on the file from a past day. This typically happens when nothing was written into a specific log file over the entire day, which can prevent the roll-over from happening. With such a lock in place, there is no way for Log File Archiver, or any other tool, to safely move the file. It will therefore be skipped and tried the next time. While unlikely, it is possible this happens multiple days in a row.
  • It is important to note that this selection happens based on the file name and NOT the age of the file (as per the file system). This is different from deleting the archived individual files (see below), which happens based on the actual age of files.
• The first action with the selected files will be to put them into a ZIP file for long-term retention. This ZIP file will be created in $WEBMETHODS_HOME/archive/logs/. The name fill follow the pattern <COMPONENT_ID>_yyyy-MM-dd_HH-mm-ss-SSS.zip. Example for Integration Server: is_2023-11-22_15-53-02-114.zip. The ZIP file generated by the first run can be rather big, if the environment has already been running for a while (see above).
• The second action will be to move all selected files into the respective archive log directory. The default location for this is $WEBMETHODS_HOME/archive/logs/<COMPONENT_NAME>/.
• As a third step all individual log files in the per-compnent sub-directories (as opposed to the ZIP files in the main log archive directory) that have "expired" will be deleted. Expired means files older than a certain number of days, based on the age of the file in the file system and not based on their names.
• The fourth and last step is deleting those long-term ZIP files that have expired. Their default retention is now 10 years.
  • Please verify the legal requirements for how long you must store these files. It is also common that after the legally imposed retention period you are actually required to delete them. Again, this needs to be handled by someone who is qualified to give legal advice within your jurisdiction.
  • Special care is needed on systems where the Log File Archiver was first run some time after the system went online. Here you can end up with the situation that a ZIP file that as a whole must be kept, contains files that must(!) be deleted. Example: Assume your system was running for 2 years prior to the installation of Log File Archiver. In that case, the first ZIP file that is 9 years old will most likley contain files that are actually 11 years old. So you may be required to delete those individual files from the ZIP file.

Installation

Scripted Installation

Currently for Windows and Linux systems only, there are batch/script files (install.bat and install.sh) to perform the installation.

• If you used the default location (c:\SoftwareAG or /opt/softwareag) or have set the environment variable WEBMETHODS_HOME to point to your installation you simply run it

  install.bat

  or

  install.sh

• Otherwise you must specify the location of the webMethods installation as a parameter

  install.bat WEBMETHODS_INSTALLATION

  or

  install.sh WEBMETHODS_INSTALLATION

The installation will place Log File Archiver into $WEBMETHODS_HOME/tool/operations/logArchiver. It is not possible to specify a different location using these installation scripts. If you need a different place, you must perform the installation manually as described in the next section.

Manual Installation

Alternatively, you can perform the installation manually. In this case you must ensure that Ant and Java are installed and set up correctly:

• Install a JDK and Ant on your machine. The details depend on your environment and are not in scope here. You will, however, find plenty of information on that subject on the Internet.
• Configure the following environment variables
  • WEBMETHODS_HOME pointing to the location directory of the webMethods suite
  • ANT_HOME pointing to your Ant installation location. The easiest way will be to use the installation that usually comes with the webMethods Suite.
  • JAVA_HOME pointing to a JDK. The easiest way will be to use the installation that usually comes with the webMethods Suite.
• Ensure that $ANT_HOME/bin and $JAVA_HOME/bin are part of the search path (PATH variable)

You can now perform the installation.

• To install into the default location ($WEBMETHODS_HOME/tools/operations):

  ant install

• To install into a ustom installation directory:

  ant -Dinstall.dir=<CUSTOM_INSTALL_DIR> install

Create Archive for Transfer to Target System

As a last option, you can also build a ZIP file for later manual installation. This will usually be your preferred way, if one or both of the two following scenario are relevant:

• You populate your servers or build container images with data from a fixed location. Those locations are typically secured and provide functionality like check-sums, auditing for file changes, etc.
• You want to use dedicated releases that were created from the source code at a certain point in time. Also, such releases are typically subject to an elaborate and audited release process.

As a prerequisite, similar to the manual installation, you also need to set up Ant and Java (environment variables and path) like explained in the section for manual installation.

Then you can simply run Ant without any additional parameters:

ant

The ZIP archive will be created into a new folder (./dist) that will be created automatically. This folder is excluded from source control by Git.

Depending on the details of how things are handled, you may have to make the run_log-archive.sh script executable on the target environment. If you are unsure about the exact permissions to set, it is recommend to err on the side of caution and use chmod 700 run_log-archive.sh. That ensures that only the owner of the file can directly execute it.

Usage

Similar to the installation, there are also batch/script files to invoke the actual log file archiver.

The script files' (run_log-archive.bat and run_log-archive.sh) logic is such that on a normal system no further setup work is required. But on systems where a more elaborate configuration exists, those specifics are taken into account. Therefore the following environment variables are used (if they are all defined, the archiver files can be stored anywhere):

• WEBMETHODS_HOME: Installation location of the webMethods Suite for which the log files should be archived. If not defined, it will be assumed that the script is installed in $WEBMETHODS_HOME/tools/operations/logArchiver and the value derived from that.
• JAVA_HOME:
  • On Linux only and if it is not set, the script uses the contents of /etc/profile.d/jdk.sh if it exists.
  • If not defined, the JVM that comes with the webMethods Suite will be used (depends on name of JVM folder and may therefore not work on all versions of the webMethods Suite).
• ANT_HOME:
  • On Linux only and if it is not set, the script uses the contents of /etc/profile.d/ant.sh if it exists.
  • If not defined, the Ant that comes as part of the webMethods Suite will be used (in $WEBMETHODS_HOME/common/lib/ant).

It is recommended for most cases to set up a scheduled job (e.g. via Cron) on the operating system to execute the script on a daily basis. Please be aware that in this case the environment variables may differ from what is normally available when a user is logged in. This is normal for running Cron jobs and in now way specific to Log File Archiver

It is also an option to run the Ant script from a Continuous Integration server (like e.g. Jenkins). In that case the control over environment variables may be easier than via a Cron job.

On Windows systems there have been cases where, due to file locks, log files could only be moved one or several days after that should have happened. While this delay may be a bit annoying, it is usually not a real problem.

Parameters

The behavior is controlled by Ant properties with defaults being in the script. Those defaults can be overridden from the command line by using standard Ant syntax.

Example:

ant -f build_log-archive.xml -DretentionDays=12

The following parameters can be used

Name	Default	Format	Description	Min. Version
webMethods.home	OS environment variable $WEBMETHODS_HOME	Valid path	Location of the webMethods installation	1.0
createZIP	true	true/false	Should a ZIP be created per invocation	1.1
retentionDays	30	positive number	Number after which log files (not ZIP) shall be deleted	1.0
retentionDaysZIP	3660	positive number	Number after which ZIP archives shall be deleted	1.1
instancesNameIS	default	text	Name of IS instance for which files shall be archived	1.1
instanceNameUM	umserver	text	Name of UM server instance for which file shall be archived	1.2

---

These tools are provided as-is and without warranty or support. They do not constitute part of the Software AG product suite. Users are free to use, fork and modify them, subject to the license agreement. While Software AG welcomes contributions, we cannot guarantee to include every contribution in the master project.

Contact us at TECHcommunity if you have any questions.