-
Notifications
You must be signed in to change notification settings - Fork 84
Troubleshooting
Troubleshooting any super workflow starts with inspecting command line output or log files. The super workflow generates a variety of log files but you should always start with the super.log.
Examples super logs of successful workflow results can be found here.
The main super workflow log is the super.log. This log is located at /Library/Management/super/logs/super.log. When run from the command line, super also outputs the content of the super.log in real time.
Not only is this the main super workflow log, it's the only log that is designed to be easily human readable. As such, it should be the first thing you look at when troubleshooting workflow issues.
When super is initially installed (via any method) or deployed via Jamf Pro only the initial installation and startup super.log entries are returned to the install process or Jamf Pro Policy log. This is by design, so as not to hang the install process or Jamf Pro agent while the rest of the super update workflow (which may take a very long time) continues.
super.log line entry example:
Tue Oct 10 06:58:29 ASVM-13-0 super[4450]: Status: A macOS minor update is available: macOS Ventura 13.6-22G120
As you can see from this super.log example, the beginning of the log line entry contains the date, time, computer hostname, process name, and process ID.
While the super.log can appear daunting, you can easily narrow your troubleshooting scope by searching for the phrases "Warning" or "Error" in the super.log. The built-in macOS Console.app is not only the default application for the ".log" file type, it also has great text search and filter functionality.
Older super logs can be found in .zip archives in the /Library/Management/super/logs-archive/ folder. Anytime a super log file grows to be more than 1 MB in size, all the current super logs are archived and new "empty" logs are started. You may also find an archive of logs for older versions of super (with a file name containing "legacy"). The super workflow never deletes any log file or log archive.
If you are logged into the macOS GUI, this option automatically opens all available super generated logs in the Console.app. This option can not be set via a MDM configuration profile.
Command option examples:
--open-logs or -o or -O
Generate additional debugging output to the command line and various super logs.
Command option examples:
--verbose-mode or -V
Command option disable examples:
--verbose-mode-off or -v
Configuration profile example:
<key>VerboseMode</key>
<true/>
super.log verbose mode line entry example:
Tue Oct 10 06:57:23 ASVM-13-0 super[3316]: Verbose Mode: Function check_current_user: current_user_is_admin is: TRUE
Even in verbose mode, the super.log remains the primary source of super workflow information. As you can see from this verbose mode super.log example, in addition to the standard log line information, the phrase "Verbose Mode" appears as does the specific super script function name that was running at the time. Knowing the specific script function can significantly help finding the source of any potential issues within the super script.
Useful information in the verbose mode super.log includes; the values of internal super script parameters, the contents of the super preference file /Library/Management/super/com.macjutsu.super.plist, and the output of other helper commands or processes used in the super workflow.
Further, when super is run with the --verbose-mode option, update credential passwords are also displayed in the command line but are not be saved to the super.log file on disk. This obviously helps identify password-related issues.
Avoid widespread deployment of the --verbose-mode option in a production environment as it creates very large log files. This mode is designed specifically for troubleshooting isolated issues and should be used sparingly.
Delete all locally saved options, except for any previously saved authentication options. Thus returning all non-authentication options to their default setting.
Command option examples:
--reset-super or -x or -X
Sometimes the super preferences file may contain settings that are causing unexpected results, so resetting these settings can help resolve issues. For example, if you make changes to super settings via a MDM configuration profile, consider also using the --reset-super option to clear any locally cached super options that may cause conflicts.
You can also use this option along with the --workflow-disable-update-check and --auth-delete-all options to fully disable any active or deferred super update workflows. With this combination of options, the super workflow deletes any local super settings, kills any existing super processes, deletes any saved credentials, and deletes the super LaunchDaemon.
This option can not be set via a MDM configuration profile. However, any other deferral options that are specified via a super MDM configuration profile remain in effect.
You can easily reinstall or update super by simply running a copy of the super script from outside its working folder /Library/Mangement/super/, or if deploying via Jamf Pro Policy, running a Policy that contains the super script again. However, reinstalling super almost never solves any workflow problems. The only issues resolved by reinstalling are those that are either fixed by a new version of super, or issues as a result of improper manual installation.
In other words, avoid deploying or installing super itself directly into the working folder. Instead, it's best to rely on the automatic installation mechanism that super provides for you.
All super dialogs and notifications are handled by IBM Notifier.app and macOS upgrade workflows leverage mist-cli. Normally, if either of these items are not found on the local system they are automatically installed by super via direct download from GitHub.
If you have network issues preventing access to these GitHub repositories then you can pre-install these items prior to running super. As long as IBM Notifier.app and mist-cli are in the expected location and are the correct version, then super does not attempt to automatically install them.
The original mist-cli package needs no modifications for a super compatible deployment. However, the IBM Notifier source provides no installer package, thus there is no default installation location and you would need to build your own installer. The super workflow expects the IBM Notifier.app to be located inside the default super working folder located at /Library/Management/super/.
Similar to reinstalling super, uninstalling and then installing super again is unlikely to resolve any workflow issues.
However, if you need to remove super from a computer, the Super-Friends folder contains a Remove-Super.sh script that deletes all super related items.
Other logs you might find in the super log folder /Library/Management/super/logs/ are not directly generated by super but are instead created by sub-processes that super leverages to accomplish the macOS update workflow. Herin lies a fundamental truth about super, it is but only a script asking other native macOS processes to do the actual work of updating macOS (because the native processes are the only ones allowed to update macOS).
The super workflow "watches" these logs to determine status. For example, if you know that a new macOS update should be available but the super.log states that there are no available updates, it's because output from the native macOS softwarupdate command claimed that there were no available updates. In this case, super can't "make the system find the update better" as macOS systems are beholden to what softwareupdate thinks is "true" about the macOS update status. This is why for many error conditions, super is relegated to simply trying again later.
Again, example super logs of successful workflow results can be found here.
Results of the last /usr/libexec/mdmclient AvailableOSUpdates command which generates structured JSON and XML text of all available Apple software updates including full macOS installers and security updates that are not otherwise exposed via the softwareupdate command. Also included in these results are the expected dates of availability for any deferred software updates or upgrades due to a restrictions configuration profile.
In other words, if the mdmclient results show that a software update or upgrade is deferred, and super detects that the system has an installed configuration profile with deferral restrictions, then the update or upgrade is not targeted. The discovery of any deferred items is always reported to the standard super.log.
In other words, if the mdmclient reports that a software update or upgrade is deferred, then super respects that deferral and does not target that item.
This is one of the few log files that does not maintain a history, as it only shows the most recent list of available Apple software updates.
Results of the last softwareupdate --list command which should show the list of all available Apple software updates. This log is only generated if the results of the mdm-client.list indicate there is an available macOS update or upgrade. Further, systems with macOS 12.3 or newer also show any available macOS upgrades here.
This is one of the few log files that does not maintain a history, as it only shows the most recent list of available Apple software updates.
Results of the last mist-cli command which should show the list of all available macOS upgrade installers. This log is only generated if the --install-macos-major-upgrades option is enabled and the workflow requires a macOS installer. Newer macOS systems do not require installers to upgrade.
This is one of the few log files that does not maintain a history, as it only shows the most recent list of available macOS upgrade installers.
Results of the last download of the Mac Admins SOFA macOS structured JSON feed. This is used to determine macOS release dates when the --schedule-zero-date-release option is enabled and there is an available macOS update or upgrade.
This is one of the few log files that does not maintain a history, as it only shows the most recentnSOFA macOS structured JSON feed.
A history of the results of all "non-list" softwareupdate commands.
The results of variety of local softwareupdate workflow tasks can be found in this log, including:
- The download and installation of all non-macOS Apple software updates.
- The download and preparation of macOS updates (including RSR updates) if there is an active user logged in.
- When not using the MDM workflow, the installation of all macOS updates (including RSR updates).
- When not using the MDM workflow, and on macOS 12.3 and newer the download, preparation and installation of macOS upgrades.
A history of the results of all "non-list" mist-cli commands and all startosinstall commands.
The results of any macOS installer-based workflow tasks can be found found in this log, including:
- The download and preparation of macOS installer applications via
mist-cli. - When not using the MDM workflow, on systems with macOS 12.2 and older the preparation and installation of macOS upgrades.
When using the MDM workflow, this log is generated by the macOS log command stream mechanism and filtered to show the MDM command progress. In other words, this log shows the exact dates and times when any MDM command is received by the system.
Interesting events in the mdm.log include;
-
Received MDM command to check for updates:
Received HTTP response (200) [Acknowledged(ScheduleOSUpdateScan) -
Received MDM command to return the list of available updates:
Received HTTP response (200) [Acknowledged(AvailableOSUpdates) -
Received MDM blank push:
Received HTTP response (200) [Idle] -
Received MDM command to start the download or install process:
Received HTTP response (200) [Acknowledged(ScheduleOSUpdate)
When using the MDM workflow, this log is generated by the macOS log command stream mechanism and filtered to show the softwareupdated reported progress. In other words, this log shows the local progress of the macOS update or upgrade process after the MDM command has been successfully received.
Interesting events in the mdmWorkflow.log include;
-
The macOS update has started downloading:
(start): phase:DOWNLOADING_UPDATE -
The macOS update is being validated and prepared:
(start): phase:PREPARING_UPDATE -
The previous download or preparation phase is complete:
(end): phase:COMPLETED -
The macOS update is complete and restart is imminent:
(start): phase:APPLYING
In addition to the display timeout options and the testing and validation timeout options, there are internal workflow error timeouts that can be manually adjusted in the super script itself.
The super workflow leverages a variety of timeouts in order to detect and handle workflow failures. For example, super "watches" the output of softwareupdate command line tool (via the asu-workflow.log) while it performs a macOS update task. If softwareupdate stops responding after a certain timeout, it's assumed that softwareupdate has hung or crashed and that the super workflow should try again later.
The original values for these timeouts are set based on significant testing but it's simply impossible to predict how long a workflow should take on every possible system and network combination. In some cases these values can be too aggressive givent the performance of your local system and network speed.
It's important to understand that while you can adjust these workflow timeouts, the timeouts themselves do not directly affect the success of any sub-process. These timeouts only adjust the amount of time super is waiting for a sub-process or the system to return results. If the underlying mechanism that super is waiting for has crashed or hung, increasing these timeouts simply increases the amount of time it takes for super to detect the issue.
Workflow timeouts that can be manually adjusted in the super script include;
-
The number of seconds to timeout various workflow startup processes if no progress is reported:
TIMEOUT_START_SECONDS=120 -
The number of seconds to timeout the
softwareupdatemacOS download/prepare workflow if no progress is reported:
TIMEOUT_ASU_SYSTEM_SECONDS=1200 -
The number of seconds to timeout the
softwareupdatenon-system update workflow if no progress is reported:
TIMEOUT_NON_SYSTEM_ASU_SECONDS=600 -
The number of seconds to timeout the
mist-climacOS installer download workflow if no progress is reported:
TIMEOUT_INSTALLER_DOWNLOAD_SECONDS=300 -
The number of seconds to timeout the macOS installation workflow if no progress is reported:
TIMEOUT_INSTALLER_WORKFLOW_SECONDS=600 -
The number of seconds to timeout MDM commands if no response is reported:
TIMEOUT_MDM_COMMAND_SECONDS=300 -
The number of seconds to timeout the MDM download/prepare workflow if no progress is reported:
TIMEOUT_MDM_WORKFLOW_SECONDS=600