-
Important Notes
- Open XDMoD is now no longer bundled with libraries that have license restrictions for commercial or government use. The charting library used in Open XDMoD has changed from Highcharts to Plotly JS, an open source library. This transition removes the license required from the Highcharts library. Please refer to the license notices for more information about the open source licenses bundled with Open XDMoD.
-
Enhancements
- Improve performance of Data Analytics Framework API calls for raw data (#1792, #1780, #1800, #1779)
- Add resource specifications realm (#1849, #1837, #1836, #1828, #1801, #1799, #1786)
- Change charting library to Plotly (#1869, #1866, #1864, #1850, #1822, #1846, #1791, #1899, #1900, #1902)
- Add Metric Explorer copy to clipboard button (#1859)
- Change warehouse raw data endpoint to respond with JSON text sequence. (#1858)
- Update description of Job Size: Normalized statistic (#1856)
- Add visualization of null values in Job Viewer charts (#1851)
- Add configuration setting for the ETL process log database output (#1848, #1827)
- Improve UI display when the database is down or otherwise unreachable from the webserver (#1793)
- Update log table SQL migration query to be more efficient (#1886)
- Provide more helpful error codes and messages for warehouse raw data REST requests (#1894, #1895)
-
Maintenance / Code Quality
- Updates for compatibility with PHP 7.4 (#1873, #1875, #1877, #1806)
- Remove
jquery-base64from NOTICE (#1853) - Update Rest Logging to include the username (#1809)
- Replace redundant exception classes with ones provided by Symfony. (#1798)
- Updates to support the OnDemand module enhancements (#1874, #1824, #1816, #1854, #1852, #1838)
- Remove obsolete XSEDE-specific code (#1773)
- Patch for symfony/http-foundation: CVE-2019-10913 (#1893)
-
Bug Fixes
- Escape double quotes when migrating portal settings. (#1814)
- Fix API token expiration message (#1817)
- Fix Token Header Auth (#1863)
- Fix typo in rest endpoint data validation check (#1782)
- Change etlv2 endpoint config value used for migration etlv2 actions (#1826)
- Add logic to catch invalid datetime values in REST requests (#1896)
- Add default value for
modw_cloud.instance_type_uniontable (#1897)
-
Uncategorized
- Bug Fixes
- Cloud
- Updates to how end times are set for instance types.
- Data Warehouse Export
- Exporting an empty timeseries dataset will no longer throw
undefined offseterrors.
- Exporting an empty timeseries dataset will no longer throw
- ETL
- Fixed a rare race condition that was causing processes to not exit has been fixed.
- Added variable substitution for hierarchy info used in ETL table definitions and
aliasused in raw statistics configuration. - Updated the table definition for the
ReportTemplateChartstable to include an index.
- General
- Additional conditions that will force a reconnect on database connection loss have been added.
- Prevent the MySQL CLI from using user-specific options as the presence of a
.my.cnffile in a user's home directory can contain conflicting credentials.
- Job Viewer
- Fixed a bug that was causing the y-axis to go up to 120% ( even though the data only went to 100%).
- Metrics Explorer
- Updated the method by which dataset id's are calculated to use integers as opposed to floating point numbers.
- Cloud
- Data Analytics Framework
- New Feature
- API Token authentication / authorization has been added to XDMoD.
- The My Profile UI component has been updated to allow for the management of API tokens.
- New Feature
- Documentation
- Data Analytics Framework
- Documentation has been added for the Data Analytics Framework.
- Data Warehouse Export
- Documentation has been added that explains the data export batch process.
- Cloud
- FAQ has been updated with information about Rocky/RHEL 8 and PHP 7.2 information.
- General
- New NSF Grant numbers have been added.
- New XMS team members have been added.
- SimpleSAMLPHP links have been updated.
- Updated the text of the node_utilization How-To section to include language about needing to restart php-fpm in addition to httpd for changes to show up in the UI.
- Updated links to presentations stored in Google Drive/Docs.
- Data Analytics Framework
- Enhancements
- Cloud
- The Core Hour statistic has been changed to CPU Hours to match the Jobs realm.
- Data Warehouse Export
- Added a new configuration option to disable export for a realm.
- ETL
- Support for GPU count data in Grid Engine, specifically Univa Grid Engine now known as Altair® Grid Engine®
- the
mod_loggerdb schema has been updated to be managed by the ETL framework. - Added support for PBS logs that use
Resource_list.ngpusto indicate the number of GPUs used by a job. - Added an index to
mod_shredder.shredded_jobto speed up ingestion actions that depend on theqos_name
- General
- Updated the syntax for variable substitution in
rawstatistics.json, this brings it inline with the format used by the ETL configuration code.
- Updated the syntax for variable substitution in
- Infrastructure
- REST request logging has been supplemented by including POST data and total elapsed processing time.
- Job Viewer
- All charts have been migrated from using HighCharts to Plotly.
- Metric Explorer
- Legend text generation has been updated and improved.
- Support for API token authentication hass been added to a number of contoller end points.
- Cloud
- Internal Dashboard
- General
- ETL Overseer logs have been added to the
Logstab.
- ETL Overseer logs have been added to the
- General
- Maintenance / Code Quality
- General
- Removed dead code from the
xd_utilitiesnamespace. - Updates to support Chrome 109
- Removed dead code from the
- General
- New Features
- ACL
- Added the ability to group by's to be hidden for a specific statistic instead of a whole realm.
- Cloud
- A new dimension has been added,
vm_state. - Inactive VMs have been added to cloud realm aggregation.
- A new dimension has been added,
- Data Warehouse Export
- A new endpoint for retrieving raw data has been added.
- ACL
- Bug Fixes
- General
- Fix handling of filters where the filter string has a quote character in it (#1749)
- General
- Bug Fixes
- General
- Added a helper script that runs chromium to generate images. This will allow us to support headless chromium v111 and later (#1704)
- General
- Uncategorized
- Bug Fixes
- General
- Update image export to support headless chromium 109 or later (#1688)
- General
- Uncategorized
- General
- Merge code updates to support Rocky 8 into single branch that supports both Centos 7 and Rocky 8 builds.
- General
- Bug Fixes
- ETL
- Fix comment removal in ETL SQL execution (#1609)
- Fix cloud resource specs and storage database tables date formats (#1600)
- Add SHOW WARNINGS to StructuredFileIngestor (#1586)
- Fix quarters start timestamp column type (#1560)
- Change slurm helper default end time (#1546)
- Fix resource ingestor default resource type (#1537)
- Fix shredder empty line checking (#1525)
- General
- ETL
- Enhancements
- Internal Dashboard
- Refactor admin dashboard user listing query and add indexes to Users and SessionManager tables (#1606)
- ETL
- General
- Convert tables in moddb database to the InnoDB table engine (#1585)
- Convert tables in modw_aggregates to InnoDB and add class to manage aggregate tables (#1584)
- Add index to mod_logger.log_table for help with better query planning (#1582)
- InnoDB Performance Improvements for ingestion and aggregation (#1579)
- Convert tables in mod_logger, mod_hpcdb, and mod_shredder to InnoDB (#1576)
- Convert tables in modw database to the innodb table engine (#1573)
- Add support for PHP 7.2 and MariaDB 10.3 (#1486)
- Cloud
- Job Viewer
- Enabled text selection / copying in Job Viewer Accounting Tab (#1561)
- Infrastructure
- Update query filtering to be based on ACLs (#1531)
- Internal Dashboard
- Uncategorized
- New Features
- General
- Add script to help convert to using innodb-file-per-table mysql setting (#1598)
- Add quality of service group by with Slurm support (#1589)
- Updated etl_profile.js to add an extra group_by parameter "show_all_dimension_values". (#1588)
- Update Metric Explorer Controller to include ability to show all possible dimension values. (#1578)
- General
- New Features
- Enhancements
- Security
- Infrastructure
- Updating file permissions for XDMoD's bin directory (#1510)
- Switch to Chromium instead of the (no-longer maintained) PhantomJS to generate chart export images. (#1413, #1428)
- Remove (no longer maintained) jasper reports and replace with PHPWord/libreoffice for Report generation. (#1341, #1496, #1424)
- General
- Infrastructure
- Bug Fixes
- Data Warehouse Export
- Fix SUPReMM data warehouse export regression (#1520)
- General
- Change chromium output error detection. (#1519)
- Ensure "Don't show me this" checkbox is honoured (#1508)
- Ensure dashboard is updated whenever reports change. (#1507)
- Add missing attributes when gearing over from Usage to Metric (#1506)
- Improve input validation for download_report controller (#1503)
- Fix bug in language definition in main html page. (#1492)
- Change top-level exception handling in scripts (#1488)
- Fix mysql command line defaults file option (#1487)
- Re-index filtered arrays (#1481)
- Improve robustness of full groupby checking (#1469)
- Re-index value returned from getRawDataRealms function (#1468)
- Update version of ini package (#1467)
- Update SSO to pull the configured SP's IdP metadata directly (#1455)
- Only show the New User Tour Dialog once per session. (#1454)
- Fix login dialog sign in button rendering on Google Chrome 85+ (#1450)
- Set the "Show in X tab" label in the Metric Explorer automatically based on whether the dashboard is enabled (#1400)
- Job Viewer
- Cloud
- Updating pipelines that get run when ingesting data in generic cloud data format (#1514)
- Changing constraints on cloud resource specifications tables to prevent extra rows (#1511)
- Fix extraneous cloud sessions (#1504)
- Fixed memory range typo for cloud memory buckets configuration file. (#1502)
- Add columns to cloudfact_by_day aggregation (#1518)
- Break cloud migration into two pipelines (#1512)
- Move to using custom Ingestor for cloud resource specifications (#1489)
- Make cloud realm compatible with ONLY_FULL_GROUP_BY (#1480)
- ETL
- Infrastructure
- Updating the logging code to not throw away all class information (#1493)
- ACL
- Update moddb.module_version table (#1484)
- Internal Dashboard
- Fix institution loading during manual user creation (#1446)
- Report Generator
- Data Warehouse Export
- Documentation
- General
- Updated the upgrade guide documentation to include information on re-ingesting cloud data due to memory buckets bug fix. (#1516)
- Create Gateways realm installation instructions (#1509)
- Update simple SAML documentation to mention the icon. (#1505)
- Update listing of XDMoD demos from Autumn 2020 (#1482)
- Remove remaining deprecated html from About tab pages (#1476)
- Updated XMS project's authors and team listings (#1474)
- Update About tab with recent Publications and Presentations details (#1470)
- Removing references to acl-config's recover mode (#1418)
- General
- Maintenance / Code Quality
- Infrastructure
- General
- Data Quality
- General
- Improve data validity checks for Open XDMoD account usernames (#1379)
- General
- Documentation
- ETL
- Update ETL table data verification help text (#1405)
- General
- Remove old documentation (#1404)
- Minor documentation changes (#1402)
- Update node utilization documentation (#1397)
- Update hierarchy dimensions removal process (#1394)
- Remove beta labels from documentation (#1386)
- Update Integration Documentation (#1383)
- Update hierarchy documentation (#1376)
- Update code block syntax highlighting identifiers (#1367)
- Update PHP and MySQL version requirements (#1364)
- Update processor buckets documentation (#1359)
- Add GPU metrics documentation (#1355)
- More detail for SSO setup (#1354)
- Update documentation and example config for HTTPS by default (#1336)
- Update shredder documentation (#1315)
- Add roadmap link to documentation (#1197)
- Update MySQL configuration suggestions (#1157)
- Storage
- Update storage documentation (#1403)
- Cloud
- Cloud documentation update for 9.0 (#1273)
- ETL
- Bug Fixes
- General
- Improve CI validate script (#1399)
- Replace bitwise operator with logical operator (#1398)
- Rename internal dashboard button from "Dashboard" to "Admin Dashboard" (#1390)
- Allow the internal dashboard to load even if a user account has an empty name. (#1382)
- Fix groupby resource (#1374)
- Use piped logging to rotate Apache logs (#1371)
- Fix bugs in definition of cloud fact tables. (#1352)
- Fix bugs in definition of job fact tables. (#1349)
- Fix regression bug when paging datasets. (#1299)
- Fix bug in highchart wrapper code. (#1285)
- Fix bug viewing raw data as demo user. (#1284)
- Display error message instead of "Loading..." (#1280)
- Add updating of version in module portal settings ini files on upgrade (#1277)
- Remove unnecessary redirect in the job viewer rest stack. (#1269)
- Fix html decoding in Job Viewer -> Edit Search (#1268)
- Fix bugs in Configuration classes (#1221)
- Don't bother trying to set headers if not possible. (#1194)
- Update organization affiliation change email (#1180)
- Do not json_encode included file (#1141)
- Focus on username field when login dialog is expanded (#1115)
- Fix chart time display when the system timezone has a positive offset from UTC (#1035)
- Fixed loading the organization abbrevation from organization.json (#951)
- Cloud
- Minor Cleanup of the OpenStack event schema (#1396)
- Convert date for cloud resource specifications to correct timezone from UTC (#1391)
- Updates to cloud migration files to fix syntax issues (#1380)
- Change unit to percent for cloud core utilization statistic (#1373)
- Change table definition for vcpus field for the cloud_resource_specs table (#1361)
- Remove bug that causes stop events to create sessions (#1344)
- Move UpdateCloudProjectToPI action to jobs-cloud-ingest-pi pipeline (#1313)
- Improve performance of OpenStackCloudEventAssetRootVolumeIngestor and GenericCloudEventAssetRootVolumeIngestor actions (#1303)
- Add alternate_group_by_column field to Project and Instance Type group by (#1295)
- changing account, instance and instance_type tables to have global un… (#1272)
- Change account, instance and instance_type tables to have single unique auto increment column (#1263)
- Update openstack json event schema definition (#1249)
- Adding table name to remove ambiguity from group by statement (#1228)
- Update cloud event schema and coerce data (#1144)
- ETL
- Update SQL delimiters (#1384)
- Prevent LSF commands encoding warnings (#1377)
- Convert Slurm job names from UTF-8 to ISO-8859-1 (#1363)
- Throw expected exception when JSON is invalid (#1275)
- Remove foreign key constraints from MyISAM table (#1232)
- Fix typo in EtlConfiguration (#1222)
- Fix the underlying JSON configuration parsing problems (#1131)
- Make the Shredder ignore empty lines (#1088)
- Open OnDemand
- Add ability to apply filters to the aggregatedata request to suppport Open OnDemand integration. (#1372)
- Job Viewer
- Data Warehouse Export
- Report Generator
- Metric Explorer
- ACL
- User Dashboard
- Fix page size in JobComponent single user view. (#1195)
- Internal Dashboard
- Storage
- Fix storage realm username GroupBy (#1156)
- General
- Enhancements
- Metric Explorer
- General
- Adding an alert to notify users when ingestion has not been run. (#1389)
- Update PHP version check (#1366)
- Update publication and presentation listing for Open XDMoD About page (#1365)
- Remove Roles and UserRole tables (#1360)
- Enforce secure cookie flag for all session cookies and the REST token (#1356)
- Change default log file permissions (#1333)
- Update source installation log directory notes (#1331)
- Change default log directory permissions (#1328)
- Update xdmod-check-config script (#1327)
- Use absolute acl-config path in xdmod-ingestor (#1306)
- Update some wording for groupbys (#1297)
- Improve ingestor option error checking (#1227)
- Open OnDemand
- ETL
- Update Slurm shredder to ignore non-ended job states (#1362)
- Add PBS jobs realm GPU data (#1353)
- Add
job_id_rawto SGE shredder (#1317) - Change ETL logging identification strings (#1316)
- Check engine during foreign key constraint verification (#1259)
- Add additional ETL DB Model error checking (#1237)
- Data Warehouse Export
- Raw stats SUPREMM refactoring (#1343)
- Add README to data warehouse export zip files (#1300)
- Update data warehouse export UI (#1298)
- Improve data warehouse export logging (#1223)
- Minor change to Data Warehouse Export controller (#1166)
- Prevent creation of duplicate export requests (#1150)
- Refactor export request deleted state (#1149)
- Internal Dashboard
- Add index to log table (#1330)
- Infrastructure
- User Dashboard
- Move the job grid into its own class (#1283)
- Report Generator
- Close report generator "Save As" dialog after saving (#1279)
- ACL
- New Features
- Infrastructure
- Adding the new ubccr/simplesamlphp-module-authoidcoauth2 module (#1378)
- Cloud
- ETL
- General
- Infrastructure
- Qa / Testing
- Experiment
- General
- Gateways Realm dead simple start (#1262)
- General
- Dependencies
- Data Quality
- General
- Update statistics and groupbys for consistency (#1089)
- General
- Bug Fixes
- General
- Fix bug parsing the
resources.jsonconfiguration file when a single resource is defined in the file. (#1130)
- Fix bug parsing the
- General
- Bug Fixes
- Enhancements
- Infrastructure
- ACL
- Updated to add resource_types and resource_type_realms (PR #1006)
- Job Viewer
- General
- Performance improvements by providing caching for Configuration objects built from JSON configuration files (PR #950)
- ETL
- Allow re-use of StructuredFile data endpoints with external filters (PR #944)
- Qa / Testing
- General
- Add Usage Explorer Tests (PR #1062)
- Multi realm installs - Allow for modification of enabled realms… (PR #1004)
- Improve robustness of UI tests (PR #994)
- Update Travis YML to explicitly use Ubuntu 14.04 (PR #959)
- Fixing timing issues in Metric Explorer UI Tests (PR #953)
- Add Dockerfiles used for test builds (PR #935)
- Less stringent DataWarehouse Descripter tests (PR #926)
- enable all component tests and make sure they work (PR #902)
- Infrastructure
- Wait for loading mask to disappear on SSO logout (PR #927)
- General
- Documentation
- New Features
- Data Warehouse Export
- Add data warehouse batch export (PR #1010)
- Metric Explorer
- User Dashboard
- Added reset summary page layout UI (PR #982)
- Add JobPortlet (PR #976)
- Adding New User Help Tour and functionality to reset if a user has seen a tour or not (PR #971)
- Add recent charts and reports portlet (PR #968)
- Add report thumbnails portlet (PR #967)
- Add guided user tours (PR #962)
- User Dashboard - Center Report Card: Support (PR #943)
- User Dashboard - Summary Statistics Portlet (PR #930)
- Initial Prototype of Novice User Portal (PR #909)
- Data Warehouse Export
- Data Quality
- Bug Fixes
- Bug Fixes
- General
- Allow upgrade to finish if cloud realm schema has not been created (#882)
- General
- Documentation
- ETL
- General
- Cloud
- Updated Cloud Documentation for 8.1 Release (#875)
- New Features
- Enhancements
- General
- Add support for configurable email subject prefix (#872)
- Add node_modules to the RPM. (#835)
- Update config read order (#818)
- Add initial summary charts for the cloud realm (#803)
- Support simplesaml's internal session naming (#757)
- Support asynchronous loading of Usage tab thumbnail charts (#750)
- Update Job Viewer API to support multiple realms (#733)
- Support non-numeric values for Usage chart filter parameters (#716)
- Make "Show raw data" for multiple realms configurable (#706)
- Update Sign On Panel to collapse local login if SSO is enabled (#701)
- Cloud
- ETL
- Improve resiliency of ETLv2 manage tables (#807)
- Add storage shredder/ingestor support (#786)
- Support ETL '$include' directive (#785)
- Update Configuration class to support merging objects in local config files (#782)
- DirectoryScanner support for last modified time based on filename and/or directory (#780)
- Move the job performance postprocessing SQL to the aggregation pipeline (#770)
- Implement dynamic fact tables via ETLv2 for job performance ETL (#742)
- Add exception code to logAndThrowException (#719)
- Skip only records that fail verificaiton instead of rest of file (#714)
- General
- Bug Fixes
- Cloud
- Fix roles file comparison to use object and not string (#878)
- Update session_records timestamps to non-nullable for MySQL 5.7 support (#877)
- Add cloud raw tables to cloud manage tables action (#874)
- Change staging table to use 1 as unknown id instead of -1 (#866)
- Update GroupBys In Cloud Aggregate Table (#863)
- Change cloud person username fields to be not null (#860)
- Change where staging action gets user id from for cloud data (#845)
- Remove event_id from the event table primary key (#844)
- Prevent null usernames being added when ingesting cloud data (#838)
- Remove duplicate join statement that was causing 1066 Not unique table/alias error (#837)
- Fix for generic cloud datetime, and openstack instance type datetime (#820)
- Update cloud realm to not throw away event precision (#811)
- Guarantees a deterministic order for events received by the event reconstructor (#805)
- General
- Add proper namespace in VerifyDatabase (#819)
- Fix path to Exception in TimeAggregationUnit::factory (#810)
- Explictly check for stdClass in VariableStore initializer (#802)
- Make lastLogin time be floored to the second instead of microtime (#755)
- Make logrotate.d file not override global settings (#749)
- Fix security vulnerabities in job performance (#738)
- Allow search panel scroll bars on small displays (#702)
- ETL
- Update primary key on resourcefact table to improve cloud ingestion (#795)
- Throw Exception if lockfile could not be obtained (#793)
- Improve debugging messages when executing SQL (#783)
- Don't automatically rewind DirectoryScanner file handle (#768)
- Explicitly cast potentiall nulls to array for array_merge() (#756)
- Improve verification of resource codes (#720)
- Fix uncaught ETL exception in PdoIngestor (#718)
- Metric Explorer
- Fix refesh button for Metric Explorer (#740)
- Cloud
- QA / Testing
- Features
- General
- Added a beta version of the Cloud realm to provide metrics relevant to cloud computing resources.
- Added a beta version of the Storage realm to provide metrics relevant to storage systems installed at a center.
- Federated XDMoD has been released for production. Federated XDMoD allows individual, locally managed, XDMoD instances to report all or a subset of their accounting data to a central Hub which provides a global view of the federation.
- All XDMoD user profiles are now associated with an organization. Previously, this was only required for Campus Champions.
- Added support for automatically detecting / assigning a new SSO User's organization.
- Added support for automatically detecting if a user's organization has changed and updating their accounts accordingly. This may include, but is not limited to, the removal of elevated privileges.
- Hardened the login and password reset process as a result of a security audit by University of Cambridge.
- Improved support for resource manager job arrays.
- Many improvements to the documentation.
- ETL
- Reorganized several ETL pipelines.
- Improved data sanitization for tighter checks present in MySQL 5.7.
- Refactored Jobs realm ingestion to utilize ETLv2.
- Standardize action names to follow the format module.pipeline.action. For example, xdmod.acls.manage-tables.
- Added character set and coalition to table definitions.
- Added support for foreign key constraints.
- Added support for the definition of ETL variables on the command line using -d variable=value.
- Add ingestion of node hostname data from SGE logs.
- Various ETL performance improvements.
- General
- Bug Fixes
- User Interface
- Deep linking when logged in using SSO has been restored.
- Update the logrotate configuration to use the su and create options.
- ETL
- Add primary keys to select ETL source queries.
- When modifying an existing table, preserve the order of the columns in the definition file.
- Ensure that file handles are flushed before inserting the final chunk of data.
- Misc
- Fixed several exceptions that were outside of a namespace.
- Fixed an issue where ACLs were not properly created on upgrade.
- Several minor bugfixes
- User Interface
- Bug Fixes
- Properly implement data access for non-feature ACLs (e.g., ACLs that provide access to data but not a feature such as a tab) that are not part of the hierarchy. For example, the Value Analytics ACL.
- Features
- General
- Improve performance of Utilization statistic
- Do not embed JavaScript in chart objects returned by the back end, instead
- include this code directly in the user interface code
- General improvements to the performance of the REST stack
- Updated Google Captcha to v2 (v1 is now discontinued)
- Added numerous component and unit tests
- Removed unused code paths
- Update greenlion/PHP-SQL-Parser to newer release that does not autoload itself
- User Interface
- Legend item edit box is now displayed next to the legend item that is being edited and is larger to accommodate longer strings
- Added a "Select All" button to the filter dialog
- Added PDF export capability, which greatly improves the quality of images included in LATEX documents
- ETL
- When ingesting Slurm data, return duplicate jobs such as those that were resubmitted due to node fail conditions
- Improve performance of filter list generation
- Application Kernels
- Several minor bug fixes and UI improvements
- Bug Fixes
- General
- Fix the Show Guidelines button in the Usage tab
- Fixed an issue with data filtering when an unprivileged user tries to view timeseries data for a restricted realm or statistic
- User Interface
- Fix the "TypeError: element is undefined" error when plotting Pie chart in metric explorer
- General
-
Features
- General
- Added support for Globus as a federated authentication provider
- Improvements to the user login dialog and matching of users to institutions
- Added the ability to use fine-grained ACLs for controlling access to features such as tabs and realms as well as data
- Code pertaining to creating/sending emails has been moved to a central location.
- Improved online documentation for installations and upgrades
- Internal Admin Dashboard
- Updated user management functionality in the administrative dashboard to support fine-grained ACLs for individual users
- Many stability improvements to the administrator dashboard, especially for managing users
- Storage Realm
- Added alpha version of the Storage realm to track resource storage utilization
- ETL
- Added an ETL pipeline for ingesting log files generated by Eucalyptus clouds
- Added support for PBS/Torque logs where the host of a job is not included in the job's ID string
- Support references into complex source records such as JSON objects
- General
-
Bug Fixes
- General
- Improved error reporting
- Fixed several issues where JSON was not properly encoded
- For end dates that fall on the current date or in the future, do not automatically adjust the date to the end of the aggregation period that it falls into.
- Report Generator
- Make timeframes editable for individual charts
- ETL
- Ignore duplicate hosts found in LSF accounting log files
- Don't verify data endpoints associated with disbaled actions
- Always regenerate source data queries prior to execution to ensure that any modified ETL variables are properly applied
- Update host list parser to ensure that empty host names are not returned
- Remove PHP memory limit when running ETL pipelines
- General
- Bug Fixes
- General
- Fixed compatibility with PHP 5.3.3 (#269)
- General
- Features
- General
- Enhanced authorization framework (#97, #146, #206)
- Improved login prompt to automatically appear when an unauthenticated user attempts to access any private tab (#110)
- Improved design of Metric Explorer Load Chart menu (#144)
- Re-enabled aggregate mode for "Wall Hours: Per Job" metric (#186)
- Added quarterly report template for center directors (#199)
- Improved support for third-party PHP libraries used by modules (#205)
- ETLv2
- Job Viewer
- General
- Bug Fixes
- General
- Fixed various compatibility issues with PHP 7 (#101, #183)
- Fixed handling of 4-byte UTF-8 characters during XRAS ingestion (#122)
- Improved handling of invalid start and end dates received by API (#160)
- Fixed validation of length of names in contact forms (#175)
- Improved handling of jobs with "0" start or end time (#197)
- Fixed case where About tab would display a blank page when loaded using Chrome (#232)
- Job Viewer
- Metric Explorer
- Usage
- General
- Miscellaneous
- Moved Node.js ETL framework to Open XDMoD repository (#106)
- Fixed build script running out of memory allocated by PHP (#118)
- Performed work in anticipation of federated instances (#148)
- Improved development workflow (#124, #157, #195)
- Improved quality assurance (#107, #116, #134, #143, #150, #163, #165, #169, #173, #184, #185, #187, #190, #193, #194, #198, #212, #235)
- Cleaned up old and/or unused code (#104, #105, #109, #112, #117, #128, #158, #159, #182, #191, #213)
- Improved documentation (#161, #202, #203, #229, #247)
- Features
- General
- ETLv2
- Added per-pipeline ETL locks (#10)
- Enhanced multi-host aggregation (#13)
- Added read support for Oracle endpoints (#34)
- Improved support for running pipelines over all dates (#77)
- Improved transformation support (#80)
- Added a tool for comparing SQL tables (#78)
- Added other improvements (#43, #45, #84, #90, #92)
- Bug Fixes
- General
- Fixed warning that could appear when using federated authentication (#19)
- Fixed unnecessary rounding in processor count statistics (#25)
- Fixed errors when attempting to preview an unsaved report (#26)
- Fixed aggregators excluding data on certain time boundaries (#47)
- Fixed handling of backslashes when using certain MySQL features (#52, #53)
- Fixed non-aggregate metrics being allowed in aggregate datasets (#74)
- Fixed REST API throwing non-standard errors (#87)
- Fixed display error with password reset form that occurred when using federated authentication (#108)
- Fixed "Show chart title" option in Usage tab not working as expected (#139)
- Metric Explorer
- Fixed Y-axis context menu disappearing after changing between linear and log scales (#12)
- Fixed typing certain characters causing chart options menu to close when using Firefox (#119)
- Fixed chart errors not displaying if help graphic is active (#121)
- Fixed chart last modified time not updating (#140)
- General
- Miscellaneous
- Performed work in anticipation of allocations/accounts data in Open XDMoD (#11, #50)
- Performed work in anticipation of cloud data in Open XDMoD (#68, #75)
- Cleaned up old and/or unused code (#54, #55, #56, #57, #73, #88)
- Improved logging (#29, #40, #46, #50, #66, #98)
- Improved quality assurance (#21, #27, #28, #35, #41, #48, #58, #67, #76, #79, #93, #94)
- Improved documentation (#32, #37, #44, #71, #103, #115, #123, #130, #132, #135)
- Features
- General
- Modified the Summary tab to reload automatically after its charts are modified in Metric Explorer.
- Modified REST stack to support multiple configuration files, allowing modules to supply their own REST resources.
- Increased maximum size of node lists in database, allowing jobs running across more nodes to be more accurately tracked.
- Improved efficiency of job host list parser and ingestor.
- Added ability to disable Basic Auth in REST API.
- Improved email validation.
- ETLv2
- Added new PHP-based ETL system to Open XDMoD.
- Vastly improved logging and debugging.
- Supports individual ETL actions as well as ordered sets of actions called pipelines.
- Optimized for performance when importing large numbers of records.
- Predefined actions support flexible ingestion of data from multiple sources.
- ETL is configured via JSON files and requires far less code modification when customizing to more easily support flexibility at customer installations.
- General
- Bug Fixes
- General
- Fixed stacked area charts not handling empty data points correctly.
- Stopped inaccurate warning about Adobe Flash content in browsers that block Flash. (XDMoD does not make use of Flash.)
- Fixed various typos and grammatical errors.
- Fixed incorrectly-oriented subpanel titles in Internet Explorer 10.
- Downgraded PHP packages that required PHP 5.3.9 instead of Open XDMoD's current minimum PHP version, 5.3.3.
- Fixed inconsistency between labels used for users and PIs on a job.
- Fixed automatic aggregation unit selection for charts not working correctly on servers running PHP 5.3.3.
- Stopped package builds from being logged to an Open XDMoD database.
- Fixed documentation files from modules overwriting core files when installing using tarballs.
- Fixed UGE shredder.
- Fixed Slurm job array index parsing.
- General
- Refactors and Miscellaneous
- Added documentation for using LDAP for federated authentication.
- Spun the App Kernels, SUPReMM, and XSEDE modules out into separate code repositories.
- Began transition to a more flexible, more efficient data warehouse that will support innovative HPC resources including cloud computing resources.
- Consolidated third-party JavaScript libraries into one library directory.
- Improved development setup process for external contributors.
- Migrated website from a single SourceForge site to multiple GitHub sites.
- Cleaned up and reorganized numerous other assets.
- Added
reqgresandreqtresto the list of Slurm fields that are shredded and stored in the Slurm job table.
- Important Notes
- Updated Highcharts from v3.0.9 to v4.2.5.
- Commercial users (as defined by the Highcharts license terms) will need to acquire a new Highcharts license if their current license does not cover the new version.
- Updated Highcharts from v3.0.9 to v4.2.5.
- Features
- Added demo user type.
- Added roadmap link to feature request dialog.
- Shredders
- Fixed calculation of missing end times from the start time and wall time.
- LSF
- Now storing the node list, exit code and exit status for SUPReMM support.
- Slurm
- Now allowing null eligible times.
- Failed parsing of datetimes are no longer a fatal error.
- SGE
- Now supporting older versions of SGE that contain only 43 fields in their accounting logs.
- Bug Fixes
- General
- Fixed some charts not rendering if more than 1000 points were plotted.
- Improved consistency of chart aesthetics across tabs.
- Fixed report generator not cleaning up old report files.
- Metric Explorer
- Fixed Add Filter list only including filter types applicable to the realm of the first data series on a chart.
- Shredders
- PBS
- Fixed parsing of newer
exec_hostformats (e.g. host/0-3, host/0*8).
- Fixed parsing of newer
- PBS
xdmod-import- Fixed importing of hierarchy and group-to-hierarchy files.
- General
- Refactors and Miscellaneous
- Refactored Open XDMoD to be the code base upon which XDMoD is built instead of the other way around.
- Updated jQuery from v1.9.1 to v1.12.4.
- Updated Node.js Mongo driver from v1.4 to v2.1.
- Improved tab loader to allow tabs to be defined in configuration files instead of being hard-coded into the loader.
- Merged public and private versions of the main page into a single page.
- Added options to more quickly build Open XDMoD for development purposes.
- Reduced duplicate code in asset setup script.
- New Features
- Federated Authentication
- Open XDMoD can now use any authentication system supported by SimpleSAMLphp (https://simplesamlphp.org/).
- For more information, please view the Federated Authentication guide.
- Roadmap
- Added interactive roadmap for XDMoD development to About tab.
- Contact Us
- Added option to contact developers for technical support.
- Data Warehouse
- Improved speed of filter (dimension value) lists by precomputing available values.
- Metric Explorer
- Added redo button.
- Shredders
- Added support for PBS Pro.
- Added additional input validation to the Slurm helper script.
- Upgrade Process
- Added warning to upgrade addon modules before migrating.
- Database
- Changed column types and added indexes to the logging table.
- Documentation
- User manual now available.
- Improved installation and configuration documentation.
- Federated Authentication
- Bug Fixes
- General
- Enabled toolbar scrolling when window is too small.
- Fixed trend line R^2 values sometimes not being calculated.
- Removed redundant warnings about role-based restrictions.
- Fixed the About tab not being able to be the default tab.
- Improved chart loading speed by removing most animations.
- Fixed incorrect trend line calculation and display for discontinuous data series.
- Fixed security vulnerability in Report Generator's image generator.
- Fixed extra slashes in URLs in password reset emails (including account creation emails).
- Fixed email template paths.
- The About tab can now be set as the default module.
- Data Warehouse
- Fixed cases where current time period would not be aggregated.
- Now that this is resolved, code that can work by scanning any aggregate table now uses the year tables, as those are smallest.
- Improved consistency of time period tables and removed unused ones.
- Fixed cases where current time period would not be aggregated.
- Metric Explorer
- Fixed handling of HTML characters in chart titles.
- Fixed issues with axis renames not persisting.
- Fixed Load Chart search not working.
- Modified chart options menu label to be a fixed length.
- This prevents the button from being placed in the toolbar's overflow area when it shouldn't be.
- Fixed changes to y-axes not being undoable.
- Modified "Show Trend Line" button in chart options to be disabled when trend lines aren't available.
- Fixed case where duration selectors in main toolbar and on x-axis popup could desynchronize.
- Shredders
- Fixed handling of "NONE" in SGE's
pe_taskidfield. - Now allowing all values that were previously resulting in fatal errors in Slurm fields that represent lengths of time.
- Fixed handling of "NONE" in SGE's
- Upgrade Process
- Fixed broken error for when portal_settings.ini is not writable.
- Usage
- Fixed bad default settings for dimensions that default to pie charts.
- Fixed long initial load time by starting on a single-chart page.
- General
- New Features
- SUPReMM support
- Metric Explorer Changes
- Improved management of charts
- Charts are no longer automatically saved after each operation. A manual Save button has been added that will indicate whether or not a chart needs to be saved.
- When displaying pie charts, only a single dataset is allowed.
- Added ability to rename saved charts.
- Added undo capability.
- Improved UI for data filtering
- Added Quick Filters for quickly restricting data. This replaces the Role selector.
- Improved data access controls
- Non-public metrics and dimensions are now restricted according to users' roles.
- Improved filter lists to only include values for which the user can view the underlying data.
- Added the ability to include a remainder data series.
- Improved management of charts
- Migrated more server calls to the new REST stack
- Simplified the chart/data retrieval infrastructure used by the Metric Explorer and Usage tabs
- Improved performance of certain queries
- Updated PBS parsing to support time fields that are formatted in seconds and not HH:MM:SS
- Bug Fixes
- Fixed report generator document formatting
- Fixed inconsistent data series ordering between timeseries charts when sorted by value in Usage and Metric Explorer tabs
- Fixed Metric Explorer sometimes loading the same chart twice
- Fixed the "Hide Tooltip" option not always working as expected in Usage tab
- Fixed filter lists occasionally getting stuck under a loading mask
- Fixed predefined date durations not being saved correctly in the Metric Explorer
- Removed auto-loading of the last open chart in Metric Explorer, a new chart is now opened when visiting the Metric Explorer
- Improved stability, consistency, and performance of numerous other components
- Prevent warning messages produced by LSF jobs that don't have a host list
- New Features
- Utilization metric may now be normalized to a specified percent allocated
- Removed javac dependency
- Add host filter shredder feature
- Improved stability of Usage tab
- Improved error handling for internal dashboard errors
- Added meaningful HTTP status codes to failed login responses
- Resolved frequent warnings in Apache logs about implicitly converting arrays to strings
- PBS memory value parsing errors are no longer fatal
- Removed Apache requirement for listening on 127.0.0.1
- Added support for Univa Grid Engine 8.2+
- Shredders now skip duplicate data
- Added support for MySQL 5.6+
- Added support for PhantomJS 2.0+
- Added helper script for updating resource specs
- Improved documentation
- Bug Fixes
- Adjust LSF unique key constraint
- Added PBS unique key constraint to prevent duplicate job data
- New Features
- Remove excessive "wall time" error messages
- Add more debug logging
- SGE memory value parsing errors are no longer fatal
- Bug Fixes
- Fix SGE parsing regression
- New Features
- Support for Torque's new "exec_host" syntax
- Updated SGE parsing to support Univa's new "job_class" field and requests for "INFINITY" memory
- Bug Fixes
- Backslash characters in LSF log files now parsed correctly
- Report generator now handles double quotes in portal_settings.ini properly
- Fixed optional Node Utilization statistic
- New Features
- Major re-write of the Metric Explorer user interface including a metric catalog and context-sensitive menus
- Report generator performance enhancements
- General user interface updates
- Improved image export dialog
- Added metric description pane in Metric Explorer
- Improved session management
- Added additional error checking and reporting to shredders
- Added LSF support
- Improved utilization metric to support resources that don't have a constant cpu count over their lifetime
- Added suport for alternate PI data sources
- Improved logging
- Added weighted job size statistic
- Improved IE compatibility
- Added xdmod-admin script
- Added automatic update checker
- Added support for slurm job arrays
- Improved documentation
- Bug Fixes
- PBS and SGE job arrays were not ingested correctly
- sacct configuration option was not recognized
- Arrow keys did not work in axis range boxes
- Parentheses not available in Metric Explorer chart titles
- Proper handling of apostrophes in search dialogs
- On some charts, data points on the last day of the specified date range were not displayed correctly
- Initial public release