- Add optional offset regex capture group to message id in asapo worker
- Fix stopping old instances for detectors that moved to other beamlines
- Add timeout to ldap search
- Add ASAPO transfer client service
- Build packages for RHEL9
- Build packages for Debian 12
- Delay emitting watchdog events for empty files
- Build package for Debian 11
- Check regex validity in AsapoWorker constructor
- Fix error storing the first file in a non-existing directory in http fetcher
- Write data to a temporary file first and rename to target file when finished
- Isolate plugins in a separate thread
- Add --config_file option to control server
- Fix shutdown of asapo producer
- Fix data loss due to unintended deletion of files when data downloading failed
- Update ASAPO plugin to version 21.03.0
- Centos 7, Debian 10, and manylinux1 packages use Python 3
- New history_size=-1 setting for http events to enable almost immediate reuse of filenames (a previously downloaded file with the same name will be overwritten)
- Subprocesses now use descriptive names shown by ps, pgrep, etc.
- Fix data loss due to unintended deletion of files when data storing failed
- Fix inotifyx events for nested subdirectories
- Fix watchdog events causing the taskprovider to hang forever on shutdown
- Fix shutdown of inotifyx eventdetector
- On Python 3 multiprocessing start method spawn is used
- Set default values of store_data and remove_data to True for control client
- Fix control client command line argument for selecting detector API version
- Fix handling of null return value in detector API version 1.8.0
- Fix hanging when connecting to offline host
- Fix unit tests
- Add docker-based end-to-end tests
- Change GIDs as well when changing UID
- Fix stopping Hidra processes in hidra.sh
- Update list of beamlines
- Fix restarting running instances via control client
- Fix missing space in start script
- Add missing dependency on python-future
- Increase wait time for control server to avoid acquiring incomplete configurations
- Add support for multiple ldap server
- Fix receiver checking scrip for newer zmq versions and improve Error messages
- Fix dockerfile for centos package building
- Fix with_confirmation for http_fetcher
- Fix stopping of http fetcher when handling huge data files
- Fix memory problems for python3 regarding logging
- Improve string to logging level convertion
- Fix log level propagation to processes
- Fix miscellaneous typos
- Fix hanging processes on shutdown
- Use general multiprocessing event to improve stopping
- Add configuration documentation for modules
- Improve check and error message if module not found
- Restructure project
- Add first version of a setup.py file
- Enable ldap search on centos6 (dectris) and Windows (switch from ldapsearch to python ldap3)
- Cache DNS lookup results
- Enable verbose logging as flag in hidrs.sh
- Use default Queuelistener of python3
- Add timestamp to error message of status request
- Suppress reoccurring communication warning
- Reduce log messages in debug mode
- Improve error message for module loading
- Improve shutdown message for remaining files
- Improve error handling for non-prio sending: do not stop serving of other applications if one has problems
- Improve code readability and clean up code
- Test on confirmation only if stream is enabled
- Remove DataHandlingError and exchange it with existing DataError
- Integrate plugin support into datareceiver
- Add option for detector specific config file for control client
- Support eiger2 filewriter
- Add possibility to explicitly configure eiger urls
- IPv6 compatibility
- Python 3 support
- ASAPO support (asapo api version 20.03) with and without substreams
- Fix memory leak in signalhandler for python2
- Cache DNS lookup to reduce DNS lookups and fix delay problems
- Enable request relaxed only if timeout is set
- Fix ignore_accumulated_events
- Make timeout of taskprovider configurable and possible to disable
- Fix zmq exception after timeout in taskprovider
- Fix typo in function call transfer
- Fix misformatted exception messages
- Improve error message when checking thread does not start
- Fix datadispatcher unittests
- Clean up code (pep8) and fix typos
- Fix typo in reply code of control server (ALREADY_STOPPED)
- Fix package dependencies (Red Hat, Debian)
- Catch wrong target definition in transfer
- Fix unittests
- Improve error message if detector_id is needed
- Fix stopping of still running instances
- Fix getinstances in hidra-control
- Add example for multi-detector use
- Fix config mapping in get_receiver_status
- Add debian 10 support for package build script
- Fix debian package build: fix conflict and missing source
- Fix centos package build to match spec file
- Fix ownership of log file of datareceiver
- Fix debian package dependencies
- Fix downloading of files in chunks for http fetcher
- Fix links in readme
- Fix links in package builds and docker images
- Fix minor bug in packaging for CentOs
- Add user change option to control server config
- Add example how to use statserver (config and communication)
- Fix config uniqueness for multiple detectors
- Fix communication of control server and StatServer if they run as different users
- Fix closing of file descriptors if anything goes wrong when storing data
- Fix RPM package building
- Move ext_ip into base_sender config
- Fix bumbversion script
- Fix backwards compatibility to yaml config type
- Fix with_confirmation when mixing python2 and 3
- Fix debian package building
- Fix data removal for multi chunk files
- Fix dispatcher name in logging
- Add pid of hanging process into log
- Simplify parameter handover in custom modules
- Fix misspellings and typos
- Fix pid display if hanging
- Fix hanging of datadispatcher on shutdown
- Fix version check between remote and local version
- Fix CPU usage of watchdog events (on linux)
- Fix freezing into executable
- Add file name information to watchdog event log
- Add information about registered streams to log
- Add option choose tag in debian build script
- Fix job forwarding from http events to cleaner
- Move, clean up and structure of utils into separate module files
- Use hierarchical config format for sender, receiver and control server
- Changed default config to yaml
- Improve test suite to handle individual modules
- Remove scripts to build SuSE packages via docker (new kernels incompatible with SuSE 10)
- Fix python3 compatibility
- Change _version_ to str
- Fix hidra usage via docker
- Enable mixture of netgroup and host list config for whitelist
- Rename ModuleNotFound to NotFound for versatility
- Remove hard dependency on logutils in python3
- Remove dependency on six
- Add script to build CentOS packages via docker
- Remove manual .egg tracking for setproctitle when freezing
- Changed effective user id inside of datamanager and datareceiver directly
- Removed underscore from event_detector and data_fetcher
- Renamed event_det_port and data_fetch_port
- Simplify log setup (no separated calls for with or without onscreen)
- Fix stopping of datadispatcher and taskprovider
- Event detector: Check monitored_dir when stopping
- Simplify inotifyx event detector
- Remove check for closed file in metadata mode
- Enable display of hanging processes on shut down
- Add inotify event detector
- Fix start up of eventdetector
- Remove restriction to supported modules only
- Add experimental ewmscp (kafka) syncing event detector
- Add data fetcher only distributing metadata (no data)
- Enable modules which do not need conf parameter
- Change default config for Pilatus and on Windows
- Fix log file permission problem if user starting hidra and the one as which is runs differ
- Enable random port usage for com_port and request_port
- Reduce number of workers when using Eiger (from 32 to 16)
- Fix netgroup resolving when result is not a fully qualified domain name
- Fix error message if writing fails (http_fetcher)
- Fix hanging processes due of missed control signal
- Control: Fix unicode compatibility
- Control client: show already running instances when start fails
- Control-server: Fixed file handle exception handling
- Extracted control server and control client config into config file
- Generalize control signals reaction
- Add host checks on control server
- Enable stopping of still running instances via control client
- Add option to show running instances via control client
- Improve error message when usage of control server
- Structured config server
- Remove config file when hidra is stopped via control server
- Introduce statserver to expose config (locally only)
- Enable control server and statserver communication
- Enable setting of beamline in config file of control client
- Fix starting of wrong beamline instances via control server
- Fix permission checking in control server
- Use one backup file per beamline
- Transfer: Checking chunk number before opening file
- Enable port receiving via transfer api
- Fix double query for request port in transfer API
- Fix debian package building via docker
- Fix freezing into executable
- Remove user change as default value from sender
- Fix version check between remote and local version
- Fix CPU usage of watchdog events
- Fix crash of signal handler when removing multiple leftover connections
- Fix pathlib2 dependency in freeze_setup, rpm and deb packages
- Fix requests with target groups
- Fix stop command for redhat init script
- Remove default config loading in hidra.sh
- Fix debian package building via docker
- Fix reading of events for watchdog event detector
- Fix typo in http_fetcher
- Fix with_confirmation on Windows
- Fix python3 compatibility
- Add workaround for log file rotation on Windows
- Add debug information for WindowsError
- Fix sending of data multiple times (after crash+restart of application)
- Fix watchdog on_move events
- Fix double sending after wake up
- Ensure that stop of datadispatcher is communicated
- Undo ignoring of accumulated events during sleeping
- Fix control server crash when backup dir missing
- Fixed MOVE_TO events in watchdog event detector
- Fixed hanging of datadispatcher
- Fixed missing events in watchdog event detector
- Fixed control signal reaction during data handling
- Fixed confirmation socket reconnect after wakeup
- Control-server: Fixed file handle exception handling
- Fixed debian package build
- Restart hidra instances on control_server start
- Control_client: fixed detector argument (fqdn)
- Control_server: Add detector id to procname and log file name
- Fixed windows startup and cx_Freeze 5.x compatibility
- Fix bumpversion for multiple changelog entries
- Fixed LD_LIBRARY_PATH for receiver status (init script)
- Transfer: Fixed timeout in get command
- Transfer: Fix started connection bookkeeping
- Receiver: Fixed deactivated whitelist
- Show receiver status in status message of hidra.sh
- Fixed typo in suse build Dockerfile
- Added branch support in suse build
- Added verbose output if ldapsearch fails.
- Fixed slow stopping of receiver when netgroup_check_time is high
- Fixed None values in ipc address cleanup
- Fixed version checks
- Receiver: handle empty LDAP return lists
- Fixed usage of netgroup in whitelist for control API
- Do not start cleaner when no fix data stream is active
- Fixed combination store_data enabled and with_confirmation set
- Changed ipc cleanup: reuse attribute instead of redefining paths
- Fixed directories where to find build packages
- Fixed backward compatibility to 4.0.x versions
- Fixed subdir creation in http_fetcher
- Fixed get in transfer API if timeout is reached
- Fixed debian package naming
- Fixed log dir permissions for debian packages
- Fixed building script for suse 10
- Added building script for debian
- Improved init script: debug option, more info when failing
- Added HiDRA Control Server to fallback script
- Fixed with_confirmation to check all chunks
- Fixed beamline to host mapping
- Fixed Windows environment
- Fixed subdir creation in transfer API
- Fixed confirmation endpoint
- Fixed hidra control client and datamanager default config values
- Init script: Fixed status output + changed commands
- Control client: Removed not supported argument target
- API: Fixed socket creation in control
- Fixed default config
- Fix config_file option in hidra.sh
- Fix freeze startup
- Added exampled
- Added debian package build scripts to bumpversion script
- Fixed undefined variable error in datamanager shutdown
- Fixed debian package build scripts
- Added automatic freeze build for suse 10 in docker container
- Changed default value for whitelist on Windows to localhost
- Fixed start up if fix_subdir cannot be created
- Fixed windows paths parsing in config
- Fixed endpoints in forwarder device
- Fixed: no ipc cleanup for Windows
- Fixed: startup problems on Windows
- Fixed version number
- Fixed Fixed bump version config for HISTORY.md
- Fixed spec file
- Fixed typo in LD_LIBRARY_PATH
- Fixed auth
- Fixed LD_LIBRARY_PATH in initscript for all platforms
- Fixed freeze (newline went missing in cleanup)
- Run hidra as user defined in the config file
- Added netgroup support in datamanager whitelist
- Added option to create fixed subdirs in monitored directory
- Added fix_subdirs awareness to http_events
- Do not create fix_subdirs in data_receiver
- If store_data is disabled requests for metadata gets an error
- Converted hosts to fully qualified domain names
- Added unittests
- Fixed sys path extension (prepend instead of append)
- Fixes stopping of datareceiver if start of transfer failed
- Fixed signal exchange with transfer API
- Fixed file loss when shutting down receiver
- More detailed response from SignalHandler to API (multipart)
- Fixed datareceiver: Include error message in log file
- Fixed return code if datareceiver stops because of an error
- Fixed memory leak in inotifyx -> version requirement of inotifyx
- Moved ldap uri into configuration files
- Changed usage and added additional options in hidra.sh
- Fixed help message in init script
- Use modified init script also for running frozen code
- Use current directory when init script is called in executable mode
- Added additional remove data strategies (stop_on_error and with_confirmation)
- Split config file (ports/internal settings vs user settings)
- Reopen not closed file if a first chunk is received again
- Added api calls to get and reset the receiver status
- Freeze: Create and use exe script
- Transfer API:
- Added method to get remote version
- Added regex support
- Added appid to signal exchange
- Added getter and setter for appid
- Renames QUERY_METADATA into QUERY_NEXT_METADATA
- Fixed get for intermixed chunks + 'reopen'
- Renamed get to get_chunk and added get method
- Fixed searching path for service file in control server
- Changed defaults for store_data and remove_data in control client
- Removed check for fix_subdirs in local_dir
- Disabled interactive shell in systemctl call (control server)
- Fixed missing files in spe
- Adjusted host names in mapping for control API
- Run control server as user hidra
- RPM: Split single package into multiple subpackages (hidra, python-hidra, hidra-control-client)
- Fixed logging in transfer when logutils is used
- Updated requirement version of zmq in spec
- Transfer API responds to confirmation requests after data handling
- Control server sets subdirectories <commissioning|current>/scratch_bl and <commissioning|current>/raw instead of commissioning and current
- Path names have always unix format (even when send from windows)
- Added parameter test to DataManager
- Removed setup method in data fetchers
- Fixed data socket reconnection after netgroup change
- Notice netgroup changes during runtime
- Fixed relative path starting with a backslash in transfer
- Changed structure of event detectors to inherit from a base class
- Converted data fetchers to classes which inherit from a base class
- Added first versions of event detectors and data fetchers for using HiDRA in series
- Fixed file closing if message size equals chunksize (Transfer)
- Renamed Eiger specific parameters to be more generic
- Removed old beamline specific control clients
- Renamed options in hidra control
- Added support for configuration of multiple detector with one HiDRA control
- Added procname to init script
- Added msi building support for freeze
- HiDRA control: Exchanged normal socket communication with ZMQ communication
- Added Dockerfiles for sender, receiver and API usage (based on Ubuntu)
- Added platform specific config file to freeze
- Changed meaning when whitelist is to None in datamanager config
- Added option "getsettings" to init script (SuSE)
- Fixed start of transfer API if one host in whitelist id not known
- Metadata dict as call by reference for datafetcher modules
- Automatic beamline extraction from config dict (in _constants.py )for hidra control
- Fixed examples and generalized them
- Added test file for hidra-fs vs API
- Added first version of hidra filesystem based on FUSE
- Changed config parameter for data stream target (one list instead of two single parameters)
- Added script to check if receiver is running
- Added beamline to receiver init script
- Closing still open files on shutdown (transfer)
- Added GPFS fallback script
- Receiver runs as pxxuser
- Fixed poller shutdown in transfer API
- Fixed bumpversion usage
- Changed log directory to /var/log/hidra
- Cleanup of freeze setup
- Reduced config file for Pilatus detector to minimum
- Added options for handling the removal of the source data:
- regular check of file writing status of receiver
- only with confirmation for each file
- Added and used cfel_optargs for config parsing
- If the receiver is not responding all processes in the datamanager enters sleep mode until this is resolved
- Added init script for SuSE
- Added type checking for parameters in config
- Made existence of fixed subdirectories mandatory
- Enabled differentiation between 32 and 64bit architecture when freezing HiDRA
- Added HiDRA receiver unit file
- Fixed parallel hidra control client usage of script and permanent open connection (e.g. Tango)
- Added option to get configured settings to hidra control client
- The connection list is now contained in a separate module
- Automatic bump versioning
- Added generic hidra control client with included authentication (host-based)
- Clean up and consolidation of helper functions
- PEP8 compliance
- Make API classes available at the package level
- Fixed init script for Ubuntu and Debian
- Reordered directory structure
- Added DNS name support
- Added support for building RPMs
- Added automatic argument and config parsing (and fixed bugs in parsing)
- Added Python 3 support
- Renamed APIs and API arguments
- Changed process name in init script
- Building executables with cx_Freeze
- IPC directory is set to world/write readable when created
- IPC directory is removed (if empty) when HiDRA is stopped
- Changed config file to use file mode by default
- exchanged all cPickle calls with json calls
- renamed init script to hidra.sh
- Added functional tests
- Fixing memory leaks
- Minor bug fixing
- Working C-API for data transfer (nexus use case only)
- Working C-API for data ingest
- Removed PyTango dependency
- Fixed file opening (only open and close once and keep it open)
- Example client (based on API) for controlling HiDRA remotely
- The communication with the controlling-server can be done with an API
- HiDRA can be controlled (start, stop,...) via an addtional server (HiDRAControl...)
- Working version of dataIngestAPI with dataTransferAPI (python)
- Merged nexusTransferAPI into dataTranferAPI
- Handling of open connections to known hosts with different send configurations
- Changed socket format in dataIngestAPI to IPC (on Linux)
- Fixed receiving of CLOSE_FILE messages in nexusTransferAPI
- Fixed getMetadata if no targets are specified
- Fixed IPC socket cleanup
- Tracker in ALIVE_TEST only used if ZMQ version is higher than 14.5.0 due to error in older ZMQ versions
- Fixed double adding to watch in InotifyxDetector
- Choose config file for dataReceiver
- Removed timeout warning in dataTransferAPI
- Fixed log rotation
- Fixed filename sending for requests (unicode problem)
- Fixed missing metadata if no target is specified
- Fixed parallel directory creation attempts
- Added method to dataTransferAPI to manually stop streams/queries
- Added option to specify which file formats to be send via zeromq
- Added option look for multiple event types in parallel (combined with file suffixes)
- DataManager can now be controlled via tango
- Added systemd service script
- Added cleanup arguments into config file
- Files get accessed only if data or metadata is send via zeromq
- Fixed DataReceiver (no shell)
- Added command line argument error handling
- Removed ringbuffer remains of old architecture
- Fixed data receiving problems with dataTransferAPI due to ZAP
- Fixed stopping: The service is shut down if one process dies
- Enabled whitelist for data receiver
- Added tests to check status of fixed data receiver
- Added init script
- Fixed clean up after shut down
- Enabled combination of data receiver whitelist with ldapsearch
- Added option to enable a clean up thread which checks the directory for missed files
- Version check does not consider bugfixes anymore
- Fixed copied file removal (Part 2)
- Fixed copied file removal (Part 1)
- Fixed too high processor usage
- Fixed suffix check in treewalk after creation of directory
- Fixed error handling with incorrect whitelists
- Fixed version checking
- Added file create time to metadata
- Added file removal safeguard
- Enabled use of IPC internal communication for Linux nodes
- Added exception definitions for dataTransferAPI
- Misc bug fixing
- Added functionality to get Data via HTTP Get
- Redesigned architecture
- Initial implementation