DEPRECATION NOTICE: Support for RHEL 5/CentOS 5 is planned to be removed in the next version of Breeze. Please contact support if this will cause you any problems.
Switch to Altair Units based licensing. Breeze will now checkout 30 HPCWorks units whilst running. Set BREEZE_UNITS=false to use an old feature license if needed.
1.2 Other Changes:
Added RPM and Debian packages for aarch64 architecture.
Added documentation surrounding Singularity/Apptainer mounting of /tmp and/or /var/tmp.
Fixed bug in monitor causing it to wrongly output an error when dealing with bind mounts.
Fixed a problem where a multithreaded program could deadlock if the main thread exits without waiting for the other threads to complete.
Breeze now measures the following aditional libc functions: execveat, _Exit, fctnl64, fstatfs, fstatfs64,fstatvfs, fstatvfs64, futimens, preadv, preadv2, preadv64, preadv64v2, pwritev, pwritev2, pwritev64, pwritev64v2, quick_exit, statfs, statfs64, statvfs, statvfs64, utimensat and copy_file_range.
Record I/O done by glob calls.
Report the most appropriate mount point when a “bind mount” has been used to mount a device at more than one location.
Documentation is now available in HTML format as well as PDF #6000.
Properly handle child jobs launched via posix_spawn.
Remove I/O to unix sockets from Network profiling views.
Fixed missing sync data in I/O profiling view.
Calls to scandir are treated as reads in much the same way as calls to readdir.
Stop showing profiling data for file systems that are not informative (e.g. device:0:5).
Adds intermediate symlinks to trace instead of needing to traverse them post job.
Handle seeks larger than 2^63 bytes.
Escape backspaces in XML output.
Fixed bug that could cause a segfault if src_addr is NULL in recvfrom.
Avoid a delay at the end of a trace when the trace directory is on NFS.
trace-program.sh will report an error if the permissions on /tmp/BREEZE would stop it from working correctly and this can’t be automatically corrected.
If you use “trace-program.sh –tcsh” to trace a tcsh builtin command, the command will be run within tcsh rather than csh.
Fix missing I/O for fork children in program profile and duration views. #6010
Properly support folding fork children in I/O Summary. #6010
Fix bug where I/O Summary was under-reporting percentage contributions. #6014
Improved handling of long file paths in Files View. #5219
Fix bug that may cause Breeze to output erroneous seek records for copy_file_range. #6008
2 Release 2024.1.1
Now also intercepts close_range when made as an indirect syscall.
Removed Symlink Chains Count Profiling View.
Removed File System Trawls Profiling View.
Removed I/O Sizing profiling views.
Fixed issue where CPU profile data at the start of the job attributed to the wrong time.
3 Release 2024.1.0
3.1 Highlights:
Completely reworked directory hierarchy to be more in-line with the FHS /usr heirarchy recommendations.
Created new RPM and debian packages of breeze, breeze-trace-only, mistral-healthcheck and mistral.
3.2 Other Changes:
Add support for Apptainer. Renamed BREEZE_SINGULARITY_BIND_PATH to BREEZE_CONTAINER_BIND_PATH.
No longer hides monitor PID in logging messages.
Changed pwrite to no longer count as a seek when writing to a file that was opened in append mode to match Linux kernel behaviour.
Fixed bug where hiding may not work as intended.
Many binary names have been changed to include {breeze} to make them more identifiable.
The monitor doesn’t take so long to exit at the end of a job.
Fixed the default path used by some exec functions to match that found in the running version of glibc.
Correct the handling of zero length monitor status files.
Fixed a problem where the monitor could be killed during startup. This resulted in directories being left that would normally have been removed when the monitor exits.
Fix a problem where a multi-threaded application which runs qrsh could modify the environment in an unsafe way.
Fixed race condition that could cause some networking applications to seg fault.
Hide anonymous inodes from the files view and related output.
Fixed pread/pwrite calls always reporting seeks from the beginning of a file.
Stop marking moved files as automatically read.
Added I/O Summary and programs to v3 output.
Fixed bug where directories could cause double counting in I/O Summary.
Fixed bug where library loads were not counted as file use in I/O Summary.
Fixed bug when clicking “Select and zoom to fit in Timeline”.
Fixed bug where network I/O is misclassified in I/O Summary.
Removed text output for programs in BreezeAP.
Fixed bug where host cpu measurements were missing from the trace.
Fixed problem where processes started by sudo were not attached to their parent.
4 Release 2023.1.2
5 Release 2023.1.1
Renamed ELLEXUS_DO_NOT_BACKTRACE to ELLEXUS_BACKTRACE and inverted related logic.
Made backtrace output less confusing, removing libbreeze.so backtrace handler.
Fixed usage messages from C shell scripts having wrong file extension when referring to self.
Fixed bug where popen and system calls could have the incorrect child process.
Fixed bug where operations on network sockets could be assigned to the incorrect port.
Fixed bug where singuarlity child traces may not be in a subdirectory of the parent trace directory.
Changed fflush(NULL) to report each individual stream that was flushed.
Added call name and better event name for posix_spawnp calls.
6 Release 2023.1.0
6.1 Highlights:
Renamed shared object library from “libdryrun.so” to “libbreeze.so”. “libdryrun.so” is now a symlink to “libbreeze.so”.
Removed Light mode from BreezeAP.
Moved Breeze license settings to their own file /breeze-license.properties this file will need to be configured whenever you update Breeze.
6.2 Other Changes:
Now lazily loads libc functions.
Reduced the amount of stack space used in programs being traced or metered.
Changed handling of –child argument, now defaults to all on and command that launch child jobs can be removed from the list with - e.g. –child=-qrsh,-qsub.
Fix problem where a process within Singularity could send information to the wrong monitor process.
Fixes –relocate option archiving all traces, not just the most recent.
Sending SIGUSR1 to the monitor process will now cause it to send status output to the error log.
Removed the MPI implementation specific versions of libdryrun.so.
Now supports pbsdsh and pbs_tmrsh jobs.
Added wrappers for sendmmsg and recvmmsg Linux system calls.
Add singularity-bind program to simplify setting the SINGULARITY_BIND environment variable.
Mistral_start_reference.sh now describes the additional steps needed when Singularity is run directly, rather than as part of a script.
Ensure shell builtins, e.g. echo are traced correctly if they are top level commands.
Fixed bug when tracing csh commands on remote hosts.
Fixed bug where top level-command could be interpreted by a shell.
Fixed bug where user names could be truncated in trace directory names.
Fixed bug for system(3) and popen(3) that could crash some applications which make use of RDMA devices.
Fixed bug for closedir(2) that caused a segfault when passed NULL.
Added wrapper for statx(2) Linux system call.
Fixed bug where stderr was unintentionally open in the monitor, which could alter the behaviour of the Python subprocess module.
Fixed problem where signal masking within libbreeze.so could fail if a signal handler had been installed before libbreeze.so was active.
Fixed bug when tracing sudo that could cause a binary in the current working directory to be run.
Fixed bug when tracing sudo that could prevent a shell from being used to interpret shebang scripts.
Removed internal IDs from all BreezeAP output.
Renamed “Shared Library” to “Library” in BreezeAP output.
Removed Read:Seek ratio from Files view.
Re-ordered fields in files XML output.
Stopped outputing 0s in XML output to reduce size.
Fix bug where directories are marked as temporary files.
Fixed no such file or directory error when using –packages option with trace-program.sh.
Change files, dependencies and events output to use new import when they are the only options selected.
Fixed bug where symlinks could be reported as being on the mountpoint of the target rather than where the link resides.
Fixed bug where some connect calls were missing from trace.
Now traces calls made to the PSCHED TM API function tm_spawn.
Fixed issue where network profiling data was not properly reported.
Added double-click to open Program Information dialog to I/O Summary.
Fixed issue with missing network I/O counts in profiling view.
The time taken by a traced function is now referred to as a “duration” rather than a “latency”. Two configuration environment variables have been renamed accordingly. (BREEZE_MONITOR_LATENCY_MAX_IO becomes BREEZE_MONITOR_DURATION_MAX_IO and BREEZE_MONITOR_LATENCY_SAMPLE becomes BREEZE_MONITOR_DURATION_SAMPLE.)
Added search box for process environment.
Failed reads/writes and execs are no longer counted as dependencies.
Directories are now more consistently marked as directories rather than just files.
Fixed bug whereby if a shebang script on the path was wrapped with trace-program.sh, then its full path was not being reported in the trace.
Improve the accuracy of symlink counts and filesystem trawls.
Consolidated file types - files are no longer marked as source files/log/configuration etc but just as Files, Directories, Symlinks etc.
Added network hostnames to network view, network profiling data and events view.
Hidden and Temporary are no longer file types, but an attribute of a file. This allows all files, directories and symlinks to be represented as hidden/temporary as necessary.
Only show views relevant to current selection.
Fix incorrect column headings in Prorgam I/O Summary.
Show program library information in tables to make each line easier to read.
Fix bug in Program Files view where the totals were for the whole job, not the selected process.
Fixed bug in Program Profile where the tooltip was not for the correct data point.
Fixed bugs in Program Profile where right click menu didn’t work or disappear.
7 Release 2022.2.1
Reduced the amount of stack space used in programs being traced or metered.
Fix bug setting strace output directory with remote traces.
Removed incorrect headings in network CSV output.
8 Release 2022.2.0
The latency of I/O on unix domain sockets is no longer counted as file system latency.
Fix problems seen on Fedora 35 and glibc 2.34.
The error log command line option has been changed from –log to –errlog.
Fix a problem which caused I/O on a pipe, or a file descriptor which is not associated with an inode, to be counted as I/O to a file in the current working directory.
The start-monitor utility can now be run without setting LD_PRELOAD.
Fix problem if Breeze is run with the path name of a command which creates a child job.
Make the –child-job option work correctly when Breeze is used with Singularity.
Fix problem when Breeze is used with an MPI job running on Altair Grid Engine.
Fix problem where child jobs over-wrote user child job (ELLEXUS_CHILD) setting.
Correctly identify mount points when running in a Singularity container.
Fix problems shown by compiling with gcc version 12.1.
Fix bug in path resolution triggered by flush in some circumstances.
Fix bug where strace option wasn’t preserved in child jobs.
ELLEXUS_OUTPUT_DIRECTORY renamed BREEZE_OUTPUT_DIRECTORY and is no longer changed whilst being used.
Fix problem where a program calls close_range or closefrom.
Add documentation of the integration with Altair Grid Engine 2022.2.
Removed obsolete option BREEZE_PROFILE_BUCKETS (same functionality available with BREEZE_VOLUMES).
Removed option BREEZE_PROFILE_FS_TRAWL (trawl detection is now always on).
Removed little-used configuration options BREEZE_PROFILE_TIME_RANGES and BREEZE_PROFILE_TRAWL_LENGTH - the previous default values are now hard-wired (powers-of-ten time ranges and trawl length threshold 4).
Improved reporting of mountpoints in the Breeze trace.
Added option to export symlinks data.
Changed branding and improved structure of Healthcheck report.
The environment variable which represents the name of the error log has been changed to BREEZE_ERR_LOG, though ELLEXUS_ERR_LOG is still supported for backward compatibility.
Fix problem where trace-program.sh could exit before the trace was complete.
Fixed problem where the number of read/write/seek calls could be under-reported in the files and network views. Errors were not always counted in # Calls.
Fixed problem with long paths preventing horizontal scrolling in Files view, when column auto-expanded.
9 Release 2022.1.3
Fixed bug in path resolution triggered by flush in some circumstances.
Fixed bug in argmangling where ELLEXUS_CHILD was not respected in child jobs.
Fix problem when Breeze is used with an MPI job running on Altair Grid Engine.
10 Release 2022.1.2
The permissions of files created by Breeze are no longer modified by the umask value.
The start-monitor utility can now be run without setting LD_PRELOAD.