You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
restore --ordered (-o) to keep order of output same as input.
this will cache 3*proccesses output waiting for the slowest job to finish.
This means that if the user requested 10 processes (-p 10) then there could
be up to 30 finished jobs waiting for a slow job to finish. If these are input
memory, they are guaranteed to take <= 1MB (+ go's overhead). If they are larger
than 1MB, then their data will be on disk.
This is implemented carefully such that the performance penalty will be small
unless there are few extremely long-running process outliers.
set $PROCESS_I environment variable for each line (or batch of lines).
read GARGS_PROCESS_BUFFER to let user set size of data before a tempfile is used.
read GARGS_WAIT_MULTIPLIER to determine how many finished processes will wait for single slow processes
higher values improve concurrency at the expense of memory.
better cleanup of tmp files in case of process halt.