Bash Execute With Progress Action

The Bash Execute With Progress Action node executes a Unix Bash command and attempts to extract progress information from the command output.

It can be used for long-running operations such as file transfers, downloads, copies, or other command-line processes that provide percentage-based progress output. When progress can be extracted successfully, the related Helmut job is updated during execution.

This node is intended for Unix-based systems such as Linux and macOS.

Configurable Settings

Async

Enables or disables asynchronous execution.

Command

The Bash command to execute.

Helmut wildcards can be used within the command.

Example:

cp {job.source} {helmut.variable.root-helmut}/Asset/{path.name.{job.source}}

Paths containing spaces should be enclosed in quotation marks.

Example:

cp "{job.source}" "{helmut.variable.root-helmut}/Asset/{path.name.{job.source}}"

Progress Regex

An optional regular expression used to extract the current progress value from the command output.

The regular expression should contain a capture group that returns the numeric percentage value.

Default example:

.*?(\d+)%.*

This pattern extracts percentage values from output such as:

Progress: 42%

The captured value 42 is then used to update the job progress.

The configured command must continuously write progress information to the command line for this feature to work.

Progress Message

The job message displayed while the command is running and progress can be determined.

Example:

Downloading...

The extracted percentage is applied to the job together with this message.

Unextractable Progress Message

The message displayed when the command is running but no progress value can be extracted using the configured regular expression.

Example:

Unable to determine progress from the command output. The operation is still running.

Failure to extract progress does not necessarily mean that the command itself has failed. The process continues, but the job progress cannot be updated with a percentage.

Success Message

The job message displayed after the Bash command completes successfully.

Example:

Download completed successfully

Progress Extraction

The node reads the command-line output while the process is running and applies the configured regular expression to it.

For example, when the command outputs:

Transferred 65%

and the Progress Regex is:

.*?(\d+)%.*

the node extracts 65 and updates the job progress accordingly.

The first capture group of the regular expression must contain a numeric value suitable for use as a percentage.

Additional Progress Regex Examples

The correct regular expression depends on the progress format written by the executed command. The first capture group must contain the numeric progress value.

Generic Percentage

Matches values such as 42%, Progress: 42%, or Transferred 42%:

.*?(\d+)%.*

Percentage With Decimal Values

Matches values such as 42.5% and captures the whole-number portion:

.*?(\d+)(?:\.\d+)?%.*

FFmpeg-Style Progress Output

When FFmpeg is executed with -progress pipe:1, it can output progress values such as:

progress=continue
out_time_ms=12500000

FFmpeg does not provide a percentage directly. The command must first convert the processed duration into a percentage, or output a custom percentage value.

For custom output such as:

progress=42

use:

.*progress=(\d+).*

rsync Percentage

Matches common rsync --progress or rsync --info=progress2 output:

.*?(\d+)%.*

Example output:

1,234,567  42%  12.34MB/s

curl Percentage

Matches the percentage column in the standard curl progress meter:

^\s*(\d+)\s+.*$

Depending on the curl output format, a more restrictive expression may be required. A custom curl format is usually more reliable:

curl -w "progress:%{http_code}\n"

For custom percentage output such as progress:42, use:

.*progress:(\d+).*

wget Percentage

Matches common wget output such as 42%:

.*?(\d+)%.*

Fraction-Based Progress

For output such as:

Processed 42/100

use:

.*?(\d+)/100.*

This works when the total is always 100.

For arbitrary totals such as 42/250, the regex can extract the current value:

.*?(\d+)/\d+.*

However, the extracted value is not automatically converted into a percentage. The command should calculate and output the percentage itself.

Bracketed Percentage

Matches output such as:

[42%] Processing file

.*\[(\d+)%\].*

Labeled Percentage

Matches output such as:

Progress: 42

.*Progress:\s*(\d+).*

JSON-Style Progress

Matches output such as:

{"progress":42}

.*"progress"\s*:\s*(\d+).*

Key-Value Progress

Matches output such as:

percent=42

.*percent=(\d+).*

Recommendation

Whenever possible, configure the executed command to output a simple and predictable progress format, for example:

PROGRESS=42

Then use:

.*PROGRESS=(\d+).*

A dedicated progress marker reduces the chance that unrelated numbers in the command output are interpreted as progress.

Typical Usage

Command:
rsync -ah --info=progress2 "{job.source}" "/Volumes/Target/{path.name.{job.source}}"

Progress Regex:
.*?(\d+)%.*
 
Progress Message:
Copying asset...

Unextractable Progress Message:
The asset is being copied, but no progress value is available.

Success Message:
Asset copied successfully

Important Notes

The command must be compatible with the Bash environment on the executing system.
The executable used by the command must be installed and available to the executing user.
The executing user requires the necessary permissions for all referenced files and directories.
Progress can only be displayed when the command writes matching progress information to its output.
The regular expression must contain a capture group for the numeric progress value.
Commands that buffer or suppress their output may not provide usable progress updates.
Quote paths that contain spaces or special characters.
The node does not automatically make commands safe. Validate wildcard values before using them in shell commands, especially when they may contain user-provided content.

Last updated 3 months ago