Variables Reference

Special Environment Variables

Dagu automatically sets these environment variables for every step execution:

Variable	Description	Example
DAG_NAME	Name of the current DAG	my-workflow
DAG_RUN_ID	Unique ID for this execution	20240115_140000_abc123
DAG_RUN_STEP_NAME	Name of the current step	process-data
DAG_RUN_LOG_FILE	Path to the main log file	/logs/my-workflow/20240115_140000.log
DAG_RUN_STEP_STDOUT_FILE	Path to step's stdout log	/logs/my-workflow/process-data.stdout.log
DAG_RUN_STEP_STDERR_FILE	Path to step's stderr log	/logs/my-workflow/process-data.stderr.log

Example usage:

steps:
  - name: log-context
    command: |
      echo "Running DAG: ${DAG_NAME}"
      echo "Execution ID: ${DAG_RUN_ID}"
      echo "Current step: ${DAG_RUN_STEP_NAME}"
      echo "Logs at: ${DAG_RUN_LOG_FILE}"

Environment Variables

System Environment Variable Filtering

For security, Dagu filters which system environment variables are passed to step processes and child DAGs.

How It Works:

System environment variables are available for expansion (${VAR}) when the DAG configuration is parsed, but only filtered variables are passed to the actual step execution environment.

Filtered Variables:

Only these system environment variables are automatically passed to step processes and child DAGs:

• Whitelisted: PATH, HOME, LANG, TZ, SHELL
• Allowed Prefixes: DAGU_*, LC_*, DAG_*

The DAG_* prefix includes the special environment variables that Dagu automatically sets (see below).

What This Means:

You can use ${AWS_SECRET_ACCESS_KEY} in your DAG YAML for variable expansion, but the AWS_SECRET_ACCESS_KEY variable itself won't be available in the environment when your step commands run unless you explicitly define it in the env section.

Defining Environment Variables

Set environment variables available to all steps:

env:
  - LOG_LEVEL: debug
  - DATA_DIR: /tmp/data
  - API_URL: https://api.example.com
  - API_KEY: ${SECRET_API_KEY}  # Explicitly reference system environment

Important: To use sensitive system environment variables in your workflows, you must explicitly reference them in your env section as shown above. They will not be automatically available.

Variable Expansion

Reference other variables:

env:
  - BASE_DIR: ${HOME}/data
  - INPUT_DIR: ${BASE_DIR}/input
  - OUTPUT_DIR: ${BASE_DIR}/output
  - CONFIG_FILE: ${INPUT_DIR}/config.yaml

Loading from .env Files

Load variables from dotenv files:

# Single file
dotenv: .env

# Multiple files (loaded in order)
dotenv:
  - .env
  - .env.local
  - configs/.env.${ENVIRONMENT}

Example .env file:

DATABASE_URL=postgres://localhost/mydb
API_KEY=secret123
DEBUG=true

Parameters

Positional Parameters

Define default positional parameters:

params: first second third

steps:
  - echo "Args: $1 $2 $3"

Run with custom values:

dagu start workflow.yaml -- one two three

Named Parameters

Define named parameters with defaults:

params:
  - ENVIRONMENT: dev
  - PORT: 8080
  - DEBUG: false

steps:
  - ./server --env=${ENVIRONMENT} --port=${PORT} --debug=${DEBUG}

Override at runtime:

dagu start workflow.yaml -- ENVIRONMENT=prod PORT=80 DEBUG=true

Mixed Parameters

Combine positional and named parameters:

params:
  - ENVIRONMENT: dev
  - VERSION: latest

steps:
  - echo "Deploying $1 to ${ENVIRONMENT} version ${VERSION}"

Run with:

dagu start workflow.yaml -- myapp ENVIRONMENT=prod VERSION=1.2.3

Command Substitution

Execute commands and use their output:

env:
  - TODAY: "`date +%Y-%m-%d`"
  - HOSTNAME: "`hostname -f`"
  - GIT_COMMIT: "`git rev-parse HEAD`"

params:
  - TIMESTAMP: "`date +%s`"
  - USER_COUNT: "`wc -l < users.txt`"

steps:
  - echo "Deploy on ${TODAY} from ${HOSTNAME}"

Output Variables

Capturing Output

Capture command output to use in later steps:

steps:
  - command: cat VERSION
    output: VERSION
  - docker build -t myapp:${VERSION} .

Output Size Limits

Control maximum output size:

# Global limit for all steps
maxOutputSize: 5242880  # 5MB

steps:
  - command: cat large-file.json
    output: FILE_CONTENT  # Fails if > 5MB

Redirecting Output

Redirect to files instead of capturing:

steps:
  - command: python report.py
    stdout: /tmp/report.txt
    stderr: /tmp/errors.log

JSON Path References

Access nested values in JSON output:

steps:
  - command: |
      echo '{"db": {"host": "localhost", "port": 5432}}'
    output: CONFIG
    
  - psql -h ${CONFIG.db.host} -p ${CONFIG.db.port}

Sub-workflow Output

Access outputs from nested workflows:

steps:
  - run: etl-workflow
    params: "DATE=${TODAY}"
    output: ETL_RESULT
    
  - |
      echo "Records processed: ${ETL_RESULT.outputs.record_count}"
      echo "Status: ${ETL_RESULT.outputs.status}"

Step ID References

Reference step properties using IDs:

steps:
  - id: risky
    command: 'sh -c "if [ $((RANDOM % 2)) -eq 0 ]; then echo Success; else echo Failed && exit 1; fi"'
    continueOn:
      failure: true
      
  - |
      if [ "${risky.exitCode}" = "0" ]; then
        echo "Success! Output was:"
        cat ${risky.stdout}
      else
        echo "Failed with code ${risky.exitCode}"
        cat ${risky.stderr}
      fi

Available properties:

• ${id.exitCode} - Exit code of the step
• ${id.stdout} - Path to stdout log file
• ${id.stderr} - Path to stderr log file

Variable Precedence

Variables are resolved in this order (highest to lowest):

1. Command-line parameters

   dagu start workflow.yaml -- VAR=override

2. Step-level environment

   steps:
     - env:
         - VAR: step-value

3. DAG-level parameters

   params:
     - VAR: dag-default

4. DAG-level environment

   env:
     - VAR: env-value

5. Dotenv files

   dotenv: .env

See Also

• Writing Workflows - Detailed guide on using variables
• YAML Specification - Complete YAML format reference
• Configuration Reference - Server configuration variables