mmap.page

Command Line Interface Testing

assert.sh

Example:

. assert.sh

# `echo test` is expected to write "test" on stdout
assert "echo test" "test"
# exit code of `true` is expected to be 0
assert_raises "true"
# exit code of `false` is expected to be 1
assert_raises "false" 1
# end of test suite
assert_end examples

Pros:

Cons:

You can implement your own assert_contains function.

ts

It is similar to assert.sh.

Differences:

roundup

Example:

#!/usr/bin/env roundup

describe "group tests"

it_displays_the_title() {
    first_line=$(rup roundup-5 | head -n 1)
    test "$first_line" "=" "roundup(5)"
}

it_exists_non_zero() {
    status=$(set +e ; rup roundup-5 >/dev/null ; echo $?)
    test 2 -eq $status
}

it_survives_edge_cases() {
    rup edge
}

Differences to assert.sh:

I think these differences add readability.

Cons are similar to assert.sh.

rnt

Save output of commands to *.actual, then compare them with *.expected via diff.

Cons are similar to assert.sh, plus:

cmdtest

Like rnt, cmdtest also uses a lot of files. But it adds support for standard error and setup/teardown.

   foo.script
          a script to run the test (this is required)

   foo.stdin
          the file fed to standard input

   foo.stdout
          the expected output to the standard output

   foo.stderr
          the expected output to the standard error

   foo.exit
          the expected exit code

   foo.setup
          a shell script to run before the test

   foo.teardown
          a shell script to run after test

   setup-once
          a shell script to run once, before any tests

   setup  a shell script to run before each test

   teardown
          a shell script to run after each test

   teardown-once
          a shell script to run once, after all tests

Cons:

urchin

urchin is similar to rnt, but it does use separate output files. Instead, exit 0 passes tests.

tests/
  setup
  setup_dir
  bar/
    setup
    test_that_something_works
    teardown
  baz/
    jack-in-the-box/
      setup
      test_that_something_works
      teardown
    cat-in-the-box/
      fixtures/
        things.pdf
      test_thingy
  teardown

Cons are similar to rnt except for ‘Separate expected output’.

Cram

Tests look like documents:


This is a comment.

  $ echo 'Lines beginning with two space followed by $ is a test.'
  $ echo 'Multiline test
  > exit 0 passes tests.'

  $ cram -h
  [Uu]sage: cram \[OPTIONS\] TESTS\.\.\. (re)

  [Oo]ptions: (re)
    -h, --help          show this help message and exit
    -V, --version       show version information and exit
    -q, --quiet         don't print diffs
    -v, --verbose       show filenames and test status
    -i, --interactive   interactively merge changed test output
    -y, --yes           answer yes to all questions
    -n, --no            answer no to all questions
    -E, --preserve-env  don't reset common environment variables
    --keep-tmpdir       keep temporary directories
    --shell=PATH        shell to use for running tests
    --indent=NUM        number of spaces to use for indentation

Lines beginning with two space not followed by $ or > is output.
Output lines ending with a space and (re) are matched against PCRE.

Cons:

shelltestrunner

# optional comment
a one-line shell command
<<<
zero or more lines of standard input
>>>
zero or more lines of expected standard output (or /REGEXP/ added to the previous line)
>>>2
zero or more lines of expected standard error output (or /REGEXP/ added to the previous line)
>>>= STATUS (or /REGEXP/)

Cons:

tf

Example:

## User comments start with double #
## command can be writen in one line with multiple tests:
true # status=0; match=/^$/
## or tests can be placed in following lines:
false
# status=1

Cons:

Bricolage

Example:

# A test is a function. $T is a variable where test keeps its internal files
mytest() {
    # ok is the only assertion helper
    # It uses `test` to check the condition, so syntax is common
    ok 1 -eq 1
    ok foo = foo
    foo="Foo bar"
    ok "$foo" = "Foo bar"

    # You can use `spy` to make a wrapper over a command.
    spy date

    date

    # Command output will be written into <spy>.stdout file:
    ok "$(cat $T/spy.date.stdout)" = "foo"

    # Fake spy output can be specified in the <spy> file:
    echo foo > $T/spy.date
    date

    # You can assert it using tail, sed, awk and other common unix tools
    ok "$(tail -n 1 $T/spy.date.stdout)" = "foo"
}

# You may override test reports as you need
pass() { echo PASS $* }
fail() { echo FAIL $* }

# You have to run your tests manually
bricolage mytest

# Clean test data
rm -rf $T

Pros:

Cons: