Command Line Interface Testing

This survey is outdated and unmaintained.

assert.sh

Example:

 1. assert.sh
 2
 3# `echo test` is expected to write "test" on stdout
 4assert "echo test" "test"
 5# exit code of `true` is expected to be 0
 6assert_raises "true"
 7# exit code of `false` is expected to be 1
 8assert_raises "false" 1
 9# end of test suite
10assert_end examples

Pros:

• simple (only assert, assert_raises and assert_end)
• light weight (just source assert.sh)

Cons:

• It can only match result exactly, no support for "output should contains ..." like function.
• No setup/teardown.

You can implement your own assert_contains function.

ts

It is similar to assert.sh.

Differences:

• setup () and teardown ().
• Every test is a function. exit 0 to pass tests.

roundup

Example:

 1#!/usr/bin/env roundup
 2
 3describe "group tests"
 4
 5it_displays_the_title() {
 6    first_line=$(rup roundup-5 | head -n 1)
 7    test "$first_line" "=" "roundup(5)"
 8}
 9
10it_exists_non_zero() {
11    status=$(set +e ; rup roundup-5 >/dev/null ; echo $?)
12    test 2 -eq $status
13}
14
15it_survives_edge_cases() {
16    rup edge
17}

Differences to assert.sh:

• Use describe at the beginning of a test group instead of assert_end in the end.
• Put tests in functions.

I think these differences add readability.

Cons are similar to assert.sh.

rnt

Save output of commands to *.actual, then compare them with *.expected via diff.

Cons are similar to assert.sh, plus:

• Separate expected output with the corresponding command.
• A lot of tests lead to complex directory structure.

cmdtest

Like rnt, cmdtest also uses a lot of files. But it adds support for standard error and setup/teardown.

   foo.script
          a script to run the test (this is required)

   foo.stdin
          the file fed to standard input

   foo.stdout
          the expected output to the standard output

   foo.stderr
          the expected output to the standard error

   foo.exit
          the expected exit code

   foo.setup
          a shell script to run before the test

   foo.teardown
          a shell script to run after test

   setup-once
          a shell script to run once, before any tests

   setup  a shell script to run before each test

   teardown
          a shell script to run after each test

   teardown-once
          a shell script to run once, after all tests

Cons:

• No support for "output should contains ..." like function.
• Separate expected output with the corresponding command.
• A lot of tests lead to complex directory structure.

urchin

urchin is similar to rnt, but it does use separate output files. Instead, exit 0 passes tests.

tests/
  setup
  setup_dir
  bar/
    setup
    test_that_something_works
    teardown
  baz/
    jack-in-the-box/
      setup
      test_that_something_works
      teardown
    cat-in-the-box/
      fixtures/
        things.pdf
      test_thingy
  teardown

Cons are similar to rnt except for 'Separate expected output'.

Cram

Tests look like documents:

This is a comment.

  $ echo 'Lines beginning with two space followed by $ is a test.'
  $ echo 'Multiline test
  > exit 0 passes tests.'

  $ cram -h
  [Uu]sage: cram \[OPTIONS\] TESTS\.\.\. (re)

  [Oo]ptions: (re)
    -h, --help          show this help message and exit
    -V, --version       show version information and exit
    -q, --quiet         don't print diffs
    -v, --verbose       show filenames and test status
    -i, --interactive   interactively merge changed test output
    -y, --yes           answer yes to all questions
    -n, --no            answer no to all questions
    -E, --preserve-env  don't reset common environment variables
    --keep-tmpdir       keep temporary directories
    --shell=PATH        shell to use for running tests
    --indent=NUM        number of spaces to use for indentation

Lines beginning with two space not followed by $ or > is output.
Output lines ending with a space and (re) are matched against PCRE.

Cons:

• No setup/teardown.
• (re) only matches a single line.

shelltestrunner

1# optional comment
2a one-line shell command
3<<<
4zero or more lines of standard input
5>>>
6zero or more lines of expected standard output (or /REGEXP/ added to the previous line)
7>>>2
8zero or more lines of expected standard error output (or /REGEXP/ added to the previous line)
9>>>= STATUS (or /REGEXP/)

Cons:

• No setup/teardown.

tf

Example:

1## User comments start with double #
2## command can be writen in one line with multiple tests:
3true # status=0; match=/^$/
4## or tests can be placed in following lines:
5false
6# status=1

Cons:

• No setup/teardown.
• Syntax: env[]=~//, env[]?=, env[][]=, etc.

Bricolage

Example:

 1# A test is a function. $T is a variable where test keeps its internal files
 2mytest() {
 3    # ok is the only assertion helper
 4    # It uses `test` to check the condition, so syntax is common
 5    ok 1 -eq 1
 6    ok foo = foo
 7    foo="Foo bar"
 8    ok "$foo" = "Foo bar"
 9
10    # You can use `spy` to make a wrapper over a command.
11    spy date
12
13    date
14
15    # Command output will be written into <spy>.stdout file:
16    ok "$(cat $T/spy.date.stdout)" = "foo"
17
18    # Fake spy output can be specified in the <spy> file:
19    echo foo > $T/spy.date
20    date
21
22    # You can assert it using tail, sed, awk and other common unix tools
23    ok "$(tail -n 1 $T/spy.date.stdout)" = "foo"
24}
25
26# You may override test reports as you need
27pass() { echo PASS $* }
28fail() { echo FAIL $* }
29
30# You have to run your tests manually
31bricolage mytest
32
33# Clean test data
34rm -rf $T

Pros:

• Extremely small (50 LOC).
• Ultimately minimal.

Cons:

• Ultimately minimal.