Download

Introduction Last updated: 2020-09-06

API tests are an important part of finding and fixing bugs in any server application, but as tests become more complex the tests themselves are likely to become a source of bugs. Since the test code itself is untested, code review is the only line of defense against bugs that cause a test to always pass, this creates a need for tests that are powerful but easy to read. AARTL facilitates writing and running tests that are declarative, easy to read for anyone knowledgeable about HTTP and JSON, and that can do most (possibly all) of the tasks you need for your HTTP API tests.

It is different from other test frameworks in that it does not require any procedural code to be written and it can help you auto generate test rules for simple smoke tests.

1.1 Example

Tip

This test POSTs the text "Hello world" to localhost:3000/posts and then GETs the same post and checks that it has text that equals to "Hello world"

1.2 Auto-generating tests

As explained in more detail below, an extension is available which can automatically generate JSON-based rules for a test consisting of a GET request by fetching the JSON response and analysing petterns therein. Here is an example of this process:

Installation

The reference implementation of AARTL is implemented using TypeScript and Node.js.

You have two options for running it on your machine:

  • Run a self-contained release
  • Build from source

Run a self-contained release

To run one of the self-contained releases: download the right one for your operating system from the Download link above and run it as a command line application.

Build from source

If you are familiar with Node.js and would like the least (possibly unstable) build you can git clone the repository and then do the following steps:
  1. npm ci
  2. npm run build
  3. node dist/aartl.js -f FILE_YOU_WANT_TO_RUN.aartl
On Linux you may need to use sudo for npm ci

2.1 Visual Studio Code extension

A Visual Studio Code extension is available which provides the following benefits:

  • Syntax highlighting
  • Code completion
  • Automatically generating test rules
  • Running tests from within the IDE

Getting Started

A test consists of a name, optional constant values, optional “After” blocks, and a required "Expect" block. JSON data is referred to within AARTL using JSON paths. For more information about the JSON path standard you may refer to: https://support.smartbear.com/alertsite/docs/monitors/api/endpoint/jsonpath.html

An After block consists of:

  • The data needed to make a request
  • Optional statements about how to handle transitioning to the next request.

Data needed for an HTTP request is:

Optional statements about how to handle transitioning to the next request are:

  • Pass on – the Pass on statement can be used to pass on a value from a response to the next block. Example: Pass on “$..id” as _id, this means that we can refer to the id returned by this request in the next request using the _id symbol. The $..id is a JSON path.
  • Wait – the Wait statement indicates that there will be a waiting period before the next request starts

An Expect block consists of:

  • Data needed to make a request (same as an After block)
  • Expectations for what the response should be, there are three types of expectations: - JSON data expectations - Header expectations - Status code expectation

JSON data expectations consist of:

  • A JSON path and either a literal value or a rule.
Consider the following example: 
					Test that it should save a post and check id
					Using values
					  @postText: Hello world
					Expect HTTP request
					  method: POST
					  url: http://localhost:3000/posts
					  body: @postText
					To match JSON rules
					  "$..id": >= 0
In this example "should save a post and check id" is the name of the test.
@postText is a constant value equal to "Hello world"
POST is the method used for this request
http://localhost:3000/posts is the URL being requested
"$..id" is JSON path
"$..id": >= 0 is a rule that means that each value that matches the JSON path must be greater than or equal to 0

3.1 Writing your first test

Once the AARTL test runner is setup on your computer you can write your first test and run it.
You can use the JSON placeholder website (no affiliation) to write your first AARTL tests.

3.2 Generating test rules

The extension can automatically generate JSON-based rules for a test consisting of a GET request by fetching the JSON response and analysing petterns therein.
To generate a set of rules for a test using the SON placeholder website, write the follwong: 

					Test that it should have id greater than 50
					Expect HTTP request
						method: get
						url: https://jsonplaceholder.typicode.com/posts 
					To match JSON rules
You should see a code lens link at the top of the test that says Generate JSON Rules, click it, if all goes well some JSON-based rules should be added to the test.

Rules

The test verifies that the response complies with certain rules.

4.1 JSON-based rules

JSON rules consist of:

  • A JSON path and either a literal value or a matcher.

The possible rules are (x, y, z in the rules refer to parameters):

Matcher What it means
is a number checks if every value that matches the JSON path is a number
> x checks if every value that matches the JSON path is a number greater than x
>= x checks if every value that matches the JSON path is a number greater than or equal to x
< x checks if every value that matches the JSON path is a number less than x
<= x checks if every value that matches the JSON path is a number less than or equal to x
is text checks if every value that matches the JSON path is text of length 1 or longer
is text containing x checks if every value that matches the JSON path is text that contains x
is text not containing x checks if every value that matches the JSON path is text that does not contain x
is any of x y z checks if every value that matches the JSON path is one of the values
is not x checks if every value that matches the JSON path is not x
matches x checks if every value that matches the JSON path matches the regular expression x
count = x checks if the number of values that matches the JSON path is x
count > x checks if the number of values that matches the JSON path is greater than x
count >= x checks if the number of values that matches the JSON path is greater than or equal to x
count > x checks if the number of values that matches the JSON path is greater than x
count >= x checks if the number of values that matches the JSON path is greater than or equal to x
count < x checks if the number of values that matches the JSON path is less than x
count <= x checks if the number of values that matches the JSON path is less than or equal to x
each has x checks if every value that matches the JSON path is an object with a property called x
properties limited to x y z checks if every value that matches the JSON path is an object the properties of which cannot be anything other than x y or z
is after x checks if every value that matches the JSON path is a date after x
is as early as x checks if every value that matches the JSON path is a date as early as x
is as late as x checks if every value that matches the JSON path is a date up to and including x
is a date checks if every value that matches the JSON path is a date
is earlier than x checks if every value that matches the JSON path is a date earlier than x
is same date as x checks if every value that matches the JSON path is a date with the same year day, month, and day as x
is same date and time as x checks if every value that matches the JSON path is a date with the same year day, month, and day and time as x
is sorted asc/desc checks if the values that match the JSON path are stored in either ascending or descending order, one of the following types is automatically inferred: number, date, string

4.2 Header rules

Header name followed by a colon and the expected value or a rule. There is one header rule, it is "must not be present". Examples:

"X-Powered-By":"Express" /* will pass if this header is sent and equals to Express */
"X-Powered-By": must not be present /* will fail if this header is sent */

4.3 Response code rules

That the expected response code was returned by the server can be checked like so: To respond with status code 200 This can be useful to prevent soft 404s and in other situations where returning the correct error code is important.

CLI Options

Option What it means
--hello Print the name of the program before running tests
--xml Output results as JUnit XML
--novalidation Don't validate test file
--r Randomize test order
-n N Rerun the tests a number of times
-m N Maximum concurrent tests. Default: no limit
--report Output an html report with failure rates
--q Don't output real-time test results
--log Output request logs
--ff Exit with error code 1 as as soon as one test fails

5.1 Examples

aatrl -f example.aartl --report --q -n 100 Means run all the tests in example.aartl 100 times and produce a report, do not output the test results in real times aatrl -f example.aartl --ff Means run all the tests in example.aartl once and exit with an error code of 1 as soon as one test fails

FAQs

Questions you might have

What does the word "agnostic" in AARTL refer to?

The fact that it is indifferent to the plaform on which your server is based.

6.1 Prerequisites

What operating system can AARTL run on?

Current versions of Windows, common Linux distributions, macOS, freeBSD