Configuring a suite

Apilicious can be used for a lot of purposes other than testing, the power is in your hands to design a suite that improves your workflow.

It is not uncommon for the same API to have several test suites.

This guide explores how suites can be configured to work for a wide range of purposes.

Monitoring

When monitoring a production environment, special care should be taken with the configuration.

Schedule	Automating with a regular schedule. "Always on" will wait for 10 minutes between tests, but "hourly" will also provide regular updates
Test concurrency	High concurrency may identify issues faster, but
Test consistency	Random will on average identify issues earlier, but consistency
Examples	Keep examples low for a smoke test, or use dynamic examples for something deeper

Regression suites

A regression suite is typically performed before updates to a production environment.

Github	Start a report when a PR is created, or when a deployment has been completed
Server	Use the API to change the target server dynamically (eg. staging or temporary environments)
Test limits	High limits will ensure everything is tested
Test consistency	A consistent test suite will ensure the same tests are performed each time
Webhook	Use a webhook to perform actions after the test report has been completed
Slack integration	Increase pass rate visibility by exposing test results to Slack

Test-Driven development (TDD)

When your specification documents features that do not exist yet, Apilicious enables you to also follow TDD principals.

With this approach, the baseline pass rate will (successfully) start low and tracked as the feature is introduced.

Test limits	High limits will ensure everything is tested but may want to be balanced with dynamic examples
Specification	If not public, provide a static specification which contains the future features
Dynamic examples	Building dynamic examples will act as an additional layer of self-testing (Note: Failing to retrieve any will not result in failed tests)

Testing authentication flows

A small suite that dedicated to ensure permissions are working as expected.

Test limits	Low limits across a wide range of operations should be sufficient to test authentication/authorization
Test consistency	A consistent test suite will ensure the same tests are performed each time
Schedule	Authentication failures are likely something that you want to fail fast
Static examples	Bare minimum examples required - Everything should fail

Testing partial access

The test runner assumes that the credentials provided are valid, but this can be used to ensure that a user account.

Scenarios	Only the baseline scenario should be needed; The auth suite will not always use the provided credentials
Server	Assume this test will eventually fail. Will this break something?

Warming up the cache

Use the test runner to reduce the liklihood that your end-user receives a MISS.

Examples	Use dynamic examples to ensure the test runner accesses the freshest data
Test limits	Setting a high limit (particularly to examples) will build deeper requests
Test consistency	Example randomisation will increase the liklihood that
Schedule / Concurrency	A combination of these settings can be used to ensure the tests are always running
Skips	GET requests should be enough to warm up the cache in most cases

Performance monitoring

Apilicious surfaces the metrics you need to monitor your API over time. What makes the test runner unique is that it will go deep into the API with some relatively simple configuration.

Examples	Use dynamic examples so that the test runner can problem your API in different
Test limits	Setting a high limit (particularly to examples) will build deeper requests
Concurrency	Configure how many requests can be sent at the same time
Schedule	Increasing the schedule will provide more snapshots

Target environments

The test runner is capable of communicating with any publicly accessible API. Update the target server URL and credentials to provide alternative endpoints, as required.

The test runner sends a User-Agent header in the format Apilicious {suiteId} {resultId}. This can be used to identify the test runner, or debug the request on your own stack.

Safety checklist for production environments

There is nothing stopping you from targeting a production environment - what else are you going to monitor?

When designing a suite, you should carefully consider the following just to be sure something unintentional won't happen,

• Is the user isolated?
• Can the user create/update/delete anything it shouldn't?
• Do I need static examples or dynamic examples?
• Will the environment be able to handle the requests?
• Can I test the suite using a staging environment first?