Skip to content

System Test - Logical Overview

System tests that follow this setup can be extended to function as Factory Acceptance Tests!

Deployment made easy

Install all deployment units
  • ssh app1Machine -> Use ssh to log into the machine you want to run your app
  • rpm -ivh app1.rpm -> use rpm installer to install all artifacts needed. TODO: Link to rpm instructions

This process is also possible to automate completely. TODO: How?

Configure all applications

There are many beliefs on how to configure your application environment. In our opinion you must distinguish the difference between: - Strictly confidential information, eg password and usernames. This must be stored locally on application machine. - Environment, application configuration. This info should preferably be fetched from a ConfigService. TODO: Link to ConfigService description.

To support automatic deploy, a ConfigService is required.

Automated test in a multi-node system

Our suggestion on how to automate your tests when your services and applications span over multiple machines and processes.

1. Verify valid system state.

TODO: Examples on how wold be nice. A description on how to run this verification (manually or automatically) should also be included in the SysAdmin Production Toolbox.

Idea Service manifesto extension: All services should include ping - "I'm alive and healthy"

2. Restore default state

Ensure that the system you want to test removes all old, uncontrolled information.

Examples are to remove all messages from all queues or drop your tables in your database.

The Helper Libraries describes how to perform this cleanup.

3. Populate with test data

This will ensure that your application is set to a known state before the testing begins. Here you will also be able to populate your test database with data from the production database.

Again Helper Libraries will help you out.

4. Prepare instrumentation of your functionality.

To ensure traceability of how your system behaves during development, and for each delivery, we need to log a few key performance indicators (KPI). Such can be successful run, number of runs, time used and memory usage. TODO: Link to examples of KPI for a system.

5. Run your test

System specific test code.

The system are often tested by sending some message to one application. Then we need to verify: - Was the message processed by the correct applications? - Verify that the result was correctly returned, or persisted to the expected location.

6. Verify your instrumentation result

Example of verification: - Was the massage processed in correct order, among the applications (verification of workflow). - Are the values as we expected? Eg. if the run took more than 10 sec, then fail the test.

TODO: Example of automated verification

7 Store metrics from test run in a repository

Persist the data so it can be used for reporting and statistics.

Data to be stored: - Name of test - Configuration of test (eg. number of users) - Configuration of the environment used. - Nodes used. - KPI result from above. ...

Reporting and Statistics

The metrics gathered should be used to generate reports and statistics. This will help document your project's success.

You must do some thinking, talking and trial and error to identify which values are important to measure. These values should be compared repeatedly during your development. The total time used for your tests to run, and the amount of memory usage will increase. We recommend that you monitor this as it progress, and assess if there are any unexpected behavior.

Other reports people might be asking for: - Are the new system better than the old one? - YOU will have to define "better". - With configB instead of configA; has the performance of app51 increased or decreased? ...