# Overloop Test (overloop-test)

[jenkins hosted version of the docs](http://jenkins.overloop.io/job/overloop-test/ws/docs/build/README.html)

It might be worth reading [Happiness and Testing](docs/HappinessAndTesting.html) for a discussion about why we test rather than how.

The other documentation generated by this module can be found:
* [docs/build/code](code/main.html)
* [docs/build/api](api/index.html)
* [docs/build/coverage](coverage/index.html)
* [docs/build/docs](docs)

### TODO

* implement karma tests
* get jenkins to run tests on all branches, this will require lots of nodes and mongos dbs to avoid collisions
* make smoke (preprod) tests platform agnostic somehow at the moment the first time they blow up is on preprod
* make smoke (perprod) tests a more valid test of live configuration
* expand coverage reporting to more flavours of testing
* make cms api security tests valid against the production configuration

## Types of testing

 | Testing Type | With | Status
--- | --- | :---: | :---:
[A](#unit-testing) | Server *unit* testing | [Mocha](https://mochajs.org/) | DONE
[B](#e2e-testing) | *e2e* testing | [Nightmare](https://github.com/segmentio/nightmare) | DONE
[C]() | *smoke* testing | [Nightmare](https://github.com/segmentio/nightmare) | DONE
[D]() | *benchmark* testing | [Loadtest](https://github.com/alexfernandez/loadtest) | DONE
[E]() | *api* testing | [Request](https://github.com/request/request) | DONE
[F](#unit-testing) | Browser *unit* testing | [Karma](https://karma-runner.github.io/) | **TODO**
[G](#linting) | *jshint* testing | [Jshint](http://jshint.com/) | DONE
[H](#coverage-documentation-generation) | Coverage documentation generation | [Blanket](https://github.com/alex-seville/blanket) | DONE
[J](#api-documentation-generation) | API documentation generation | [ApiDoc](http://apidocjs.com/) | DONE
K | A/B testing | [A/B](http://blog.christianposta.com/deploy/blue-green-deployments-a-b-testing-and-canary-releases/) | *UNSUPPORTED*
L | Canary testing | [Canary](http://blog.christianposta.com/deploy/blue-green-deployments-a-b-testing-and-canary-releases/) | *UNSUPPORTED*
M | Manual testing | ["That's you, that is..."](https://www.youtube.com/watch?v=aEQcsuXnnnc) | *UNSUPPORTED*
N | Health checking | [UptimeRobot](https://uptimerobot.com/), [Logentries](https://logentries.com/) | *UNSUPPORTED*
[O](#coverage-enforcement) | *coverage* enforcement | [H](#coverage-documentation-generation) | DONE

## Use of testing

When | Where | What | TODO | Best Practice
--- | --- | :---: | ---: | ---
Before Change | *localhost* | A,D?,E | F |
Before Push | *localhost* | A,B,D?,E,G,H,I,J,M | F,H | The viable *localhost* tests should be wrapped up into `$ grunt test` and documentation generation into `$ grunt build` in each [project](#customising-grunt)
After Push | e2e & [jenkins](http://jenkins.overloop.io) | A,B,D,E,G | C?,F | Ideally every push to any branch would kick off a jenkins build
Before Deployment (*merge to test*) | `*-test.overloop.io` & [jenkins](http://jenkins.overloop.io) | A,B,D,E,G,H,I,J,O | C?,F | The viable *jenkins* tests should be wrapped up in `$ grunt jenkins:test` and documentation generation into `$ grunt jenkins:build` in each [project](#customising-grunt)
Before Production Deployment (*merge to master*) | `*-preprod.overloop.io` & [jenkins](http://jenkins.overloop.io) | A,B,D,E,G,H,I,J,O + C | |
After Production Deployment | `*.overloop.io` | M,N | K,L | Ideally we would be able to do a canary deployment 

## Installation

Most of the notes here refer to installing this module into a repository that already has its own test tasks
and configuration, you'll be able to skip a lot of that if you're adding this to a new repo!

### 1. Remove modules from your repository

Some test related modules are now part of *overloop-test* and you should remove them from your repository:

`npm rm grunt-blanket grunt-contrib-clean grunt-contrib-copy grunt-contrib-jshint grunt-mocha-test --save-dev`

There may be other modules you can remove now too like jsbeautifier and grunt-todo...

And remove these lines to load them from your `gruntfile.js` that look like these:

```
grunt.loadNpmTasks('grunt-blanket');
```

### 2. Install overloop-test

`npm i --save-dev git+ssh://git@github.com/overloop-io/overloop-test.git#<version_tag>`

#### version_tag

Current version of the module, avoid deployments breaking because of changes to this module.

### 3. Wire into grunt

*gruntfile.js*:
```
grunt.initConfig({
    ... my grunt config ...
});

var test = require('overloop-test');
test.gruntInit(grunt);

grunt.loadNpmTasks('grunt-my-extra-tasks');

// You must define the test task for this repository
grunt.registerTask('test', ['jshintMinimal','test:unit', 'test:e2e']);

```

### 4. Restructure your test files

Move the various flavours of tests that already exist in your repo to the correct directories for `overloop-test`,
the files are in [Usage](#Usage)

### 5. Clean up your grunt configuration and tasks

`overloop-test` provides lots of configuration and tasks which are now duplicated in your repository 
(`overloop-test.gruntInit` merges its configuration into yours). In general we are trying to remove local 
configuration in your `grunt/` directory and move anything that you still need to override from 
your `grunt` directory to the main `gruntfile.js`

This bit is difficult, and can't easily be explained, each repository will have its own set of testing tasks
and configuration, just removing files in a blanket fashion will probably break things.

In most cases you will be able to remove the files wholesale as overloop-test has its own copies, the following are
candidates:

```
grunt/mochaTest.json
grunt/jshint.json
grunt/jshint_count_reporter.js
grunt/coverage.json
grunt/tasks/jenkins.js
grunt/tasks/test.js
```

### 6. Update .gitignore

You need to add these two lines to the `.gitignore` file for your repository and remove any other references to 
'coverage' (from the old coverage testing configuration).

```
docs/build/
test/coverage/
```

### 7. Enable coverage testing

There is an out of the box configuration for coverage testing however the *requireAll* grunt task my need to be
overridden in your repository.

The coverage testing that we use only measures the coverage of files that have been required, normally each file is
required by its associated tests, however, we often don't have tests for all our files so we *requireAll* the application
files before we start running coverage tests. This prevents developers from being penalised for adding tests to files
that haven't had tests before, since we are enforcing increasing test coverage on jenkins.

This is the *requireAll* task included in overloop-test:

```
grunt.registerTask('requireAll', function() {
    var glob = require('glob');
    var path = require('path');
    glob.sync('test/coverage/**/*.js').forEach(function(p) { require(path.resolve(process.cwd(),p)); });
});
```

You may find that you need to require some files in advance of others (like the models), you can override this by simply adding
an updated version of the above to your `gruntfile.js`.

You should now be able to run `grunt test:coverage` which will return a number for the test coverage in your repository and generate [docs/build/coverage/index.html](coverage/index.html) (good for finding low hanging fruit) and [docs/build/coverage/coverage.json](coverage/coverage.json).

### 8. Make sure you're using best practice on jenkins for your builds

In the new jenkins world, all the code that runs for a build lives in `.jenkins/run.sh` and `.jenkins/preprod.sh`,
the best jobs to look at on jenkins are [overloop-api-test](http://jenkins.overloop.io/job/overloop-api-test/configure) and [overloop-api](http://jenkins.overloop.io/job/overloop-api/configure).

*Make sure you're using* `set -e` *in your scripts so that when a test fails the script exits with a failure code.*

### 9. Make sure you're handling the database well on jenkins

To allow multiple builds to run concurrently on jenkins that rely on mongo you also need to make sure that you use a job specific db, to achieve this in your repository you need to add this snippet to `config/config.js` in your repository:

```
// Modify the db if we're on jenkins and in e2e
if (process.env.NODE_ENV === 'e2e' && process.env.JOB_NAME) {
    var db = module.exports.db;
    module.exports.db = db + '_' + process.env.JOB_NAME;
    console.log('Changing config.db from %s to %s',db,module.exports.db);
}
```

### 10. Add coverage enforcement

To your `.jenkins/run.sh`, you just need to add a line at the end: `grunt --no-color enforceCoverage`, take a look at it's implementation to work out how you
might test it locally.

#### Troubleshooting

##### *overloop-test* works fine locally but blows up on jenkins and complains about missing grunt modules

Run `npm prune --production && npm install` to reinstall the devDependencies

## Usage

overloop-test expects to find files in the test directory of your project:

    test/
         e2e/**/*.js
         smoke/**/*.js
         unit/
              server/app/**/*.js
              browser/**/*.js # TODO
         api/**/*.js
         benchmark/**/*.js
         coverage/ # Generated
                  app/
                  config/
                  test/

...and will generate the following documentation :

    docs/
         v0.1.0/ # version from package.json
                api/*.html
                code/*.html
                coverage/coverage.html

All the tasks can be run using grunt *(`$ grunt --help` will show you all the installed tasks)* and there 
follows a more detailed explanation of the ones provided by overloop-test.

---
### Documentation Generation

Build all the documentation [docs/build](README.html)

`$ grunt docs`

#### Code

`$ grunt docs:api`

Generates documentation from comments in the `test/api/**/*.js` files it finds and drops them into the [docs/build/api](api/index.html) directory.

This is implemented with [Apidoc](http://apidocjs.com/) and is an alias for:

`$ grunt apidoc` 

#### Coverage

`$ grunt docs:coverage`
`$ grunt test:coverage`

Runs the server unit tests to generate a coverage report in [docs/build/coverage](coverage/index.html).

It also generates a `coverage.json` file which can be used to [enforceCoverage](#coverage-enforcement) on jenkins.

Eventually we should be able to run all the tests against coverage insrumented code.

This is implemented with [Blanket](https://github.com/alex-seville/blanket).

`$ grunt showCoverage`

This will output the current coverage from the coverage report.

#### Markdown

`$ grunt docs:markdown`

Generates documentation from `**/*.md` creating html files in [docs/build/docs](docs)

This is implemented with [Marked](https://github.com/chjj/marked).

`$ grunt markdown`

### Linting

`$ grunt jshint`

**overloop-test** provides default configuration for the following sub-tasks:

* jshint:test
* jshint:node
* jshint:browser
* jshint:grunt
* jshint:config
* jshint:scripts

You can extend or modify these in your own configuration see [Customising Grunt](#customising-grunt).

`$ grunt jshintMinimal`

Minimal reporting jshint (just flags errors without identifying each one), good for part of a larger test run.

### E2E Testing

`$ grunt test:e2e --show`

Starts a server, seeds the DB, and runs the nightmare e2e tests in `test/e2e/**/*.js`

`$ NODE_ENV=e2e grunt mochaTest:nightmareE2E`

Just runs the nightmare e2e tests (you have to do everything else), make sure you have Nightmare installed as a devDependency in your project.

    options:

    --show
        Shows the browser as it works
        
    --DEBUG
        Shows the server side logs

### Unit Testing

`$ grunt test:unit`

Runs all unit tests

#### Server side unit testing

`$ grunt test:unit:server`

Runs the Mocha unit tests in `test/unit/server/**/*.js`.

Best practice is to structure the test files in the same way you would in the app, ie. if you wanted to write tests for `app/controllers/myController.js`
you would put them in `test/unit/server/app/controllers/myController.js`

#### Browser side unit testing

`$ grunt test:unit:browser`

**TODO**

Karma spins up a browser (or browsers) to run browser based tests in against a dummy server, this allows us to do the same unit testing on the browser side as we do on the server side.

Best practice is to structure the test files in the same way you would in the app, ie. if you wanted to write tests for `public/js/controllers/myController.js`
you would put them in `test/unit/browser/js/controllers/myController.js`

### Smoke Testing

`$ grunt test:smoke`

Runs the nightmare tests in `test/smoke/**/*.js` against the preprod server.

    options:
    
    --show 
        Shows the browser as it works.

### API Testing

`$ grunt test:api`

Starts an e2e server, seeds the DB, and runs the mocha tests in `test/api/**/*.js` which fire requests against the server and check the responses.

These tests do not involve a browser and have been used for security testing so far.

### Coverage Enforcement

`$ grunt enforceCoverage`

*Requires COVERAGE_DIR to be defined*

Compares the coverage testing between sucessful builds and requires `$ grunt docs:coverage` to have already been run.

At the moment it simply compares the overall test coverage between the current run `docs/build/coverage/coverage.json` and the previous successful run
saved at `${COVERAGE_DIR}/<%= pkg.name %>-<%= gitinfo.local.branch.current.name %>.json`.

For some repositories you will need to add a custom *requireAll* grunt task to make sure that **all** files that do not have tests count towards the coverage stats. Usually you will only need to do this if the supplied version fails for your repo.

### Benchmarking

`$ grunt benchmark:e2e`

Starts an e2e server, seeds it, and then fires requests at it using [Loadtest](https://github.com/alexfernandez/loadtest), see code documentation for the [benchmark function](code/main.html#functions).

## Writing tests with overloop-test

**TODO** [Generated documentation](http://jenkins.overloop.tv/job/overloop-test/ws/docs/index.html)

### Customising Grunt

overloop-test provides baseline configuration for testing that can be overridden, the configurations injected into the gruntfile can
be found in `grunt/config` but the local module's configuration overrides this.

```
grunt.initConfig({
    ...
    "mochaTest": {
        ...
        "myCustomTests": {
            "src": ["test/myCustomTests/**/*.js"]
        }
    },
    ...
});

var test = require('overloop-test');
test.gruntInit(grunt);

grunt.loadNpmTasks('grunt-my-extra-tasks');

// you must override the 'test' task locally
grunt.registerTask('test', ['jshintMinimal'])

```

Then the nightmare e2e tests can be run with:

`$ grunt mochaTest:nightmareE2E --show`

This will normally be aliased so that any pre-setup can be done like seeding and environment config:

`$ grunt test:e2e --show`

## Updating the version of this Module

Things to remember:
* All work should be done in a branch and merged with a pull request
* After a successful build on jenkins you can, if you choose, increment the version locally on master using:

`npm version patch`