Fork me on GitHub

Bisect_ppx   version 1.4.1 Travis status

Bisect_ppx is a code coverage tool for OCaml and Reason. It helps you test thoroughly by showing what's not tested.

Bisect_ppx usage example

You can browse the report seen above online here. The details of how it is generated are in the worked example.


Table of contents


Usage

Dune

Refer to aantron/bisect-starter-dune, which produces this report.

  1. Depend on Bisect_ppx in your opam file:

    depends: [
      "bisect_ppx" {dev & >= "1.5.0"}
    ]
    
  2. Preprocess the code under test with bisect_ppx (but don't preprocess the tester itself):

    (library
     (public_name my_lib)
     (preprocess (pps bisect_ppx --conditional --no-comment-parsing)))
    
  3. Run your test binary. In addition to testing your code, when exiting, it will write one or more files with names like bisect0123456789.out. Then, generate the coverage report in _coverage/index.html:

    BISECT_ENABLE=yes dune runtest --force
    bisect-ppx-report --html _coverage/ -I _build/default/ `find . -name 'bisect*.out'`
    
  4. During release, you have to manually remove (preprocess (pps bisect_ppx)) from your dune files. This is a limitation of Dune that we hope to address in ocaml/dune#57.


BuckleScript

Refer to aantron/bisect-starter-bsb, which produces this report.

  1. Depend on Bisect_ppx in package.json, and install it:

    "dependencies": {
      "@aantron/bisect_ppx": "*",
      "bs-platform": "^6.0.0"
    }
    
    npm install -g esy
    npm install
    

    If you have not used esy before, the first install of Bisect_ppx will take several minutes while esy builds an OCaml compiler. Subsequent builds will be fast, because of esy's cache.

  2. Add Bisect_ppx to your bsconfig.json:

    "bs-dependencies": [
      "@aantron/bisect_ppx"
    ],
    "ppx-flags": [
      "@aantron/bisect_ppx/ppx.exe"
    ]
    
  3. If your tests will be running on Node, call this function somewhere in your tester, which will have Node write a file like bisect0123456789.out when the tester exits:

    Bisect.Runtime.write_coverage_data_on_exit();
    

    If the tests will be running in the browser, at the end of testing, call

    Bisect.Runtime.get_coverage_data();
    

    This returns binary coverage data in a string option, which you should upload or otherwise get out of the browser, and write into an .out file yourself.

  4. Build in development with BISECT_ENABLE=yes, run tests, and generate the coverage report in _coverage/index.html:

    BISECT_ENABLE=yes npm run build
    npm run test
    ./node_modules/.bin/bisect-ppx-report.exe --html _coverage/ *.out
    

Js_of_ocaml

Refer to aantron/bisect-starter-jsoo, which produces this report.

  1. Follow the Dune instructions above, except that the final test script must be linked with bisect_ppx.runtime (but not instrumented):

    (executable
    (name my_tester)
    (libraries bisect_ppx.runtime))
    
  2. If the tests will run on Node, call this function at the end of testing to write bisect0123456789.out:

    Bisect.Runtime.write_coverage_data ()
    

    If the tests will run in the browser, call

    Bisect.Runtime.get_coverage_data ()
    

    to get binary coverage data in a string option. Upload this string or otherwise extract it from the browser to create an .out file.

  3. Build the usual Js_of_ocaml target, including the instrumented code under test, then run the reporter to generate the coverage report in _coverage/index.html:

    BISECT_ENABLE=yes dune build my_tester.bc.js
    bisect-ppx-report --html _coverage/ *.out
    

Ocamlfind, Ocamlbuild, and OASIS


Sending to Coveralls.io

You can generate a Coveralls JSON report using the bisect-ppx-report tool with the --coveralls flag. Note that Bisect_ppx reports are more precise than Coveralls, which only considers whole lines as visited or not. The built-in Coveralls reporter will consider a full line unvisited if any point on that line is not visited, check the html report to verify precisly which points are not covered.

Example using the built-in Coveralls reporter on Travis CI (which sets $TRAVIS_JOB_ID):

  bisect-ppx-report \
      -I _build/default/ \
      --coveralls coverage.json \
      --service-name travis-ci \
      --service-job-id $TRAVIS_JOB_ID \
      `find . -name 'bisect*.out'`
  curl -L -F json_file=@./coverage.json https://coveralls.io/api/v1/jobs

For other CI services, replace --service-name and --service-job-id as follows:

CI service --service-name --service-job-id
Travis travis-ci $TRAVIS_JOB_ID
CircleCI circleci $CIRCLE_BUILD_NUM
Semaphore semaphore $REVISION
Jenkins jenkins $BUILD_ID
Codeship codeship $CI_BUILD_NUMBER

Controlling coverage with [@coverage off]

You can tag expressions with [@coverage off], and neither they, nor their subexpressions, will be instrumented by Bisect_ppx.

Likewise, you can tag module-level let-declarations with [@@coverage off], and they won't be instrumented.

Finally, you can turn off instrumentation for blocks of declarations inside a module with [@@@coverage off] and [@@@coverage on].


Real-world example examined

Refer to:

The details:

  1. The project depeds on package bisect_ppx, so that Bisect_ppx is installed by opam pin --dev-repo markup and opam install .

  2. There are three libraries in src/, each set to have its sources preprocessed by Bisect_ppx:

    Because of the --conditional flag, preprocessing is enabled only when BISECT_ENABLE=yes is set in the environment, so it is off by default.

  3. A coverage build is triggered by running make coverage. This target...

  4. make coverage is also used in Travis to submit coverage reports to Coveralls. At the end of make coverage, the bisect*.out files are still present, so .travis.yml runs bisect-ppx-report again to generate the Coveralls report. This follows the Coveralls instructions exactly.

    Coveralls can be configured to leave comments about changes in coverage. It is usually configured to at least add an additional check to branches and PRs — see the "3 checks passed" in the hidden Details of the linked PR.

  5. During release, (preprocess (pps bisect_ppx)) is removed from all libraries that are being released. This is typically in a one-commit release branch off master, which is what ends up being tagged.

    This won't be necessary after ocaml/dune#57 is addressed.


Other topics

See advanced usage for how to exclude files from coverage, and supported environment variables. Use of these features is discouraged. They are meant for working around build system issues and for build debugging.


Bisect_ppx users

A small sample of projects using Bisect_ppx:


Contributing

Bug reports and pull requests are warmly welcome. Bisect_ppx is developed on GitHub, so please open an issue.

Bisect_ppx is developed mainly using opam. To get the latest development version, run

opam source --dev-repo --pin bisect_ppx

You will now have a bisect_ppx subdirectory to work in. Try these Makefile targets:

This page uses github-markdown-css.