Using Clover in various environment configurations

 

Introduction

This is a tutorial showing how to use Clover:

  • with projects containing multiple modules,
  • running on multiple application severs,
  • in multiple test phases (e.g. unit tests, integration, manual testing),
  • in multiple test runs (snapshots and history points)
  • in distributed environment

This tutorial shall help you answering questions like:

  • should I use cloverDatabase, singleCloverDatabase or cloverMergeDatabase?
  • should I deploy instrumented code on all application servers and run tests at once or sequentially?
  • what should I copy (or not copy) to test server?
  • should I use distributed.coverage=on?
  • should I merge Clover databases?

 

Confused which scenario you shall use? Have a quick look at the Decision Matrix.

 

 

Decision matrix

Step 1: Building your application with Clover

Q1 How many applications do you build, for which you want to have single, consolidated report? (term 'application' means a separate source code and independent build)
  one application many applications
Q2 How many modules your application(s) has(have)? (term 'module' means a part of source code, built in the same session as other parts of code, like the Maven module)
  one module many modules one module many modules
Solution  
 

only one database will be produced under default location so there's nothing to do

one database is created for every module

keep all applications under a common root; every application will have one Clover database in its default location

keep all applications under a common root; one database is created for every module for each application

 

Step 2: Running / testing your application with Clover

Q3 Do you use separate machines or working directories for compilation and testing?
  no yes
Q4 Do you have many Clover databases generated from Step 1?
  n/a one database many databases
Solution  
  nothing to do; path to database
is already stored in instrumented code
copy database to a test machine; provide clover.initstring at runtime pointing to database location on a test machine ensure that you've used relative paths in Step 1; copy all databases to a test machine preserving directory structure; use clover.initstring.basedir pointing to common root folder
  as soon as tests are finished, you shall have coverage recording files in the same directory where database is located as soon as tests are finished, copy coverage recording files to a build machine into a directory with a database as soon as tests are finished, copy coverage recording files generated for all databases to a build machine preserving directory structure

 

Step 3: Aggregating coverage data and generating report

Q5 Do you have many Clover databases generated from Step 1?
  one database many databases from one application many databases from many applications
Solution  
  nothing to merge, just generate a report using clover:clover merge databases after tests using clover:aggregate and next generate report using clover:clover merge databases after tests using clover:aggregate (to merge modules of every single application), followed by clover:merge (to merge merged databases from all applications into one database)

 

Optional: Optimizing your environment configuration

Q6 In case you use different machines for build/reporting and testing - do you have a shared network drive? 
  yes no
Q7 Do you execute the same application (i.e. binaries produced in one build and using the same clover.db) on several machines?
  no yes no yes
Solution  
  nothing to do (clover.db created during build
is available on test machine too thanks to
a network drive; coverage recording files
are written to the same directory)
nothing to do (clover.db created during build
is accessible on all test machines too; coverage
recordings from all test machines are written to
the same directory; coverage files generated on different
machines will not clash because they're using
unique file names)

copy clover.db from build to test server

execute tests

copy clover.db and coverage files from test to report server

copy clover.db from build to test servers

execute tests

copy clover.db from build to report server
copy coverage files from all test servers to
report server into the same directory

 

Optional: Per-test coverage and test optimization and distributed coverage

Q8 Are you interested in per-test coverage report or in test optimization?
  no yes
Q9 Do you have distributed application, so that single test case executes application logic on several machines?
  no yes no yes
Solution  
  nothing to do

don't set up distributed coverage feature

just run your application and gather
coverage recording files from all
machines in one common directory

don't set up distributed coverage feature

use showUniqueCoverage=true for reporting

instrument code with distributedCoverage option
(define host/port for server hosting unit tests)

at runtime, designate one server where unit tests are
executed and run with clover.server=true runtime property

see Using Distributed Per-test Coverage

use showUniqueCoverage=true for reporting

 

Assumptions for all scenarios:

For the simplicity of the tutorial it's assumed that:

  • we have separated machines for build, tests and reporting
  • we have a shared network drive accessible from all machines at the same absolute path

In case when your environment is different and:

  • build, application or reporting server is the same machine => skip points talking about copying files
  • shared network drive is accessible from all machines but at different absolute paths => instrument sources using relative paths, use clover.initstring.basedir / clover.initstring.prefix at runtime
  • shared network drive is not available => copy clover.db / recording snapshots / history points between machines

 

Example #1 - one application executed on several servers

Assumptions

  • there is one application having one Clover database
    (e.g. a Maven project with a <singleCloverDatabase> or an Ant project with one <clover-setup>)
  • application is deployed to several machines
  • we're not interested in per-test coverage recording
  • we want to have a single report showing combined coverage from all machines

 

(lightbulb)TIP: this scenario applies also to a case when:

  • application is deployed on one server or
  • application is deployed on one server and running in multiple JVMs or
  • application is executed multiple times (for example: unit tests + integration tests + manual testing)

Overview chart


Steps

1) Build application with Clover

a) using Ant

Define initstring attribute for <clover-setup> or <clover-instr> tag, e.g.:

<clover-setup initstring="/path/to/network/drive/clover.db" />

b) using Maven

Define <cloverDatabase> and <singleCloverDatabase> property in pom.xml, e.g.:

<plugin>
    <groupId>org.openclover</groupId>
    <artifactId>clover-maven-plugin</artifactId>
    <configuration>
        <singleCloverDatabase>true</singleCloverDatabase> 
        <cloverDatabase>/path/to/network/drive/clover.db</cloverDatabase>
    </configuration>
</plugin>
mvn clean clover:setup install


2) Deploy instrumented application

Copy clover.jar and your application EAR/WAR. There's no need to copy clover.db as it's on a network drive.


3) Run tests

Execute your tests. There is no need to provide clover.initstring parameter at runtime as path to clover.db database is already hardcoded in instrumented sources.

 

4) Generate coverage report

a) using Ant

Execute <clover-report> task; use initstring pointing to clover.db on a network drive, e.g.:

<clover-report initstring="/path/to/network/drive/clover.db">

b) using Maven

Execute clover:clover goal. Note that you don't need to call clover:aggregate or clover:merge as there is only one database. The <cloverDatabase> defined in pom.xml will be used for reporting.

<plugin>
    <groupId>org.openclover</groupId>
    <artifactId>clover-maven-plugin</artifactId>
    <configuration>
        <singleCloverDatabase>true</singleCloverDatabase>
        <cloverDatabase>/path/to/network/drive/clover.db</cloverDatabase>
    </configuration>
</plugin>
mvn clover:clover

 

Example #2 - one application with many Clover databases executed on several servers

Assumptions

  • one application containing multiple Clover databases
    • like a Maven project with many modules, each with own clover.db,
    • or an Ant project with several modules, each with own clover.db
  • an application is deployed to several machines
  • we're not interested in per-test coverage recording
  • we want to have a single report showing combined coverage for all modules from all servers

Overview chart


Steps

1) Build application with Clover

a) using Ant

<target name="all">
   <!-- Enable Clover for every module, each with own database -->
   <clover-setup initstring="/path/to/network/drive/moduleX/clover.db">
</target>


b) using Maven

  • don't define singleCloverDatabase and cloverDatabase, so that default values will be used
  • keep databases on a shared drive (or copy them to test machines)
<!-- top level pom.xml -->
<plugin>
    <groupId>org.openclover</groupId>
    <artifactId>clover-maven-plugin</artifactId>
    <configuration>
        <!-- no <cloverDatabase> and <singleCloverDatabase> -->
    </configuration>
</plugin>


2) Deploy instrumented application

Copy clover.jar and your application EAR/WAR. There's no need to copy clover.db files as they're on a network drive.

 

3) Run tests

Execute your tests. Provide clover.initstring.basedir=/path/to/top-level-module/dir or use clover.initstring.prefix at runtime. There is no need to define such property if databases are available under the same absolute path on test machines as on the build machine.

 

4) Generate coverage report

Before generating report you have to merge all databases. For example:

a) using Ant

<clover-merge initstring="/path/to/mergedClover.db">
   <cloverDb initstring="/path/to/network/drive/moduleA/clover.db"/>
   <cloverDb initstring="/path/to/network/drive/moduleB/clover.db"/>
</clover-merge>
<clover-report initstring="/path/to/mergedClover.db"/> 

b) using Maven

<!-- Top-level pom.xml -->
<plugin>
    <groupId>org.openclover</groupId>
    <artifactId>clover-maven-plugin</artifactId>
    <configuration>
        <cloverMergeDatabase>/path/to/network/drive/cloverMerged.db</cloverMergeDatabase>
    </configuration>
</plugin>
mvn clover:aggregate clover:clover

 

Example #3 - multiple applications executed on several servers in isolation

Assumptions

  • we have multiple applications (each of them can have one or more modules)
    • every application has separate code base (i.e. no shared source files)
    • every application is built and instrumented separately (i.e. separate clover.db for every app)
  • we deploy instrumented applications to several application servers
    • every application runs on it's own server (i.e. no case when two apps runs in the same JVM)
  • tests run independently on each application server and for each application
  • we're not interested in per-test coverage recording
  • we want to have a single report showing combined coverage for all modules of all applications from all application servers

Overview chart


Steps

1) Build application with Clover

a) using Ant

Define initstring attribute for <clover-setup> or <clover-instr> tag, which will point to different directory for every application ,e.g.:

<!-- App1 build.xml -->
<clover-setup initstring="/path/to/network/drive/app1/clover.db">

<!-- App2 build.xml -->
<clover-setup initstring="/path/to/network/drive/app2/clover.db">

b) using Maven

Define <cloverDatabase> property for Clover plugin in pom.xml. You can use singleCloverDatabase in case your application is multi-module. Databases for all applications must be stored under common root (it's a limitation of clover:merge goal; Ant clover-merge task is more flexible regarding paths). Example:

<!-- App1 pom.xml -->
<plugin>
    <groupId>org.openclover</groupId>
    <artifactId>clover-maven-plugin</artifactId>
    <configuration>
        <cloverDatabase>/path/to/network/drive/common-root/app1/clover.db</cloverDatabase>
    </configuration>
</plugin>

<!-- App2 pom.xml -->
<plugin>
    <groupId>org.openclover</groupId>
    <artifactId>clover-maven-plugin</artifactId>
    <configuration>
        <cloverDatabase>/path/to/network/drive/common-root/app2/clover.db</cloverDatabase>
        <singleCloverDatabase>true</singleCloverDatabase> <!-- assuming that app2 is multi-module -->
    </configuration>
</plugin>
mvn clean clover:setup install

 

2) Deploy instrumented application to Application Servers

Copy clover.jar and your application jar/war to proper machines. There's no need to copy clover.db as it's on a network drive.


3) Run tests on Application Servers

Execute your applications. As every application runs in their own JVM and due to fact that we have used cloverDatabase (and optionally singleCloverDatabase) pointing to absolute path on a network drive, we don't need to provide clover.initstring parameter at runtime, because correct path is hardcoded in instrumented classes.

 

4) Generate coverage report

Before generating report you have to merge all databases. for example:

a) using Ant

<clover-merge initstring="/path/to/network/drive/cloverMerged.db">
   <cloverDb initstring="/path/to/network/drive/app1/clover.db"/>
   <cloverDb initstring="/path/to/network/drive/app2/clover.db"/>
</clover-merge>
<clover-report initstring="/path/to/cloverMerged.db"/> 

b) using Maven

<!-- Top-level pom.xml -->
<plugin>
    <groupId>org.openclover</groupId>
    <artifactId>clover-maven-plugin</artifactId>
    <configuration>
        <cloverMergeDatabase>/path/to/network/drive/cloverMerged.db</cloverMergeDatabase>  <!-- output database -->
        <baseDir>/path/to/network/drive/common-root</baseDir>  <!-- common root -->
        <includes>*.db</includes> <!-- filename pattern, separated by comma or space -->
    </configuration>
</plugin>
mvn clover:aggregate             # run it for multi-module applications
mvn clover:merge clover:clover  # run it for final report

 

 

Example #3b - multiple applications executed on several servers in partial isolation

This scenario is a variation of Scenario 3 in such way, that:

  • location of clover.db(s) on test server is different than on build server (so you have to provide/change initstring at runtime)
  • some applications are executed in the same JVM (as a consequence, you cannot pass clover.initstring as JVM argument, because you need a different value for each application).

 

In such case, you have to:

  • instrument these applications using relative paths in <cloverDatabase> parameter (Maven) or <clover-setup initstring=""> (Ant), like below:
app1/clover.db
app2/clover.db
app2/moduleA/clover.db
app2/moduleB/clover.db

 

  • copy generated clover.db(s) to test server, keeping their relative paths (under a common root), for instance:
/path/to/common-root/app1/clover.db
/path/to/common-root/app2/clover.db
/path/to/common-root/app2/moduleA/clover.db
/path/to/common-root/app2/moduleB/clover.db

 

  • provide clover.iniststring.basedir=/path/to/common-root at runtime

 

 

Example #4 - multi-module application with distributed execution on several servers

Assumptions

  • we have a multi-module project (like Maven project with modules, or Ant project with several build.xml files with inheriting properties across ant calls)
  • we deploy instrumented application (the same code) to several application servers
  • unit tests are executed on one <<server>> machine, but these unit tests call business logic on one or more <<client>> machines
  • we are interested in per-test coverage recording, showing combined coverage from distributed execution
  • we want to have a single report showing combined coverage for all modules from all application servers

Overview chart


Steps

1) Build application with Clover

  • define how many clients will connect to JVM running unit tests;

(warning) a number greater than 0 means that server will hold until all clients are connected before it continues execution; number equal 0 means that tests will start immediately

(warning) you might have a dependency loop so that server waits for clients and clients wait for server - see below

  • define server host name (default is localhost) and listening port (default is 1198)
  • optionally define connection timeout (in milliseconds), retry period 

 

a) using Ant

<clover-setup initstring="/path/to/network/drive/clover.db">
    <distributedCoverage host="my.server.com" port="1234" numClients="2" timeout="10000"/>
</clover-setup>

b) using Maven

<plugin>
    <groupId>org.openclover</groupId>
    <artifactId>clover-maven-plugin</artifactId>
    <configuration>
        <cloverDatabase>/path/to/network/drive/clover.db</cloverDatabase>
        <singleCloverDatabase>true</singleCloverDatabase>  <!-- In case of multi-module application (optional) -->
        <distributedCoverage>
            <host>my.server.com</host>
            <port>1234</port>
            <numClients>2</numClients>
            <timeout>10000</timeout>
        </distributedCoverage>
   </configuration>
</plugin>


2) Deploy instrumented application to Application Servers

Copy clover.jar and your application jar/war to proper machines. There's no need to copy clover.db as it's on a network drive.

 

3) Run tests on Application Servers

On <<server>> machine

java ... -Dclover.server=true

On <<client>> machines

You don't have to provide any runtime options for JVM. They're already compiled in the code.

 

Potential problems

Server does not wait for clients, despite having numClients != 0 in build configuration

Do not use -Dclover.distributed.coverage=ON runtime option if numClients!=0 was set in instrumentation. The clover.distributed.coverage provided at runtime will override numClients setting from instrumentation, setting it to 0. As a consequence your tests on server will start immediately, without waiting for clients to connect. It can result in lower or zero coverage.

Instead of this:

  • enable distributed coverage option in build file or
  • use -Dclover.distributed.coverage=numClients=N (where N is a number >= 0) at runtime

Execution of tests hangs when with numClients != 0

It can happen that your server will wait for clients to connect, while clients will wait until server starts unit test execution. This is a typical case for web applications running in container (like Tomcat, JBoss), when your unit test calls a servlet class (e.g. via HTTP request). The issue is as follows:

  • unit tests on <<server>> are waiting until all clients are connected (numClients != 0) but
  • none of the clients will connect until servlet class is loaded in the container, which happens only when first request comes (and it will not come, due to point above)

See Working with Distributed Applications how to fix this circular dependency.

 

4) Generate coverage report

a) using Ant

<clover-report initstring="/path/to/network/drive/clover.db">
    <current showUniqueCoverage="true" outfile="/path/to/clover/report">
        <format type="html"/>
        <fileset dir="src"/>
    </current>
</clover-report>

b) using Maven

In order to show per-test coverage in the HTML report (showUniqueCoverage), you have to use the custom <reportDescriptor> in pom.xml and in the report descriptor set the showUniqueCoverage=true. For example:

report-descriptor.xml
<project name="Clover Report" default="current">
    <clover-setup initString="${cloverdb}"/>
    <target name="historical"/>
    <target name="current">
        <clover-report>
            <current showUniqueCoverage="true" outfile="${output}">  <!-- Show per-test coverage in report -->
                <format type="html"/>
            </current>
        </clover-report>
    </target>
</project>
pom.xml
<plugin>
    <groupId>org.openclover</groupId>
    <artifactId>clover-maven-plugin</artifactId>
    <configuration>
        <cloverDatabase>/path/to/network/drive/clover.db</cloverDatabase>
        <outputDirectory>/path/to/clover/report</outputDirectory>
        <reportDescriptor>report-descriptor.xml</reportDescriptor>    <!-- Use custom report -->
    </configuration>
</plugin>


mvn clover:clover

 

More information about format of report descriptor can be found here:

Example #5 - multi-module application with history points

Assumptions

  • we have a multi-module project (like Maven project with modules, or Ant project with several build.xml files with inheriting properites across ant calls)
  • we deploy instrumented application to one application server
  • we are not interested in per-test coverage or test optimization
  • we want to have a single report showing combined coverage for all modules

Overview chart


Steps

 1) Build application with Clover

a) using Ant

<target name="all">
   <!-- Enable clover for top level module -->
   <clover-setup initstring="/path/to/network/drive/clover.db">
   <!-- Build sub-modules ensuring that properties are passed -->
   <ant inheritrefs="true" inheritprops="true" file="sub-module-a/build.xml" target="all"/>
   <ant inheritrefs="true" inheritprops="true" file="sub-module-b/build.xml" target="all"/>
</target>

b) using Maven

<!-- TOP LEVEL POM.XML -->
<plugin>
    <groupId>org.openclover</groupId>
    <artifactId>clover-maven-plugin</artifactId>
    <configuration>
        <cloverDatabase>/path/to/network/drive/clover.db</cloverDatabase>
        <singleCloverDatabase>true</singleCloverDatabase>
    </configuration>
</plugin>
mvn clover:setup test 


2) Deploy instrumented application to Application Servers

Remove previous version of application and copy clover.jar and your application jar/war. There's no need to copy clover.db as it's on a network drive.

 

3) Run tests on Application Servers

Execute your application. As we have used singleCloverDatabase and cloverDatabase pointing to absolute path on a network drive, we don't need to provide clover.initstring parameter at runtime.


4) Generate coverage report

a) using Ant

<clover-report initstring="/path/to/network/drive/clover.db">
    <current outfile="/path/to/clover/report/current" title="Coverage Report">
        <format type="html"/>
        <fileset dir="src"/>
    </current>
    <historical outfile="/path/to/clover/report/historical" title="Historical Report" historyDir="/path/to/clover/historypoints">
       <format type="html"/>
    </historical>
</clover-report>

<clover-historypoint historyDir="/path/to/clover/historypoints">
       <fileset dir="src"/>
</clover-historypoint>

b) using Maven

<!-- Top-level pom.xml -->
<plugin>
    <groupId>org.openclover</groupId>
    <artifactId>clover-maven-plugin</artifactId>
    <configuration>
        <cloverDatabase>/path/to/network/drive/clover.db</cloverDatabase>
        <singleCloverDatabase>true</singleCloverDatabase>
        <generateHistorical>true</generateHistorical>
        <generateHtml>true</generateHtml>
        <historyDir>/path/to/clover/historypoints</historyDir>
        <outputDirectory>/path/to/clover/report</outputDirectory>
    </configuration>
</plugin>


mvn clover:clover clover:save-history

 

References

 

Don't define <cloverDatabase> and <singleCloverDatabase> so that default values will be used (relative path target/clover/clover.db and false).