Tag: Software Quality

In Part 1 of this tutorial we created a Property-based test (PBT) from a normal JUnit test with basic types. Now let us extend the domain object PostalParcel with a list of Products.


public class Product {

    private UUID uuid;
    private String name;
    private int weight;

    public Product(UUID uuid, String name, int weight) {
        this.uuid = uuid;
        this.name = name;
        if(weight > 0 ) {
            this.weight = weight;
        } else {
            throw new IllegalArgumentException("Weight not acceptable, less than 0.");
        }
    }

    public int getWeight() {
        return weight;
    }

}
public class PostalParcel {
    public static final double MAX_DELIVERY_COSTS = 4.99;
    public static final double MIN_DELIVERY_COSTS = 1.99;

    private UUID uuid;
    private List<Product> products;

    public PostalParcel(UUID uuid, List<Product> products) {
        this.uuid = uuid;
        this.products = products;
    }

    public double deliveryCosts() {
        if (weight() > 20) {
            return MAX_DELIVERY_COSTS;
        }
        return MIN_DELIVERY_COSTS;
    }xeba

    public int weight() {
        return products.stream().map(Product::getWeight).mapToInt(Integer::intValue).sum();
    }
}

All examples are written with Java 8 and can be downloaded from my gitlab repository.

Write an unit test for the function deliveryCosts with a fixture for the entities. An implementation could look something like this:

private PostalParcel generatePostalParcel(int weight) {
    return new PostalParcel("UUID", generateProducts(weight));
}

private List<Product> generateProducts(int weight) {
    List<Product> products = new ArrayList<>();
    for (int i = 0; i < 5; i++) {
        products.add(new Product("UUID", "Name", weight/4));
    }
    return products;
}

As explained in the previous part, these tests will not cover the entire behaviour we are testing here. We will refactor this to a Property-based test with the help of generators.

Generators

In this certain case we use entities as our input for our unit tests. A possible way of using generated properties for entities is to generate a basic type for each of the needed arguments for our entities. It is easier to let JUnit-Quickcheck generate the entities by creating your own generators; this way your test does not get bloated with properties and is therefore better readable. To create a generator you need to do three things. First create a class that extends from generator<T>:


public class PostalParcelGenerator extends Generator<PostalParcel>

Secondly, implement the constructor with a super to the type you are generating. Lastly, override the function T generate:

public PostalParcelGenerator() {
    super(PostalParcel.class);
}
public PostalParcel generate(SourceOfRandomness sourceOfRandomness, GenerationStatus generationStatus) {
      return null;
}

We can use the generator in our Property-based test in two ways, we can use the @From(T.class) annotation in front of your entity. This annotation tells Junit-Quickcheck which generator to use, for example:


public void deliveryCostsShouldBeMaxWhenWeightIsGreaterThan20(@From(PostalParcelGenerator.class) PostalParcel postalParcel)

It is much easier to let the serviceloader of JUnit-Quickcheck discover the generators by creating a file called com.pholser.junit.quickcheck.generator.Generator in META-INF/services. In this file you add the package name plus class name of the generator. This way, there is even less plumbing going on in your unit tests.

Implement generate

The function generate looks pretty straightforward. It requires you to return the type  you are generating.

public PostalParcel generate(SourceOfRandomness sourceOfRandomness, GenerationStatus generationStatus) {
    return new PostalParcel(gen().make(UUIDGenerator.class).generate(sourceOfRandomness, generationStatus),
            generateProducts(sourceOfRandomness, generationStatus));
}

private List<Product> generateProducts(SourceOfRandomness sourceOfRandomness, GenerationStatus generationStatus) {
    ProductGenerator productGenerator = gen().make(ProductGenerator.class);
    List<Product> products = new ArrayList<>();
    int randomTotalWeight = sourceOfRandomness.nextInt(minWeight, maxWeight);
    while(randomTotalWeight > 0) {
        int maxWeight = sourceOfRandomness.nextInt(1, randomTotalWeight);
        productGenerator.configureMaxWeight(maxWeight);
        products.add(productGenerator.generate(sourceOfRandomness, generationStatus));
        randomTotalWeight = randomTotalWeight-maxWeight;
    }
    return products;
}

There are multiple JUnit-Quickcheck functions going on here. First is the argument SourceOfRandomness. You need this to create random primitive types, like you see in generating the randomTotalWeight. Secondly, you have the argument GenerationStatus, which we do not use in this example, but we can use this to influence the results of a generation for a particular property parameter.

When you want to generate other types than primitives, use the gen() interface. The most common usages are:

  • The gen().make() method makes the generators available to current instance, and configures it with whatever configuration annotations live on the generator class.
  • The gen().type() method asks for an arbitrary generator that can produce instances of the given type. Use this for a string, it will create a total random string with all characters available (even Chinese and Korean characters).

Take notice, as was said in the previous part, if you take a string as argument then the works of Shakespeare in Japanese & Korean are ONE valid input. If you can, wrap the string into its own type, like explained in the Object Calisthenics practise.

Configuring generators

In the implementation of our PostalParcelGenerator you might have noticed the arguments minWeight and maxWeight in generating the randomTotalWeight. These are two variables which we need to configure the generator with. We can do this by creating a public function void called configure. JUnit-Quickcheck requires us to name it like this in order to match this with the annotation we are going to use. In this case we will use the standard JUnit-Quickcheck annotation @InRange, but you can also use your annotation.

private int minWeight = 1;
private int maxWeight = Integer.MAX_VALUE;

public void configure(InRange range) {
 this.minWeight = range.minInt() == Integer.MIN_VALUE ? 1 : range.minInt();
 this.maxWeight = range.maxInt();
}

You can easily configure your generators with the behaviour you need for your test cases. The only problem that remains is that where code is written errors can be made; we all make mistakes. This is also why in my previous post I advised to both use configuration by annotation and to use assumeThat; with this approach mistakes are easily spotted during your tests. After we implemented the rest of the generators, (which you can see in my gitlab repository) we can write our PBT.

@RunWith(JUnitQuickcheck.class)
public class PostalParcelPBTTest {

    @Property
    public void deliveryCostsShouldBeMaxWhenWeightIsGreaterThan20(@InRange(minInt = 21) PostalParcel postalParcel){
        assumeThat(postalParcel.weight(), greaterThan(20));

        assertThat(postalParcel.deliveryCosts(), equalTo(com.baasie.pbt.part1.PostalParcel.MAX_DELIVERY_COSTS));
    }

    @Property(trials = 25)
    public void deliveryCostsShouldBeMinWhenWeightIsLessThanOrEqualTo20(@InRange(maxInt = 20) PostalParcel postalParcel){
        assumeThat(postalParcel.weight(), is(both(greaterThan(0)).and(lessThanOrEqualTo(20))));

        assertThat(postalParcel.deliveryCosts(), equalTo(com.baasie.pbt.part1.PostalParcel.MIN_DELIVERY_COSTS));
    }

}

Creating the generators will cost some startup investment, but afterwards you really benefit from it. For each PBT, you can just add the type of the generator, and maybe some added behaviour you want with it. The minor investment will result in time saving in the long run, and better designed and written software. Additionally the tests will find edge cases that could cause bugs in your software, and resolve them before they occur.

Performance

After my last post I got a comment with a question about performance. I do not know how the exact implementation works, but I do know that running it does not take 100x slower. I did a run in my Intellij for both the tests and the PBT is about 10x slower. In my opinion 10x is not unreasonable, considering the benefit it gives you.

This concludes part 2 of this tutorial, in the next part I will show you how to repeat failed tests.

Also, this post is published on the blog of Xebia

To be able to show you what Property-based testing (PBT) is, let’s start by grasping the concept of a property in programming languages. Since this is a Java tutorial, I will start with Oracle and their definition of a property in their glossary:

Characteristics of an object that users can set, such as the color of a window.

Property is neither a variable/field or a method; it is something in between which is always true in your context. An example is weight in a postal parcel: this always is greater than zero.  In Java the following example implementation would follow:

public class PostalParcel {

    private int weight;
    private String uuid;

    public PostalParcel(String uuid, int weight) {
        this.uuid = uuid;
        if(weight > 0) {
            this.weight = weight;
        } else {
            throw new IllegalArgumentException("");
        }
    }

}

In this case, we made weight a property that always needs to be greater than 0. When designing your software, it is important to search for these properties, because they usually have some sort of business behaviour associated with them.

What is Property-based testing (PBT)

Back to PBT, let’s add a function to PostalParcel that decides the delivery costs of the package. When the weight is greater than 20 the delivery costs will be 4.99 euro, otherwise the delivery cost will be 1.99 euro. We will first write our tests for this, which without PBT, will usually end up something like this:

public class PostalParcelTest {

    @Test
    public void deliveryCostsShouldBeMaxWhenWeightIsLargerThan20(){
        PostalParcel postalParcel = new PostalParcel("uuid", 23);
        assertThat(postalParcel.deliveryCosts(), equalTo(PostalParcel.MAX_DELIVERY_COSTS));
    }

    @Test
    public void deliveryCostsShouldBeMinWhenWeightIsLessThanOrEqualTo20(){
        PostalParcel postalParcel = new PostalParcel("uuid", 19);
        assertThat(postalParcel.deliveryCosts(), equalTo(PostalParcel.MIN_DELIVERY_COSTS));
    }

    @Test(expected = IllegalArgumentException)
    public void shouldThrowIllegalArgumentExceptionWhenWeightIsBelowOne() {
        PostalParcel postalParcel = new PostalParcel("uuid", -100);
    }
} 

Secondly, we have to make these test succeed by writing the implementation of deliveryCosts in a Postal Parcel:


public static final double MAX_DELIVERY_COSTS = 4.99;
public static final double MIN_DELIVERY_COSTS = 1.99;


public double deliveryCosts() {
    if(weight > 20) {
        return MAX_DELIVERY_COSTS;
    }
    return MIN_DELIVERY_COSTS;
}

When you run the tests, they will succeed, but if we think about it, 2 problems will remain. First of all we nicely described our behaviour in the test function name, but the implementation does not match the behaviour. The second test, deliveryCostsShouldBeMinWhenWeightIsLessThanOrEqualTo20, only tests with a weight of 19, but the behaviour clearly states Less Than Or Equal To 20. We can add another test which tests with a weight of 20, but what about the other cases? It will be a lot of work, and a waste of time, to create all the tests our self. Luckily, PBT comes to the rescue!

Furthermore, and this is what PBT is all about, we need a UUID as String for the test.  Since we do not care about the value of that property, we usually use a fixture for a case like this. It might happen that a UUID can give strange errors in executing the behaviour under test now, or in the future. Because we can use anything as a String input, or like Romeu Moura once tweeted:

I advice everyone to watch his talk about Property-based testing.

Now what you usually want to do, is make a Value Object of a UUID, but for now, because i want to show you a PBT example, we will leave it as a String. Even though, you can already see the benefit PBT will give: it makes you think about your input values.

For a more in depth explanation of what Property-based testing is, you can also check the blog post of Hypothesis. Hypothesis also has a Java version, but at the moment this is still a prototype.

JUnit-Quickcheck implementation

Start of with adding the JUnit-Quickcheck dependency to our project.

In order to use JUnit-Quickcheck we need to change 2 things.

First add the @RunWith annotation above our test class.

@RunWith(JUnitQuickcheck.class)

Second, instead of using @Test we use @Property.

@Property
public void deliveryCostsShouldBeMaxWhenWeightIsLargerThan20(){
.....

JUnitQuickcheck will now run each property test 100 times. We will get the added value of this when we let the JUnitQuickcheck generate the property parameters: for each run it will generate a random parameter. Let’s add them:

    @Property
    public void deliveryCostsShouldBeMaxWhenWeightIsGreaterThan20(String uuid, int weight)
        PostalParcel postalParcel = new PostalParcel(uuid, weight);
        assertThat(postalParcel.deliveryCosts(), equalTo(PostalParcel.MAX_DELIVERY_COSTS));
    }
    ....

Running this will most likely fail the tests, because the number can be any int now, but what we want is, for example, Larger Than 20. This can be accomplished in 2 ways, either use assumeThat of JUnit, or use the @InRange of JUnitQuickcheck  in front of the variable.

@RunWith(JUnitQuickcheck.class)
public class PostalParcelTest {

    @Property
    public void deliveryCostsShouldBeMaxWhenWeightIsGreaterThan20(String uuid, @InRange(minInt = 21) int weight){
        assumeThat(weight, greaterThan(20));

        PostalParcel postalParcel = new PostalParcel(uuid, weight);
        assertThat(postalParcel.deliveryCosts(), equalTo(PostalParcel.MAX_DELIVERY_COSTS));
    }

    @Property
    public void deliveryCostsShouldBeMinWhenWeightIsLessThanOrEqualTo20(String uuid, @InRange(minInt = 1, maxInt = 20) int weight){
        assumeThat(weight, is(both(greaterThan(0)).and(lessThanOrEqualTo(20))));

        PostalParcel postalParcel = new PostalParcel(uuid, weight); 
        assertThat(postalParcel.deliveryCosts(), equalTo(PostalParcel.MIN_DELIVERY_COSTS)); 
    }

    @Property
    public void shouldThrowIllegalArgumentExceptionWhenWeightIsBelowOne(String uuid, @InRange(maxInt = 0) int weight) {
        assumeThat(weight, lessThanOrEqualTo(0));

        IllegalArgumentException illegalArgumentException = null;

        try {
            PostalParcel postalParcel = new PostalParcel(uuid, weight);
        } catch(IllegalArgumentException e) {
            illegalArgumentException = e;
        }
        assumeThat(illegalArgumentException, notNullValue());
    }
} 

I would advice to combine both the options, for 2 reasons. First, assumeThat won’t work sufficient enough in small ranges, like in the second example. You will get errors of violating the assumptions. Adding the assumeThat will make your tests more readable and explicit though.

Trials and Success

JUnit-Quickcheck by default will run the test 100 times, each with a random value. In some cases, like the LessThanOrEqual20, running 100 can be a waste (can, because it still is random). For this we can set the trials modifier on the @Property(trials = 25) annotation. This tells JUnit-Quickcheck it should only run the amount of trials set here, in this case 25 should be enough. It should, but it is still random, so it might happen that the checks will succeed the first time, but will fail after a while. This i.m.o. is the real benefit of PBT, it will eventually test all the corner cases for you, and find possible bugs in your system. Success just means it did not find errors this time, but maybe next time it will.

Now we have written our very first, basic, Property-based test for Java. This is already really helpful. In the next part of this tutorial we will go deeper into the framework and we will let JUnit-Quickcheck generate our entities for us as input values.

You can find the sourcecode of this example on my gitlab.

Also, this post is published on the blog of Xebia