In Part 1 of this tutorial we created a Property-based test (PBT) from a normal JUnit test with basic types. Now let us extend the domain object PostalParcel with a list of Products.


public class Product {

    private UUID uuid;
    private String name;
    private int weight;

    public Product(UUID uuid, String name, int weight) {
        this.uuid = uuid;
        this.name = name;
        if(weight > 0 ) {
            this.weight = weight;
        } else {
            throw new IllegalArgumentException("Weight not acceptable, less than 0.");
        }
    }

    public int getWeight() {
        return weight;
    }

}
public class PostalParcel {
    public static final double MAX_DELIVERY_COSTS = 4.99;
    public static final double MIN_DELIVERY_COSTS = 1.99;

    private UUID uuid;
    private List<Product> products;

    public PostalParcel(UUID uuid, List<Product> products) {
        this.uuid = uuid;
        this.products = products;
    }

    public double deliveryCosts() {
        if (weight() > 20) {
            return MAX_DELIVERY_COSTS;
        }
        return MIN_DELIVERY_COSTS;
    }xeba

    public int weight() {
        return products.stream().map(Product::getWeight).mapToInt(Integer::intValue).sum();
    }
}

All examples are written with Java 8 and can be downloaded from my gitlab repository.

Write an unit test for the function deliveryCosts with a fixture for the entities. An implementation could look something like this:

private PostalParcel generatePostalParcel(int weight) {
    return new PostalParcel("UUID", generateProducts(weight));
}

private List<Product> generateProducts(int weight) {
    List<Product> products = new ArrayList<>();
    for (int i = 0; i < 5; i++) {
        products.add(new Product("UUID", "Name", weight/4));
    }
    return products;
}

As explained in the previous part, these tests will not cover the entire behaviour we are testing here. We will refactor this to a Property-based test with the help of generators.

Generators

In this certain case we use entities as our input for our unit tests. A possible way of using generated properties for entities is to generate a basic type for each of the needed arguments for our entities. It is easier to let JUnit-Quickcheck generate the entities by creating your own generators; this way your test does not get bloated with properties and is therefore better readable. To create a generator you need to do three things. First create a class that extends from generator<T>:


public class PostalParcelGenerator extends Generator<PostalParcel>

Secondly, implement the constructor with a super to the type you are generating. Lastly, override the function T generate:

public PostalParcelGenerator() {
    super(PostalParcel.class);
}
public PostalParcel generate(SourceOfRandomness sourceOfRandomness, GenerationStatus generationStatus) {
      return null;
}

We can use the generator in our Property-based test in two ways, we can use the @From(T.class) annotation in front of your entity. This annotation tells Junit-Quickcheck which generator to use, for example:


public void deliveryCostsShouldBeMaxWhenWeightIsGreaterThan20(@From(PostalParcelGenerator.class) PostalParcel postalParcel)

It is much easier to let the serviceloader of JUnit-Quickcheck discover the generators by creating a file called com.pholser.junit.quickcheck.generator.Generator in META-INF/services. In this file you add the package name plus class name of the generator. This way, there is even less plumbing going on in your unit tests.

Implement generate

The function generate looks pretty straightforward. It requires you to return the type  you are generating.

public PostalParcel generate(SourceOfRandomness sourceOfRandomness, GenerationStatus generationStatus) {
    return new PostalParcel(gen().make(UUIDGenerator.class).generate(sourceOfRandomness, generationStatus),
            generateProducts(sourceOfRandomness, generationStatus));
}

private List<Product> generateProducts(SourceOfRandomness sourceOfRandomness, GenerationStatus generationStatus) {
    ProductGenerator productGenerator = gen().make(ProductGenerator.class);
    List<Product> products = new ArrayList<>();
    int randomTotalWeight = sourceOfRandomness.nextInt(minWeight, maxWeight);
    while(randomTotalWeight > 0) {
        int maxWeight = sourceOfRandomness.nextInt(1, randomTotalWeight);
        productGenerator.configureMaxWeight(maxWeight);
        products.add(productGenerator.generate(sourceOfRandomness, generationStatus));
        randomTotalWeight = randomTotalWeight-maxWeight;
    }
    return products;
}

There are multiple JUnit-Quickcheck functions going on here. First is the argument SourceOfRandomness. You need this to create random primitive types, like you see in generating the randomTotalWeight. Secondly, you have the argument GenerationStatus, which we do not use in this example, but we can use this to influence the results of a generation for a particular property parameter.

When you want to generate other types than primitives, use the gen() interface. The most common usages are:

  • The gen().make() method makes the generators available to current instance, and configures it with whatever configuration annotations live on the generator class.
  • The gen().type() method asks for an arbitrary generator that can produce instances of the given type. Use this for a string, it will create a total random string with all characters available (even Chinese and Korean characters).

Take notice, as was said in the previous part, if you take a string as argument then the works of Shakespeare in Japanese & Korean are ONE valid input. If you can, wrap the string into its own type, like explained in the Object Calisthenics practise.

Configuring generators

In the implementation of our PostalParcelGenerator you might have noticed the arguments minWeight and maxWeight in generating the randomTotalWeight. These are two variables which we need to configure the generator with. We can do this by creating a public function void called configure. JUnit-Quickcheck requires us to name it like this in order to match this with the annotation we are going to use. In this case we will use the standard JUnit-Quickcheck annotation @InRange, but you can also use your annotation.

private int minWeight = 1;
private int maxWeight = Integer.MAX_VALUE;

public void configure(InRange range) {
 this.minWeight = range.minInt() == Integer.MIN_VALUE ? 1 : range.minInt();
 this.maxWeight = range.maxInt();
}

You can easily configure your generators with the behaviour you need for your test cases. The only problem that remains is that where code is written errors can be made; we all make mistakes. This is also why in my previous post I advised to both use configuration by annotation and to use assumeThat; with this approach mistakes are easily spotted during your tests. After we implemented the rest of the generators, (which you can see in my gitlab repository) we can write our PBT.

@RunWith(JUnitQuickcheck.class)
public class PostalParcelPBTTest {

    @Property
    public void deliveryCostsShouldBeMaxWhenWeightIsGreaterThan20(@InRange(minInt = 21) PostalParcel postalParcel){
        assumeThat(postalParcel.weight(), greaterThan(20));

        assertThat(postalParcel.deliveryCosts(), equalTo(com.baasie.pbt.part1.PostalParcel.MAX_DELIVERY_COSTS));
    }

    @Property(trials = 25)
    public void deliveryCostsShouldBeMinWhenWeightIsLessThanOrEqualTo20(@InRange(maxInt = 20) PostalParcel postalParcel){
        assumeThat(postalParcel.weight(), is(both(greaterThan(0)).and(lessThanOrEqualTo(20))));

        assertThat(postalParcel.deliveryCosts(), equalTo(com.baasie.pbt.part1.PostalParcel.MIN_DELIVERY_COSTS));
    }

}

Creating the generators will cost some startup investment, but afterwards you really benefit from it. For each PBT, you can just add the type of the generator, and maybe some added behaviour you want with it. The minor investment will result in time saving in the long run, and better designed and written software. Additionally the tests will find edge cases that could cause bugs in your software, and resolve them before they occur.

Performance

After my last post I got a comment with a question about performance. I do not know how the exact implementation works, but I do know that running it does not take 100x slower. I did a run in my Intellij for both the tests and the PBT is about 10x slower. In my opinion 10x is not unreasonable, considering the benefit it gives you.

This concludes part 2 of this tutorial, in the next part I will show you how to repeat failed tests.

Also, this post is published on the blog of Xebia

To be able to show you what Property-based testing (PBT) is, let’s start by grasping the concept of a property in programming languages. Since this is a Java tutorial, I will start with Oracle and their definition of a property in their glossary:

Characteristics of an object that users can set, such as the color of a window.

Property is neither a variable/field or a method; it is something in between which is always true in your context. An example is weight in a postal parcel: this always is greater than zero.  In Java the following example implementation would follow:

public class PostalParcel {

    private int weight;
    private String uuid;

    public PostalParcel(String uuid, int weight) {
        this.uuid = uuid;
        if(weight > 0) {
            this.weight = weight;
        } else {
            throw new IllegalArgumentException("");
        }
    }

}

In this case, we made weight a property that always needs to be greater than 0. When designing your software, it is important to search for these properties, because they usually have some sort of business behaviour associated with them.

What is Property-based testing (PBT)

Back to PBT, let’s add a function to PostalParcel that decides the delivery costs of the package. When the weight is greater than 20 the delivery costs will be 4.99 euro, otherwise the delivery cost will be 1.99 euro. We will first write our tests for this, which without PBT, will usually end up something like this:

public class PostalParcelTest {

    @Test
    public void deliveryCostsShouldBeMaxWhenWeightIsLargerThan20(){
        PostalParcel postalParcel = new PostalParcel("uuid", 23);
        assertThat(postalParcel.deliveryCosts(), equalTo(PostalParcel.MAX_DELIVERY_COSTS));
    }

    @Test
    public void deliveryCostsShouldBeMinWhenWeightIsLessThanOrEqualTo20(){
        PostalParcel postalParcel = new PostalParcel("uuid", 19);
        assertThat(postalParcel.deliveryCosts(), equalTo(PostalParcel.MIN_DELIVERY_COSTS));
    }

    @Test(expected = IllegalArgumentException)
    public void shouldThrowIllegalArgumentExceptionWhenWeightIsBelowOne() {
        PostalParcel postalParcel = new PostalParcel("uuid", -100);
    }
} 

Secondly, we have to make these test succeed by writing the implementation of deliveryCosts in a Postal Parcel:


public static final double MAX_DELIVERY_COSTS = 4.99;
public static final double MIN_DELIVERY_COSTS = 1.99;


public double deliveryCosts() {
    if(weight > 20) {
        return MAX_DELIVERY_COSTS;
    }
    return MIN_DELIVERY_COSTS;
}

When you run the tests, they will succeed, but if we think about it, 2 problems will remain. First of all we nicely described our behaviour in the test function name, but the implementation does not match the behaviour. The second test, deliveryCostsShouldBeMinWhenWeightIsLessThanOrEqualTo20, only tests with a weight of 19, but the behaviour clearly states Less Than Or Equal To 20. We can add another test which tests with a weight of 20, but what about the other cases? It will be a lot of work, and a waste of time, to create all the tests our self. Luckily, PBT comes to the rescue!

Furthermore, and this is what PBT is all about, we need a UUID as String for the test.  Since we do not care about the value of that property, we usually use a fixture for a case like this. It might happen that a UUID can give strange errors in executing the behaviour under test now, or in the future. Because we can use anything as a String input, or like Romeu Moura once tweeted:

I advice everyone to watch his talk about Property-based testing.

Now what you usually want to do, is make a Value Object of a UUID, but for now, because i want to show you a PBT example, we will leave it as a String. Even though, you can already see the benefit PBT will give: it makes you think about your input values.

For a more in depth explanation of what Property-based testing is, you can also check the blog post of Hypothesis. Hypothesis also has a Java version, but at the moment this is still a prototype.

JUnit-Quickcheck implementation

Start of with adding the JUnit-Quickcheck dependency to our project.

In order to use JUnit-Quickcheck we need to change 2 things.

First add the @RunWith annotation above our test class.

@RunWith(JUnitQuickcheck.class)

Second, instead of using @Test we use @Property.

@Property
public void deliveryCostsShouldBeMaxWhenWeightIsLargerThan20(){
.....

JUnitQuickcheck will now run each property test 100 times. We will get the added value of this when we let the JUnitQuickcheck generate the property parameters: for each run it will generate a random parameter. Let’s add them:

    @Property
    public void deliveryCostsShouldBeMaxWhenWeightIsGreaterThan20(String uuid, int weight)
        PostalParcel postalParcel = new PostalParcel(uuid, weight);
        assertThat(postalParcel.deliveryCosts(), equalTo(PostalParcel.MAX_DELIVERY_COSTS));
    }
    ....

Running this will most likely fail the tests, because the number can be any int now, but what we want is, for example, Larger Than 20. This can be accomplished in 2 ways, either use assumeThat of JUnit, or use the @InRange of JUnitQuickcheck  in front of the variable.

@RunWith(JUnitQuickcheck.class)
public class PostalParcelTest {

    @Property
    public void deliveryCostsShouldBeMaxWhenWeightIsGreaterThan20(String uuid, @InRange(minInt = 21) int weight){
        assumeThat(weight, greaterThan(20));

        PostalParcel postalParcel = new PostalParcel(uuid, weight);
        assertThat(postalParcel.deliveryCosts(), equalTo(PostalParcel.MAX_DELIVERY_COSTS));
    }

    @Property
    public void deliveryCostsShouldBeMinWhenWeightIsLessThanOrEqualTo20(String uuid, @InRange(minInt = 1, maxInt = 20) int weight){
        assumeThat(weight, is(both(greaterThan(0)).and(lessThanOrEqualTo(20))));

        PostalParcel postalParcel = new PostalParcel(uuid, weight); 
        assertThat(postalParcel.deliveryCosts(), equalTo(PostalParcel.MIN_DELIVERY_COSTS)); 
    }

    @Property
    public void shouldThrowIllegalArgumentExceptionWhenWeightIsBelowOne(String uuid, @InRange(maxInt = 0) int weight) {
        assumeThat(weight, lessThanOrEqualTo(0));

        IllegalArgumentException illegalArgumentException = null;

        try {
            PostalParcel postalParcel = new PostalParcel(uuid, weight);
        } catch(IllegalArgumentException e) {
            illegalArgumentException = e;
        }
        assumeThat(illegalArgumentException, notNullValue());
    }
} 

I would advice to combine both the options, for 2 reasons. First, assumeThat won’t work sufficient enough in small ranges, like in the second example. You will get errors of violating the assumptions. Adding the assumeThat will make your tests more readable and explicit though.

Trials and Success

JUnit-Quickcheck by default will run the test 100 times, each with a random value. In some cases, like the LessThanOrEqual20, running 100 can be a waste (can, because it still is random). For this we can set the trials modifier on the @Property(trials = 25) annotation. This tells JUnit-Quickcheck it should only run the amount of trials set here, in this case 25 should be enough. It should, but it is still random, so it might happen that the checks will succeed the first time, but will fail after a while. This i.m.o. is the real benefit of PBT, it will eventually test all the corner cases for you, and find possible bugs in your system. Success just means it did not find errors this time, but maybe next time it will.

Now we have written our very first, basic, Property-based test for Java. This is already really helpful. In the next part of this tutorial we will go deeper into the framework and we will let JUnit-Quickcheck generate our entities for us as input values.

You can find the sourcecode of this example on my gitlab.

Also, this post is published on the blog of Xebia

 

The practise of Behaviour Driven Development (BDD), and using tooling to support this like cucumber, has inspired many posts and talks. The recurring questions are predominantly about how teams and people pull these tools out of its collaboration context, and bring it into the test automation context. Cucumber’s post might perhaps be one of the best on this topic,

Last year I started giving talks about BDD based on my own experiences. One pitfall that always seems to pop up, at least in the Netherlands, is about imperative versus declarative (or implementation vs intention).

To give you an idea what the difference is, here is an example of describing your specification in a imperative way (Implementation based):


Scenario: Redirect user to originally requested page after logging in

   Given a User 'dave' exists with password 'secret'
   And I am not logged in
   When I navigate to the home page
   Then I am redirected to the login form
   When I fill in 'Username' with 'dave'
   And I fill in 'Password' with 'secret'
   And I press 'login'
   Then I should be on the home page

Here is the same script but written in a declarative way (Intention based):


Scenario: Redirect user to originally requested page after logging in
   Given I am an unauthenticated guest
   And I have a valid user account
   And I attempt to access restricted content
   When I log in
   Then I have access to the restricted content

Describing our specifications imperatively can bring significant problems. Our application, and in this case our UI, will be highly coupled to our specifications. When we change the UI, then we automatically need to change our specifications. Even worse, when we test-automate these specifications trough our UI, you are coupling your test code directly on your UI, and any change will lead to a lot more effort. In this case you most likely be putting more effort into refactoring your test-automation code, instead of adding that business value.

When we describe our specifications declaratively, changing the UI does not mean changing the intention, or behaviour of the code. in this case we can change the way a customer authenticates in the UI, but our intention is still the same so the specification script stay the same. Only the page objects and the reference to the page objects in your glue code needs to be changed. This could sound familiar to most developers: making your application (that means your test code too) as low coupled as possible.

The most common argument I’ve heard when giving talks or consultations is “well my domain expert knows the UI, so they will always be referencing the UI instead of the behaviour or intention of the software”. This is even more likely the case with UI designs. I always had a hard time giving people a workable solution to their problem when I tried explaining the above. That is, until I attended a 2-day Master Class “Event Storming” by Alberto Brandolini. For those who are unfamiliar with Event Storming, it is a workshop format for quickly exploring complex business domains. I can try to explain everything here, but it is better to read it first hand from the creator himself on his blog: or read and order his book (Highly recommended!).

The thing that stood out most to me during this workshop is that Event Storming is
a collaboration tool that does not require IT-knowledge. Event Storming is a visual way of talking about models and behaviour which leaves out the technical implementations that domain experts do not know about or need to know about. This model can  be implemented in the code right away! The only real challenge I encounter when using Event Storming is that developers have the tendency to talk in IT language.

Getting the UI out of the domain expert is also the hard part for running good refinements with an UI design. In my opinion, this is why using techniques like Event Storming or Example mapping are so important and useful to use during your refinements. At least for me and the teams I worked with, this was a major improvement to the quality and sharing the right mind-set about behaviour and specifications

In one of my next blog posts, I will explain more about combining these two techniques!