Project FoX: A Tool That Offers Automated Testing Using a Formal Approach

PROJECT FOX
A TOOL THAT OFFERS AUTOMATED TESTING USING A FORMAL
APPROACH

Ivo Neskovic
CITY College Thessaloniki, an International Faculty of the University of Sheffield
462

AGENDA

> SOFTWARE ENGINEERING: WHAT COULD GO WRONG?
> FORMAL METHODS
> PROJECT FOX
> CASE STUDY: THE BUFFER SYSTEM
> CONCLUSIONS AND FUTURE WORK
> BIBLIOGRAPHY

2

THE PROBLEM OF SOFTWARE ENGINEERING
> Faulty systems are a common notion nowadays.
> SE is an engineering discipline, yet lacking the engineering formality.
> Subjective and informal testing.
> Impossible to prove that the system:
– Does what it is supposed to do.
– Does not do what it is not supposed to do.
> Needs structured and precise system designs.

3

FORMAL METHODS
> The applied mathematics of computer systems engineering, used to specify and
model the behaviour of a system and mathematically verify that the system
design and implementation satisfy functional and safety properties.
> Specification Languages:
– Abstract State Machines
– Generalized State Machines
– Communicating Sequential Processes
– Specification and Description Language
– Petri Nets
– Temporal Logic of Actions
– B and event-B method
– Z

4

FORMAL METHODS AT A TRIAL
> Benefits:
– Specification may be used as a basis for proving the presence or lack of
certain properties in the design and by inference in the developed system.
– Mathematical proof of correctness (Theorem proving).
– Model checking (Proving desired properties in a design).
– Formal Testing.
> Used mainly for safety critical systems such as aerospace engineering.
> Criticism:
– Expensive and time consuming approach (though questionable).
– Lack of tooling support.

5

INCORPORATING FORMAL METHODS IN THE
DEVELOPMENT CYCLE

6

PROJECT FOX
> Produce the complete set of test cases from a formal specification.
> Execute the tests on the systems implementation.
> Locate errors and non-equivalences and report them to the user.
> Developed in Java for Java.
> Compatible with Java Standard Edition, Enterprise Edition, Mobile Edition.
> Can be extend to work in conjunction with popular Java frameworks.
> Operates on compiled bytecode with the addition of a few specific annotations.
> Utilizes the test drivers of JUnit.
> FoX provides a bridge between regular Java developers and the benefits of
complete positive and negative testing, proven to find all faults.

7

USING PROJECT FOX
> Two artefacts necessary:
– Formal specification of the
system.
– The system's implementation.

8

BUFFER CASE STUDY – DESCRIPTION
> Simple buffer in a factory.
> Accepts parts, any parts.
> Parts have a name and an ID.
> The buffer has a capacity of 2.
> The buffer can be empty, partially full
or completely full.
> Supports adding and removing items.
> If the capacity is reached, no
additional items can be placed in the
buffer unless an item is removed first.

9

BUFFER CASE STUDY – FORMAL
SPECIFICATION
> Modeled as a Generalized State
Machine (stream X-Machine).
> A theoretical model of computing,
pioneered by Samuel Eilenberg in
1974 (X-Machine).
> Separates flow control from
processing.
> Flow control is abstracted to a level
suitable for representation as a finite
state machine.
> Complex data structures are modeled
as an infinite memory.
> Able to model both static (data) and
dynamic (control) parts of a system.
10

SPECIFICATION (cont.)
> Simple buffer in a factory.

< xMachine name = " Buffer " >

> The buffer can be empty, partially full or completely full.

< states >
    < state initialState = " true " > empty </ state >
    < state > non_empty </ state >
    < state > full </ state >
</ states >

11

> Accepts parts, any parts.

< input name = " part " ref = " BufferObject " / >

> The buffer has a capacity of 2.

< types >
    < builtInType name = " capacity " type = " integer " / >
    < builtInType name = " buffer " type = " set: BufferObject " / >
</ types >
< memory >
    < memoryBlock ref = " buffer " initialValue = " null " / >
    < memoryBlock ref = " capacity " initialValue = " 2 " / >
</ memory >

12

> Parts have a name and an ID.

< types >
    < complexType name = " ItemType " >
        < attributes >
            < builtInType name = " type " type = " string " / >
        </ attributes >
    </ complexType >
    < complexType name = " BufferObject " >
        < attributes >
            < complexType name = " type " ref = " ItemType " / >
            < builtInType name = " itemId " type = " integer " / >
        </ attributes >
    </ complexType >
< /type > 13

> Supports adding and removing items.
< functions >
    < function name = " add_part " >
        < guard >
            !buffer. contains ( part ) && buffer . size () + 1 < capacity . value ()
        </ guard >
        < body > buffer . add ( part ) ; </ body >
        < output > Part Added </ output >
    </ function >
    ...
</ functions >
< transitions >
    < transition >
        < startingState > empty </ startingState >
        < appliedFunction > add_part </ appliedFunction >
        < endingState > non_empty </ endingState >
    </ transition >
    ...
14
</ transitions>

BUFFER CASE STUDY – IMPLEMENTATION
public class ItemType {

    private String type;

    public ItemType(String type) {
        this.type = type;
    }
}

public class BufferObject {
    private int itemId;
    private ItemType type;

    public BufferObject(int itemId, ItemType type) {
        this.itemId = itemId;
        this.type = type;
    }
15
}

> @Xmachine annotating the class representing the system modeled with the
specification.
> XMachineModel – a class representing the model, containing a number of
useful helper methods.

@XMachine(inputType = "BufferObject",
sampleInputs = {
    "integer: 10, ItemType: (string:Box)",
    "integer: 17, ItemType: (string:HeavyBox)",
    "integer: 25, ItemType: (string:ReallyHeavyBox)",
    "integer: 20, ItemType: (string:Dragon)",
    "integer: 17, ItemType: (string:Planeswalker)",
    "integer: 187, ItemType: (string:Nekrataal)",
    "integer: 23, ItemType: (string:Michael Jordan)"
})
16
public class Buffer extends XMachineModel {

> @XMMemoryBlock – a field level annotation, associating Java data structures
with their specification equivalents

@XMMemoryBlock(name = "buffer")
private List<BufferObject> buffer;
@XMMemoryBlock(name = "capacity")
private int capacity;

public Buffer() {
    super("Buffer");
    buffer = new LinkedList<BufferObject>();
    capacity = 2;
}
17

> @XMFunction – a method level annotation, referencing the modeled functions
implementations.
> reportOutcome( outcome: String) – one of the many helper methods
of the XmachineModel class.

@XMFunction(name = "add_part")
public void addPart(BufferObject part) {
    if (!buffer.contains(part) && buffer.size() + 1 <
        capacity) {
        buffer.add(part);
        reportOutcome("Part Added");
    }
}
18

BUFFER CASE STUDY – EXECUTING FOX

19

BUFFER CASE STUDY – EXECUTING FOX
(implanted error)
if (!buffer.contains(part) && buffer.size() + 1 <
        capacity) {
    buffer.add(part);
    capacity++;
    reportOutcome("Part Added");
}

20

BUFFER CASE STUDY – GENERATED TEST
CASES
> Tests report the sequence of inputs used for the specific scenario, the sequence
of expected outputs and the actual output.
> Outcome is reported to the user via the usual JUnit red / green notifications.

<tests>
    <test testID=”1”>
        <input>[ itemId: 187 type: Nekrataal ]</input>
        <expectedOutput>[ Part Added ]</expectedOutput>
        <output>[ Part Added ]</output>
    </test>
    <test testID=”2”>
        <input>[ itemId: 17 type: Planeswalker, itemId: 20 type: Dragon]</input>
        <expectedOutput>
            [ Part Added, Part Added – Become Full ]
        </expectedOutput>
        <output>[ Part Added, Part Added – Become Full ]</output>
    </test>
</tests>
21

CONCLUSIONS AND FUTURE WORK
> FoX enables developers to leverage the already proven theories for formal testing.
> Provides a fully automated testing process, ranging from complete test set
generation (satisfying some design for test conditions), to test preparation and
execution.
> Operates on any Java based software system, being transparent to it's underlining
technologies.
> Provides complete positive and complete negative testing.
> Nest steps:
– Thorough evaluation.
– An additional tool to make the specification step easier and closer to the
developer, aiming to “hide” the formality as much as possible.
– NetBeans and Eclipse integration.
– A standalone X-Machine IDE providing additional related functionalities.
– Branch out to other languages and frameworks (eg. C# and .NET).
22

BIBLIOGRAPHY
> S. Eilenberg, Automate, Languages and Machines, Vol. A. Academic Press, London,
1974.
> M. Holcombe, “X-Machines as a basis for dynamic system specification,” Software
Engineering Journal, vol. 3(2), pp. 69-76, 1988.
> F. Ipate and M. Holcombe, “Specification and Testing using Generalized Machines: a
Presentation and a Case Study,” Softw. Test. Verif. Reliab, vol. 8, pp. 61-81, 1998.
> M. Holcombe and F. Ipate, Correct Systems: Building a Business Process Solution.
Springer, Applied Computing Series, November 1998.
> G. Eleftherakis and A. Cowling, “An Agile Formal Development Methodology,” in 1st
South Eastern European workshop on Formal Methods (SEEFM 03), (Thessaloniki),
pp. 36-47, Nov. 2002. Agile Formal Methods: Practical, Rigorous Methods for a
changing world.
> P. Kefalas, G. Eleftherakis, and E. Kehris, “Communicating X-Machines: a practical
approach for formal and modular specification of large systems,” Information and
Software Technology, vol. 45, pp. 269-280, Apr. 2003.
23

Ivo Neskovic http://twitter.com/trumpets
CITY College ivo.neskovic@gmail.com

Project FoX: A Tool That Offers Automated Testing Using a Formal Approach

More Related Content

What's hot (20)

Viewers also liked (8)

Similar to Project FoX: A Tool That Offers Automated Testing Using a Formal Approach (20)

Recently uploaded (20)

Project FoX: A Tool That Offers Automated Testing Using a Formal Approach