Master Teset Specification SRCP

Master Test Specification - SRCP

Master Test Specification
for
Simple Railroad
Command Protocol(SRCP)
23 July 2010

Tester,
Ankit Singh
High Integrity System
Fachhochschule Frankfurt am Main
University of Applied Sciences

1

Master Test Specification for SRCP

Table of Contents

1. Chapter 1: Introduction............................................................ 4
2. Chapter 2: Test plan................................................................. 5
2.1 Test level....................................................................... 5
2.1.1 Black Box Testing.............................................. 5
2.1.2 White Box Testing............................................. 6
2.2 Test Object and Environment....................................... 7
2.3 Tools.............................................................................. 7
2.4 Schedule........................................................................ 7
2.5 Test areas...................................................................... 8
2.6 Test tuning..................................................................... 9
3. Chapter 3: Test Specification.....................................................10
3.1 Testing techniques......................................................10
3.2 Test Metrics used........................................................10
3.3 Requirement list.........................................................11
4. Chapter 4: Communication........................................................17
5. Chapter 5: Black Box Test Cases …...........................................18
5.1 Mode/Protocol Testing ….............................................18
5.2 Wrong Documentation of Specification....…................21
5.3 Cause-Effect Graph ….................................................23
6. Chapter 6: White Box Testing....................................................26
6.1 White Box Testing Results............................................. 26
6.1.1 Module: ddl-s88.c............................................. 26
6.1.2 Module: srcp-fb.c.…...........................................27
7. Findings …................................................................................30
8. Conclusion …............................................................................31
9. References: ..............................................................................32

2


List of Tables

2.4 Project Schedule …..............................................................8
3.2 Test Metrics ….....................................................................10
3.3.1 General Requirement ….................................................. 13
3.3.2 Bus Allocation................................................................. 16
5.1.1 Valid / correct commands …............................................ 18
5.1.2 Invalid / incorrect commands …...................................... 20
5.2 Wrong Test Specification commands …...........…................ 22
5.3 Handshake Cause-Effect graph Decision ….........................24

List of Figures

2.1.1 Black Box Testing …...........................................................5
2.1.2 White Box Testing …..........................................................6
3.3 General flow of operations................................................12
4 Communication diagram.........................................................18
5.3 Cause-effect Graph...............................................................23
6.1.2.1 General Functionality Relations …...............…............... 27
6.1.2.2 FB generation calls ….....................................................28
6.1.2.3 SET Function of FB …..................................................... 28
6.1.2.4 GET Function of FB …......................................................28
6.1.2.5 INIT Function of FB …......................................................28

List of Charts

5.1 Bar Chart Pass vs Fail ….................................................... 23
5.2 Pie Chart Pass vs Fail ........................................................ 23

3

Chapter1
Introduction

Simple Railroad Command Protocol(SRCP), is a middle man platform for various

vendors of model railway. It is an Internet protocol for controlling and programming

(digital) railway systems. The protocol is based on the application layer (layer 7) of

the OSI Reference Model. The major goals of SRCP is to provide a common

operating language among railway models as well as providing multiuser support.

Moreover, scalability issues like safety features are bundled into SRCP.

The aim of this document is to identify layout and investigate test scenarios and test

cases for the software at hand. The first section of the document focus on the myriad

components of test planning in view of SRCP followed by specification of test areas.

The final section examine the test cases identified, defined and be investigated.

4


Chapter 2
Test plan

This section details the various components required for the testing of SRCP. It identifies
the level of testing, the environment and tools to be used. Additionally, time frame and
major test areas are scrutinized.

2.1. Test level

The level of required at this version of testing is unit testing with black box testing
method and white box testing.

2.1.1 Black-box Testing

The black-box tests are going to be automatized using PyUnit python test package.
Since only the SRCP server application is going to be tested, a client application is
going to run unit tests on server. The general black-box testing strategy includes:
1. Running different valid tests on each of commands in combination with each
device group; and
2. Running different invalid tests on each of commands in combination with
each device group.

5

Figure 2.1.1 Black Box Testing

Black-box tests are going to be performed by running client application written in
Python programming language. The client application is going to connect to, and
communicate with SRCP server via TCP protocol.

2.1.2 White-box Testing

Main criteria in white-box testing is going to be code coverage, which means that all
branches in control ow graph are going to be covered. The white-box testing is going
to be performed on specific modules of SRCP server to ensure the safety of the
tested modules. The list of modules to be tested is listed below:
1. srcp-fb.c and
2. ddl-s88.c

Figure 2.1.2 White Box Testing

White-box tests are going be performed using VIM or Emacs with help of ctags &
etags respectively for moving around the C code of the SRCP Project.

6


2.2. Test Object And Environment

SRCP Server version 2.1.1 is a platform independent model. One SRCP server is
also called Central Unit (CU) and covers the functionality of exactly one CU. The

SRCP server is open for communication in two directions. The first side covers the
communication between the railroad and the SRCP server over several buses. And
The other side of communication is between the SRCP server and the clients. The
servers task is to convert commands from the clients to commands for the model
railroad and send reports back to the clients. The communication between clients and
SRCP server in both directions is done with via SRCP.

We choose Linux 9.10 as a major platform for setting up the test environment.
In liaison, windows based platforms will be used if found necessary for clients of
SRCP.

2.3 Tools

Various tools for testing will be in action for the testing process. We mainly categorize
testing tools into
• Documentation tools: Open office suit
• Unit test scripting tools: pyUnit/Python based script

2.4 Schedule

As this is a project for the fulfillment of Advanced Testing Methods module, the
duration of the project will be based on the semester duration. The submission of the
Project is on 25-7-2010.
The Detailed testing project Schedule table is given below.

7


2010 APRIL MAY JUNE
WEEK 3 4 1 2 3 4 1 2 3 4
Client Provide Specification
Documents for SRCP
Test Planning
Master Test Specification I
Master Test Specification II
Black Box Testing
White Box Testing
Test Report
Final MTS SRCP 0.8.3
Delivered to Client
Table 2.4 Project Schedule

Working Working Parallel
2.5 Test areas

Client – Server Network Communication

Network Communication between a client and a server is the base of software. The
client and server communicate in network using TCP protocol. The network
establishment should be proper so that client sends requests to server and server
acknowledge to the request & sends back appropriate reply to the client. So, Network
communication is one of the important area to be tested.

SRCP connection Mode

After a client established a successful connection with SRCP server, a client can
switching into different mode given in SRCP. Mainly, there are three modes: I)
handshake, II) command, III) info modes. Switching between different modes need to
tested properly so that the commands of the individual modes can be used for
different type of service given by server.

8

Test areas are defined according to the priorities set out by the documentation of
SRCP. We have pointed out the following test areas:
1. Level of priority
a. MUST/MUST NOT tagged requirements
b. SHALL/RECOMMEND tagged requirements
c. CAN/OPERATIONAL tagged requirements
2. Valid/Invalid requirements
3. General flow of operations requirements

2.6 Test tuning

As this is a very first test document, tuning and improvement are always at the table.
Thus, we will employ rudimentary test tuning techniques like that of feedback during
presentations, one to one observation, traceability matrix the SRCP main
documentation.

9


Chapter 3
Test Specification

This section of the document examine requirements of the system at hand and propose list
of requirements for the advancement of test cases.

3.1 Testing techniques

Since the level of testing required at this stage is unit with black box testing
techniques, we found that it is advisory to explore and identify best method for
accomplishing the object at hand. Equivalence partitioning, boundary value analysis,
all pair testing for parameter and traceability matrix specification as an appropriate fit.

3.2 Test Metrics Used
Table 3.2 Test Metrics

Test metric Definition Purpose How to calculate

Defect severity The severity level of a Provides indications Every defect has
defect indicates the about the quality of severity levels
potential business the product under attached to it.
impact for the end user test. High-severity Broadly, these
(business impact = defects means low are Critical, Se-
effect on the end user x product quality, and rious, Medium
frequency of vice versa. At the and Low.
occurrence). end of this phase, this
information is useful to
make the release

10

decision based on the
number of defects and
their severity levels.

Test coverage Defined as the extent This metric is an in- Coverage could
to which testing covers dication of the com- be with respect
the products complete pleteness of the testing. to requirements,
functionality. It does not indicate functional topic
anything about the list, business
effectiveness of the flows, use cases,
testing. This can be etc. It can be
used as a criterion to calculated based
stop testing. on the number
of items that
were covered vs.
the total number
of items.

3.3 Requirement list

Before listing out of the details of the SRCP requirements, it is useful to identify the
major blocks of the protocol. The following picture depict the overall process of SRCP
protocol.

11


Figure 3.3 General flow of operations

Based on the test areas specified in section 2.5, we have the following
1. Commands
a. Command mode
b. Info mode
2. Reply
3. Server state
4. General valid/ invalid requirements

12

Table 3.3.1 General Requirements

Sr.No Item Requirement
1 General: SRCP server can manage and provide several CUs is
SRCP Server valid
2 General: Separating CUs is made by different bus numbers
SRCP Server
3 General: SRCP is based on the data transfer technologies of TCP.
SRCP Server One port is occupied
4 General: SRCP servers have to be prosecuted on the different TCP
SRCP Server ports if there are several CUs connected to a computer
without a common
SRCP server
5 General: All character codes are described in their decimal form as
Lexical Units # followed by the numeric values of the character
6 General: Entire communication consists of the characters of the
Lexical Units 7 bit ASCII character set in the range from #32 to #127.
Also,#9(TAB) for space ,#10 and #13 (CR,LF) for end
line are valid
7 General: Numbers are processed at least as singed 32bit integers.
Lexical Units Floating point numbers are NOT used.
8 Command Commands are processed in hand shake and command
mode only. They are not valid during info mode and
MUST be ignored
9 Command Commands consist of a command word, must followed
by a command parameter list separated by white space.
10 Command It is not valid to resume a command in the following line.
11 Command If the list consist of several elements they need to be
separated by white space. Elements containing white
space are not valid
12 Command The end of line always consists of the character #10 n .A
prefixed #13r is valid .The character #13 is ignored.
13 Command A single line including the end of the line MUST NOT
exceed over a length of 1000 characters
14 Command and All words are case-sensitive .Commands and replies of
Reply the SRCP all always written in uppercase letters
15 Reply A server HAS TO reply to every command of a client
16 Reply A reply SHALL be single line. With the sign (Backslash,

13


#92) before the LF(resp CRLF) it is marked that the reply
contains another line
17 Reply IF the reply is an error message the command MUST
NOT be executed
18 Reply A reply is complete with the effective end of line .
Processing more than one answer in one line is not valid
19 General: After the server has sent its reply it processes the next
SRCP Server command. That way commands are handled by a strictly
two-way communication between the client and the
server in a changing order of commands and reactions.
The server HAS TO process the commands in the
chronological order of receiving them
20 General : On connection the hand shake mode is activated.
SRCP Server Further operation parameters are determined by the client
in this mode. The connection changes from the hand
shake mode to either the command mode or the info
mode according to client’s demand .From all three modes
the connection can be closed any time.
21 Command Commands MUST be processed by the server in the
order of receiving them
22 Command Unknown or unrecognized commands MUST be replied
with the message “410 ERROR unknown command”
23 General: A client establishes a TCP/IP connection with the server.
SRCP The server sends a single line welcome string .Then the
Server Communication parameters and the desired operation
mode are negotiated between client and server. The
operation mode starts with the final command GO sent
by the client and it has to be confirmed by the server.
24 General: The server MUST differ between the client session
SRCP Server internally.
25 General: After establishing connection between client and server,
SRCP Server The server sends a text line (The welcome) to client .
26 General: Every of these information MUST be seen as a key-
SRCP Server value pair. The order of these keys is undetermined .It is
also generally possible to specify keys with different
values repeatedly .For every key exactly one value is
valid.
27 General: During the hand shake phase no other commands than

14

SRCP Server listed are valid.
28 Command SET PROTOCOL SRCP<VERSION>,
The server sends as reply either 201 OK PROTOCOL
SRCP OR 404ERROR unsupported protocol
29 Command and SET CONNECTIONMODE SRCP<MODE>,valid are
Reply INFO FOR the unidirectional command mode .The
server send either 202 OK or 401 ERROR unsupported
connection mode
30 Command and GO by this command the hand shake phase is
Reply Completed and the chosen operation mode is activate.
The server send 200 OK<ID> immediately before that
the to the client, Otherwise 402 ERROR insufficient data
and remains in the hand shake mode .The field<ID>
marks the numerical session ID given by the server
MUST be unique and MUST NOT be identical to zero.
31 General: After completing the hand shake the agreed attributes can
SRCP Server not be change and are valid for the entire connection.
32 General: Poll of valid messages during handshake 200 OK <ID>,
SRCP Server 201 OK PROTOCOL SRCP,202 OK CONNECTION
OK ,400
ERROR unsupported protocol, 401 ERROR unsupported
connection mode, 402 ERROR insufficient data,500
ERROR out of resource.
33 Command Valid command types:
GET,SET,CHECK,WAIT,INIT,TERM,RESET,VARIFY
34 Command After the INIT the indicated element is in default state.
During INFO mode 101 INFO INIT <devicegroup>
<addr> is sent for the affected element
35 Command By the command TERM the initialization is cancelled
and the effected element is in default state. Before using
it again the affected element has to be reinitialized.
During INFO mode a message in the form of 102
INFO<devicegroup><addr> is sent for the affected
element.

15


The allocation of device groups and commands on the buses

The following overview is supposed to show the allocation of device groups and commands
on the buses.
Table 3.3.2 Bus Allocation
SET | GET WAIT INIT TERM RESET VERIFY
CHECK
GL 1 1 -- 1 1 1 --
SM 1 1 -- 1 1 1 1
GA 1 1 -- 1 1 1 --
FB 1 1 1 1 1 1 --
TIME 0 0 0 0 0 0 --
POWER 1 1 -- 1 1 -- --
SERVER -- 0 -- -- 0 0 --
SESSION 0 0 -- -- 0 -- --
LOCK 0 0 -- 0 0 -- --
DESCRIPTION 0 0 -- -- -- -- --

16

Chapter 4
Communication

A client – server communication diagram of SRCP
Figure 4 Communication Diagram

17


Chapter 5
Black Box Test Cases

This section contains all test cases for Black Box testing of the Simple Railroad Command
Protocol.

5.1 Mode/Protocol Testing:-

1. SRCP server Protocol version 0.8.3

Table 5.1.1 VALID / CORRECT COMMANDS

Test Case Command Expected Output Behavior Description
ID No.
1_0 TCP connection 4303 srcpd V2.0.12; Indicates server is
SRCP 0.8.3; running & returns the
SRCPOTHER 0.8.4- version
wip
1_1 SET PROTOCOL SRCP <TimeStamp> 201 Client establishes
0.8 OK connection with SRCP
PROTOCOL SRCP server
1_2 SET PROTOCOL SRCP <TimeStamp> 201 “qwerty” is ignored
0.8 qwerty OK PROTOCOL and client establishes
SRCP connection with the
server
1_3 SET CONNECTIONMODE <TimeStamp> 202 Client program
SRCP INFO OK SRCP INFO switches to INFO mode

18

1_4 SET CONNECTIONMODE <TimeStamp> 202 “123” is ignored and
SRCP INFO 123 OK SRCP INFO the client program
switches to INFO mode
1_5 SET CONNECTIONMODE <TimeStamp> 202 Client program
SRCP COMMAND OK switches to COMMAND
CONNECTIONMOD mode
E
1_6 SET CONNECTIONMODE <TimeStamp> 202 “22” is ignored and
SRCP COMMAND 22 OK client program switches
CONNECTIONMOD to COMMAND mode
E
1_7 GO <TimeStamp> 200 Client sends above
OK <numerical entered commands to
session ID> server (handshake
completes)
********** Command Mode ******* ****************** ***********************

1_8 INIT 1 FB 0x1111 200 OK Initialization of bus 1
with address
1_9 GET 1 FB 0x1111 100 INFO 1 FB 0 Getting information of
9846074 bus 1
1_10 INIT 0 TIME 1 1 200 OK Initialization of time
1_11 SET 0 TIME 364 22 22 22 200 OK Setting time with
22H:22M:22S
1_12 GET 0 TIME 100 INFO 0 TIME Getting time of Bus 0
364 22 22 28
1_13 INIT 1 POWER <TimeStamp> Initialize power
200 OK device on bus 1
1_14 SET 1 POWER ON <TimeStamp>200 Enable power on bus
OK 1

19


1_15 TERM 1 POWER <TimeStamp>200 Disable power on
OK bus 1
1_16 GET 0 SERVER <TimeStamp>100 Informs the current
INFO state of the
0 SERVER server is running
RUNNING
1_17 TERM 0 SERVER <TimeStamp>200 Quits the running
OK server and closes all
connections
1_18 GET 1 DESCRIPTION <TimeStamp>100 covers all device
INFO 1 groups
DESCRIPTION GA supported by the
GL FB SM POWER concerned
LOCK INFO server on bus 1
1_19 GET 0 DESCRIPTION <TimeStamp>100 INFO covers all
INFO 0 device groups
DESCRIPTION supported by the
SESSION concerned
SERVER TIME GM server on bus 0
1_20 INIT 1 GA 200 N 1 <TimeStamp>200 Command initializes
1 OK a GA at 200
address addr in the
server.
1_21 SET 1 GA 200 1 1 200 OK The port 1 of the
100 decoder with the
address 200 is set
on the value 1
for 100 ms.
1_22 GET 1 GA 200 1 <TimeStamp>100 Server sends all
INFO 1 GA 200 1 0 available
information about
the current
state of the switch
decoder
specified by 200
address
1_23 SET 1 LOCK GA 101 10 <TimeStamp> 200 Sets a lock on GA on
OK bus 1 address 101 for
10 msec
1_24 GET 1 LOCK GA 101 <TimeStamp> 100 Returns a lock status
INFO 1 LOCK GA of GA on bus 1

20

100 <LOCK> address 100
1_25 TERM 1 LOCK GA 101 <TimeStamp> 414 Terminates lock on
ERROR device device GA on bus 1
locked address 101
1_26 TERM 0 SERVER <TimeStamp> 200 Terminates Server
OK successfully

Table 5.1.2 INVALID / INCORRECT COMMANDS

ID No.
2_1 SET PROTOCOL SRCP 400 ERROR Not establishes
0.9 unsupported protocol connection with wrong
version
2_2 <TimeStamp> 402 Pressing enter without
ERROR insufficient any input return Error
data
2_3 SET <TimeStamp> 410 Returns Error with
ERROR unknown incomplete command
command

5.2 Wrong Document Specification

Table 5.2 Wrong Test Specification Commands

ID No.
3_1 set ConnEctionMode Srcp 410 ERROR Fail : <TimeStamp>

21


Command unknown command 202 OK
CONNECTIONMODE
3_2 SET 1 FB 0x111111 0 100 INFO 1 FB Fail: <TimeStamp> 422
0x111111 0 ERROR unsupported
device group
3_3 GET 0 SESSION 2 100 INFO 0 Fail : <TimeStamp>422
SESSION 2 ERROR unsupported
device group
3_4 RESET 0 SERVER 101 INFO 0 Fail :
SERVER <TimeStamp>423
ERROR unsupported
operation
3_5 TERM 0 TIME 102 INFO 0 TIME Fail : <TimeStamp>422
ERROR unsupported
device group

This all above Test cases is been implemented in a python unit testing script
'mod_srcpconn_handler.py'

All the above result can be found by running the python script with default as
HOSTID = localhost
PORTID = 4303

22


5.3 Cause effect graph

C1: SRCP version set to >= 0.9
C2: Mode selection is “COMMAND”
C3: Mode selection is “DEFAULT”
C4: Mode selection is “INFO”
C5: GO command is issued to the server

E1: Version NOT accepted
E2: Acceptable commands can be entered and their respective reply messages can be
displayed
E3: Commands are ignored, only the info output is sent to the client in a unidirectional
mode
Graph 5.3: Cause-Effect Graph

23

Table 5.3: Handshake Cause-Effect graph Decision

Direction Causes & Rule 1 Rule 2 Rule 3 Rule 4
down Effects

V C1(>=0.9) 1 0 0 0

V C2(cmd) 0 1 0 0

V C3(def) 0 0 1 0

V C4(inf) 0 0 0 1

V C5(go) 0 1 1 1

V E1(verNOTok) 1 0 0 0

V E2(cmdentry) 0 1 1 0

V E3(infentry) 0 0 0 1

We ran 32 black box tests cases written in 'mod_srcpconn_handler.py'.

And we found 5 Bugs which was wrongly described in the SRCP specification.

BarChart 5.1: Pass vs Fail Test cases
Test Run
32 Tests Run (27 Pass & 5 Failed)

30 27

25

20
Pass

15

10
5

5

0
Pass Fail

24


Pie Chart 5.2: Pass vs Fail Test cases

Test Run
32 Tests Run (27 Pass & 5 Failed)

5

Pass
Fail

27

The simple statistics done on the results of Test cases implemented is given below:

Pass Fail
27 5
Percentage 84.38% 15.63%

Fail: 15.63% is very high if considered with the safety. We need to reduce this percentage
to ensure the accuracy and safety of the product.

It may goes down if implementing more test cases in future or may also goes up. But
Developer need to work on the clarity of SRCP specification and functioning of the
commands and services in SRCP modules.

25

Chapter 6
White Box Test

We performed white box testing on the following modules of the SRCP:
1. srcp-fb.c
2. ddl-s88.c
Followings are the bugs findings performed by doing statement and logic coverage white-
box testing.

6.1 White Box Testing Results

Comments:
• Inspections and walk through of the code performed and reviewed
• General safe/secure coding guidelines were studied and applied

6.1.1 MODULE: ddl-s88.c

1. function: init_bus_S88:
Bugs: result from call to ioperm(S88PORT, 3,1) (which is checking port accessibility)
is not checked.
Suggestions: makes sure the return value is OK

2. function: *thr_sendrec_S88:
Bugs: usleep not checked for incorrect result (lines 463 and 468 in source)
Suggestions: check the result returned by usleep()

26


3. function: *thr_sendrec_S88:
Bugs: sleep function result not checked against incorrect values
Suggestion: see 2.

4. function: FBSD_ioperm:
Bugs: sprintf considered unsafe since the size of data is not checked dynamically,
occurs on line 540 of the source
Suggestion: replace sprintf() occurences with snprintf(), see man snprintf

6.1.2 MODULE: srcp-fb.c

• Call graph diagram of main source files and their relation to srcp-fb.c:

Figure 6.1.2.1 General Functionality Relations

27

• FB module generation calls:

Figure 6.1.2.2 FB generation calls
• GET & SET function call graphs of FB (Internal calls)

Figure 6.1.2.3 SET functions of FB

Figure 6.1.2.4 GET functions of FB

• Initialization (INIT) function of FB (Internal Calls)

Figure 6.1.2.5 INIT function of FB

1. function: enqueueInfoFB:
Bug: msg length HARDCODED and the the msg length is not checked dynamically
Suggestions: should be passed any specified message, and calculate size
dynamically

28


2. function: queue_reset_fb:
Bug: usleep not checked for incorrect result
Suggestions: check the result returned by usleep()

3. function: infoFB:
Bug: msg overflow can occur, so message length should be dynamically checked
Suggestion: Check for the overflow

Conclusion:

We managed to find Bugs in the module by doing Inspections and walk through of the
code. The suggestions should be implemented to avoid bugs in the modules.

29

Findings

• TCP-port 4303 assigned by IANA but not mentioned in specification

• Incomplete data in specification leads to confusion

• Insufficient details in Specification

• In some cases commands are not working on certain device groups

• Differences between Specification & srcpd implementations.

• Proper Return checks are not done in the source codes which can lead to

crashing of the system.

• Overflow check and avoid passing hard-coded message string length. It should

be taken dynamically.

30


Conclusion

• Full code coverage using black-box testing isn’t feasible

• White-box in combination with black-box testing considerably increases code

coverage

• Increased code coverage = Increased code safety

31

References

1) Lecture notes and slides of Dr. Schoenfelder.

2) The Art of Software Testing, Second Edition, Glenford J. Myers

3)
http://en.wikipedia.org/wiki/Test_case

32

Master Teset Specification SRCP

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Master Teset Specification SRCP (20)

More from Ankit Singh (16)

Master Teset Specification SRCP