QSM Mendix Benchmark Report

Mendix
Benchmark Report
September 2014

“You’ve got to be very careful
if you don’t know where you’re going,
because you might not get there.”
[Lawrence Peter “Yogi” Berra]
2

DESIGNING THE RIGHT COURSE
Once a ship sets sail on the ocean, the helmsman consults a nautical chart. Even though
he cannot see the bottom of the sea, he will be aware of the obstacles below the surface.
QSM has a broad industry database at its disposal with more than 12,000 completed software
projects developed in Europe, Japan and the United States. Altogether, these thousands of
projects are of great value in determining the feasibility of new software development projects
and complex package implementations.
This broad database can be compared with a nautical chart’s legend. Using the database,
QSM can map out the obstacles in a project in much the same way sailors are able to
determine a navigable course. This allows us to determine whether the project is feasible
within the appointed budget and the desired time.
In other words, it warns you when your project is in danger of wrecking and enables you
to change course in time. We would be glad to pilot your ship.
3

“To reach a port, we must sail - sail, not tie at anchor - sail, not drift.”
[Franklin D. Roosevelt]
4

CONTENTS
Introduction 3
About QSM 6
About Mendix 8
Approach 9
Summary 10
Explanation of the infographics 12
Project #1: Global Securities Service Provider 14
Project #2: Leading National Pension Provider 16
Project #3: Global Beauty Care Company 18
Project #4: European Postal Services Company 20
Project #5: Global Logistics Service Provider 22
Project #6: Public Sector Agency 24
Project #7: Global Information Services Company 26
QSM Project Service Portfolio 28
5

ABOUT QSM
QSM is an independent authority in the field of quantitative software management. We offer
insight into the feasibility, predictability and transparency of software projects and complex
package implementations. At this, Function Points play the central part.
Function Point Analysis
Making projects and implementations predictable and transparent is only possible when we
are able to determine the scope of the project explicitly. For this, QSM uses an internationally
recognized standard method: the Function Point Analysis (FPA), a method to determine the
functional size of an information system.
The use of the Function Point (FP) offers us an objective unit of measurement that can convey
the size of a software system. Meanwhile, it can also be used to compare software systems.
Extensive and broad database
QSM starts from absolute facts. Over the past 30 years we have set up a validated database
consisting of more than 12,000 completed software projects from Europe, Japan and the
United States. The database is continuously updated.
However much they differ, all software projects posses certain characteristic similarities.
From that, QSM has developed a methodology that enables the comparison of large-scale
software projects with similar projects from the industry. This is the basis from which we can
decide whether the entire project is realistically achievable within the available budget and
development time. We can determine whether you are on course or if you will have to make
adjustments in advance.
6

“Modest doubt is called the beacon of the wise.”
[William Shakespeare]
Active all over the world
Quantitative Software Management (QSM) was founded in 1978 by Lawrence H. Putnam,
a distinguished expert in the software measurement industry. Since its inception, QSM’s
mission has been to develop effective solutions for software estimating, project control,
productivity improvement analysis and risk mitigation.
Over the years QSM has established itself as the leading total solution provider of choice
for software developers in high-performance mission-critical environments.
QSM serves the American market from its headquarters in McLean, Virginia while QSM
Europe serves the EMEA market from its office in Breukelen, Netherlands.
7

ABOUT Mendix
Mendix is a software company that provides an integrated app platform as a service offering
to design, develop, deploy, run and manage enterprise applications. The Mendix App
Platform is used by organizations across the globe to build web- and mobile applications to
engage with customers, empower employees and improve business operations.
One platform to bring IT and business together
Mendix strives to accelerate the delivery of new applications by bringing IT and the business
together on one platform, enabling rapid, iterative and collaborative development.
Development in Mendix takes place through a visual, model-driven development approach
that allows both developers and non-technical domain experts to build apps without low-level
coding. Social collaboration capabilities in the platform keep stakeholders aligned throughout
the project.
Typical Mendix use cases
The Mendix App Platform is used for building a variety of applications, including transactional,
data-driven apps with complex business logic and integration with multiple systems.
Examples include customer and partner portals, legacy migrations, vertical apps, workflow
apps, HR on-boarding, ecommerce, contract management, CRM and supply chain apps.
8

Accountability
Mendix invited QSM to perform an independent benchmark study on the Mendix App Platform.
Mendix customers and partners cite the ability to deliver specific custom business applications
in a significantly shorter timeframe, at lower costs and with smaller teams compared to traditional
application development approaches. This study was executed to validate those claims.
Approach
In order to quantify the productivity of the Mendix App Platform, we compared the results
of the build and test phase (which is finished when the application is ready for use) for Mendix
projects with those from industry benchmarks. In other words we are comparing Mendix
projects against software projects from the broad QSM industry database. We first validated
the data we received from the Mendix projects for consistency before including the projects
in the research.
For each project, we determined the scope using a Function Point Analysis (FPA). Subsequently,
we used the FPs to calculate a Productivity Index. With that, we applied a set of four
parameters: project duration, number of hours (investment), team composition and quality.
This approach resulted in a more general conclusion for the productivity of the Mendix App
Platform. In order to obtain an industry perspective, we compared the Mendix results with
the business industry average (namely that of application development).
Project characteristics
We analyzed seven projects built using the Mendix App Platform and the Mendix visual,
model-driven development approach. The size of these projects ranged between 242 and
533 Function Points, while peak staffing ranged between 0.6 and 3.6 FTE. All projects were
delivered within a four-month period (with a duration between 1.6 and 3.8 months).
To conclude
The outcome and conclusions that we are presenting in this report are based on the data from
seven completed and representative Mendix projects from North America and Europe. These
projects were all executed using the Mendix App Platform. We directly performed the research
and verified the results.
Approach
9

We analyzed the data of the application development projects
that were selected by Mendix as a representative set of applications
built on the Mendix App Platform. We determined the size of
the project in Function Points. We compared this data with the
benchmark derived from the QSM industry benchmark database
for business application projects.
Key Findings
1. The productivity of the projects that we analyzed for applications
built on the Mendix App Platform was above market average.
In most of the projects, the Mendix platform outperformed
projects built in a traditional environment significantly. The
average productivity of Mendix projects, expressed in Function
Points per development hour, is 6 times higher than the industry
benchmark.
2. Mendix projects realize a productivity advantage through a
combination of the following factors compared with the average
from our benchmark database:
a. Project teams are 60% smaller
b. Project duration is 50% shorter
c. Development effort is reduced by 70%
Key Takeaways
These results from the analyzed projects validate the Mendix
proposition to deliver higher productivity than traditional
programming methods. They show that Mendix projects require
only 1/6 of the development hours compared to the industry
average for business application projects.
Summary
10

This graph shows that Mendix delivers on average 6x lower cost per unit for
the projects assessed in this benchmark.
5,000
4,000
3,000
2,000
1,000
0
200 250 300 350 400 450 500 550
Hours to deliver a project of a certain size
Size in Function Points Size of the involved projects
This graph shows that Mendix delivers on average 50% faster for the
projects assessed in this benchmark.
250
200
150
100
50
0
200 250 300 350 400 450 500 550
QSM Benchmark Mendix Trendline
11
Number of Function Points produced during one month

Size X Function Points (FP)
Duration X.X Months
Effort X PHR
Peak Staff X.X People (FTE)
MTTD X.X Days
PI X.X
PI=X.X Size=X FP
Project Summary
Size: Functional scope of the project measured in Function Points.
Duration: Difference between start and end date of the project. Expressed
in Months.
Effort: Number of hours spent on the project. Expressed in Person
Hours (PHR).
Peak staff: The highest number of people who worked on the project,
measured during one month. Expressed in FTE.
MTTD: Mean Time to Defect is the average time between the discovery
of new software defects. Expressed in Days.
PI: Productivity Index. “Productivity” embraces many important
factors in software development, including:
· Management influence
· Software development methods
· Software development tools
· Skills and experience of development team members
· Availability of development computer(s)
· Application complexity
· System size
The PI is a macro measure of the total development environment. Values
from 1 to 40 are adequate to describe the full range of projects. Low values
generally are associated with poor working environments, poor tools and high
product complexity. High values are associated with good environments, tools
and management and well-understood, straightforward projects.
Explanation of the infographics
12

FP
100 200 500 1,000
100 200 500 1,000
100 200 500 1,000
Effort (PHR) vs FP
This graph displays the scope in Function Points (FP) on the
x-axis and the invested effort in person-hours on the y-axis.
A logarithmic scale is used. The blue dots show the Mendix
projects. The orange line is the business trend line that is based
on the 12,000 projects in the QSM database. The orange shaded
area indicates the 1-sigma line. Within this bandwidth 68.26%
of the projects from the QSM database have been realized.
This graph displays the scope in Function Points (FP) on the x-axis
and the project duration in months on the y-axis. A logarithmic
scale is used.
This graph displays the scope in Function Points (FP) the x-axis
and the team size in Fulltime Equivalent (FTE). A logarithmic
scale is used.
Mendix project 1 Sigma bandQSM Business Trendline
FP
100
10
1
Duration(Months)
Duration (Months) vs FP
FP
100
10
1
0.1
AverageStaff(FTE)
Average Staff (FTE) vs FP
Effort(PHR)
100,000
10,000
1,000
100
13

Global Securities Service Provider
• Build cost with Mendix is five times lower.
• Overall QSM benchmark team size is two and a half times larger.
Comparison with the market average
Explanation of the graph
The first graph shows the productivity
in terms of the average number of
hours spent on the project.
The second graph shows the average
team size.
QSM - Industry Database
5.0
4.0
3.0
2.0
1.0
Average Staff (FTE)
14
12
10
8
6
4
2
Average Hours per FP
Project Description
The customer’s core business is providing clearing services in
the financial market. This project was to build an application for
calculating penalties and bonuses for market parties following a
complex set of trading rules. The
application automated manual and error-prone processes and
integrates with existing core systems and middleware solutions.
14

100 200 500 1,000
100 200 500 1,000
100 200 500 1,000
Size 303 Function Points
Duration 2.3 Months
Effort 527 Person Hours
Peak Staff 1.5 People (FTE)
MTTD 2.8 Days
PI 21.0
Global Securities Service Provider | Details
Overview core metrics
Mendix project
QSM Business
1 Sigma band
PI=21.0 Size=303 FP
Project Summary
FP
Effort (PHR) vs FP
FP
100
10
1
0.1
AverageStaff(FTE)
FP
100
10
1
Duration(Months)
Effort(PHR)
100,000
10,000
1,000
100
15

Leading National Pension Provider
• Build cost with Mendix is four times lower.
• Overall QSM benchmark team size is three times larger.
The first graph shows the productivity in
terms of the average number of hours spent
on the project.
The second graph shows the average team
size.
Project Description
The customer is a leading national pension provider. The project involved
the development of a web portal through which more than one million
participants of sixty different pension funds have self-service access to view
the status of their pension plan and update personal data. The portal is a
single application that delivers a tailored user experience based on the style
guide of the specific pension fund in which the individual participates.
14
12
10
8
6
4
2
5.0
4.0
3.0
2.0
1.0
Average Staff (FTE)
16

100 200 500 1,000
100 200 500 1,000
100 200 500 1,000
Duration 3.4 Months
Effort 1,141 Person Hours
MTTD 2.4 Days
PI 19.7
Leading National Pension Provider | Details
PI=19.7 Size=478 FP
Project Summary
FP
Effort (PHR) vs FP
FP
100
10
1
0.1
AverageStaff(FTE)
FP
100
10
1
Duration(Months)
Mendix project
QSM Business
1 Sigma band
Effort(PHR)
100,000
10,000
1,000
100
17

Global Beauty Care Company
terms of the average number of hours spent
on the project.
The second graph shows the average team
size.
Project Description
The customer is a leader in premium, world-renowned beauty-care brands
that are sold through retail and salon channels. The objective of this project
was to streamline the processing of orders from the salon channel by providing
sales representatives with a mobile application that integrates with the ERP
back-office application. The customer has standardized the back-office
on SAP, while the sales representatives work with an iPad application. The
project concerned the middleware solution between the native iPad app
and SAP. The system manages the security aspects of the iPad application as
well as the exchange of information between SAP and the iPad application,
including product information, promotions and orders. The application is
rolled out in twelve countries.
14
12
10
8
6
4
2
5.0
4.0
3.0
2.0
1.0
Average Staff (FTE)
18

100 200 500 1,000
100 200 500 1,000
100 200 500 1,000
Duration 3.3 Months
MTTD 3.4 Days
PI 19.5
Global Beauty Care Company | Details
PI=19.5 Size=368 FP
Project Summary
FP
Effort (PHR) vs FP
FP
100
10
1
0.1
AverageStaff(FTE)
FP
100
10
1
Duration(Months)
Mendix project
QSM Business
1 Sigma band
Effort(PHR)
100,000
10,000
1,000
100
19

European Postal Services Company
• Overall QSM benchmark team size is 35% larger.
The first graph shows the productivity
in terms of the average number of
hours spent on the project.
team size.
Project Description
The customer’s core activity is providing postal services. There was a need
in the HR domain for better support of job mobility and career planning
processes. This project was to build such solution with built-in flexibility for
making dynamic changes as per evolving rules and regulations. The application
is integrated with SAP HR, being the system of record for employee data,
and SAP BW for business intelligence and reporting.
14
12
10
8
6
4
2
5.0
4.0
3.0
2.0
1.0
Average Staff (FTE)
20

100 200 500 1,000
100 200 500 1,000
100 200 500 1,000FP
Effort (PHR) vs FP
FP
100
10
1
0.1
AverageStaff(FTE)
FP
100
10
1
Duration(Months)
Effort(PHR)
100,000
10,000
1,000
100
Duration 2.2 Months
Effort 1,203 Person Hours
MTTD 1.5 Days
PI 22.5
European Postal Services Company | Details
PI=22.5 Size=533 FP
Project Summary
Mendix project
QSM Business
1 Sigma band
21

Project Description
The customer is a global logistics service provider. The scope of this project
was to build a single workflow solution for custom pricing to be used across
multiple countries to replace a disparate set of locally developed solutions.
The customer processes more than three million requests per year.
The application supports various pricing models for specific offerings and
handles the workflows for approval of pricing requests. The application is
multilingual and is integrated with a document management system.
Global Logistics Service Provider
• Build cost with Mendix is seven times lower.
terms of the average number of hours
spent on the project.
team size.
14
12
10
8
6
4
2
5.0
4.0
3.0
2.0
1.0
Average Staff (FTE)
22

100 200 500 1,000
100 200 500 1,000
100 200 500 1,000FP
Effort (PHR) vs FP
FP
100
10
1
0.1
AverageStaff(FTE)
FP
100
10
1
Duration(Months)
Effort(PHR)
100,000
10,000
1,000
100
Duration 2.4 Months
MTTD 3.4 Days
PI 21.0
Global Logistics Service Provider | Details
PI=21.0 Size=325 FP
Project Summary
Mendix project
QSM Business
1 Sigma band
23

Public Sector Agency
team size.
• Build cost with Mendix is seven times lower.
• Overall QSM benchmark team size is four times larger.
Project Description
The customer is a Public Sector Agency. The IT organization was faced with
the challenge to optimize use of software licenses across the organization.
The project was to build a central administration of software licenses and
a marketplace to trade these licenses within the organization, respecting
conditions and constraints associated with the respective license contracts.
14
12
10
8
6
4
2
5.0
4.0
3.0
2.0
1.0
Average Staff (FTE)
24

100 200 500 1,000
100 200 500 1,000
100 200 500 1,000FP
Effort (PHR) vs FP
FP
100
10
1
0.1
AverageStaff(FTE)
FP
100
10
1
Duration(Months)
Effort(PHR)
100,000
10,000
1,000
100
Duration 3.7 Months
MTTD 5.5 Days
PI 18.0
Public Sector Agency | Details
PI=18.0 Size=242 FP
Project Summary
Mendix project
QSM Business
1 Sigma band
25

Global Information Services Company
team size.
• Build cost with Mendix is thirty times lower.
• Overall QSM benchmark team size is eight times larger.
Project Description
The company’s core business is providing information services and consulting
based on market research in the information technology domain. The aim of this
project was to automate the workflows and information gathering from technology
companies to support the research. This process had been manual and based on
exchange of spreadsheets. The application is a secure portal for companies
participating in the research to submit their data and inputs.
14
12
10
8
6
4
2
5.0
4.0
3.0
2.0
1.0
Average Staff (FTE)
26

100 200 500 1,000
100 200 500 1,000
100 200 500 1,000FP
Effort (PHR) vs FP
FP
100
10
1
0.1
AverageStaff(FTE)
FP
100
10
1
Duration(Months)
Effort(PHR)
100,000
10,000
1,000
100
Duration 1.6 Months
MTTD 3.6 Days
PI 26.1
Global Information Services Company | Details
PI=26.1 Size=421 FP
Project Summary
Mendix project
QSM Business
1 Sigma band
27

QSM Project Service Portfolio
QSM offers various services in the startup phase, during the project implementation phase
and when the project is completed. In all these phases, the information provided by QSM is
100% fact-based. We perform these services independently or implement them in the client
organization. To ensure objectivity, QSM avoids active participation in projects.
Project Sizing
Sizing is arguably the most challenging part of any software estimate. Without a notion of
functional size it’s quite difficult to estimate realistic schedules on the set development time.
QSM has a specialized unit which determines the size of projects unambiguously. This team
plays an important role in offering project transparency and project predictability.
Project Scenarios
On the basis of project size and benchmark data, QSM identifies the scenarios that are
located within the acceptable ranges. The Project Scenario Services provide a set of feasible
scenarios for the successful delivery of the relevant software project.
Project Snapshots
QSM standard services provide factual insight into the status of a project. It makes clear
whether the project is progressing as expected. This project snapshot also provides an image
of the expected results, if and when the project continues unchanged.

28

Project Benchmark
The QSM database is essential in all of our consulting service engagements. It represents
the largest and most complete set of validated and completed software project data in the
world.
Based on this data we can provide factual insight into delivery of a project, in relation to its
own or industry standard productivity trends. These metrics are highly useful for defining
scenarios for future projects. Project Benchmark services also determine the extent to which
the project delivery complies with the initial agreement or contract.
29

QSM Mendix Benchmark Report

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to QSM Mendix Benchmark Report (20)

QSM Mendix Benchmark Report