Map Server comparison, OGC WMS - Random Extent

Map server comparison
OGC WMS – Random Extent

Author: Ing. Michal Šeliga, Ph.D., E-mail: michal.seliga@tmapy.cz

Date: 2010-03-25

The objective of this test is to check a server ability to handle WMS requests for random extent
and constant output size without having to perform data transformation between different
SRS.

Test description and setting
Test client is increasing the number of active clients linearly in time, who simultaneously query
the server. Each of an active clients sends its request immediately after that what is previous
request finished and each of the newly generated request has a different random extent.
The proposed test should check the following qualities:
• Dealing with physical data,
• Scaling,
• Transformation of raster data into the desired output format (PNG),
• Ability to respond to the huge application server load.

Data
As test data the aerial photographs of Prague were used, with resolution of 10 cm, which were
mosaic by EIM into one large IMG file, than were built pyramids the IMG file. The final size of
the test data was 290GB, including the pyramids.

Setting
Number of users: 1 – 300
Delay between two users tests: 0 s
Delay to create an additional user: 15 s
Total test time: 5000 s
Output size: 1200x800
Output format: PNG

HW
Server: Windows 2003 64bit, Sun Fire X2200 (2x QuadCore, 8GB RAM, 2x HDD 400GB, 3x
Ethernet 1Gb, 1x Ethernet 100Gb), System and data are stored on different HDD.
Client: Linux Ubuntu 8.10 LTS, PC 1x DualCore, 2GB RAM, 1x Ethernet 1Gb.
Test client machine is connected to the server directly via one STP CAT6 cable. Connection use
two reserved network cards, which are configure to special subnet.

ArcGIS Server
Installation: LIKU

Installation and configuration:
− AGS 9.3.1 .NET,
− IIS,
− Instances: min. 7 - max. 7,
− Only WMS,
− No user restrictions defined.

WMS service average response time
With native technology that uses AGS is behavior of AGS surprisingly harmonized with
relatively small demands on system resources and hardware. Server behavior is characterized
by nearly 100% linear dependence, the calculated coefficient of linear regression dependence
is equal to 0.99 after rounding to two decimal places.

GetMap
AGS + IIS
60000

50000
f(x) = 307.15 x^0.89
40000 R² = 0.99
Response [ms]

AvgTotal
30000 Power Regression for AvgTotal
8x Core
20000 1x Core

10000

0
0 50 100 150 200 250 300
Users

Graph 5: The average response time of WMS service, including downloading a picture

§
Curves labeled as 1x Core and 8x Core illustrate the theoretical course of conduct in case of a
map server load balancing on one and eight cores. The calculation is based on the average
value of "Response time" for a single user.

Number of requests processed, based on the number of users
In this case, we can leave the test results almost uncommented. Only an explanation of errors
that occurred during the test might deserve our attention. These errors are replicable and
clearly not related to the tests carried out. After analysis, recording errors, we concluded that
it is an internal error in the implementation of AGS. This error occurs always at the
requirements, which require “non-standard extent”. Specifically, an extent, which has a ratio
of height and width smaller than 1/3000. This problem probably does not occur in average use
and therefore, we do not give too much importance to it.

GetMap: request count
AGS + IIS
100

80

60
Requests

OK
IOError
40 ServiceError

20

0
0 50 100 150 200 250 300
Users

Graph 6: Number of requests processed, depending on the number of users

GetMap: wrong response count detail
AGS + IIS
5

4

3
Requests

OK
2 IOError
ServiceError
1

0
0 50 100 150 200 250 300
Users

Graph 7: Number of requests processed, depending on the number of users (detail)

GetMap: requests per second
AGS + IIS
5

4
Requests / s

3
OK/s
2 IOError/s
ServiceError/s
1

0
0 50 100 150 200 250 300
Users

Graph 8: Number of requests processed, depending on the number of users per second

GetMap: wrong responses per second
AGS + IIS
0.5

0.4

0.3
Requests / s

OK/s
IOError/s
0.2 ServiceError/s

0.1

0
0 50 100 150 200 250 300
Users

(detail)

As already mentioned AGS is responsible for some of the ServiceErrors, the amount of such
responses is summarized in the following chart. Dependence of number of errors is linear in
the number of random requests.

AGS
Users Median Sum
Requests/s Requests Response time Requests IOError ServiceError
1 1.03 19.00 836.00 19 0 0
10 3.31 66.50 1262.00 568 0 2
50 2.86 74.00 5207.00 3561 0 5
100 2.41 77.00 9941.50 7507 0 8
200 1.75 79.00 19102.00 15838 0 21
300 1.41 82.00 27427.00 24531 0 29
Table 1: Tests result conclusion

ERDAS Apollo Server
Installation: MISE

Installation and configuration:
− EAS 9.3.2,
− JAVA 1.5.0_18-b02,
− Tomcat6 (NIO adapter),
− No user restrictions defined.

Behavior EAS showed considerable fluctuations, which were probably result of the "Garbage
collector" work in the JVM and, then by the greater memory load was running the application
server itself. EAS was tested in combination with the application server Tomcat6, but it is also
possible to use a "big application server”. Deployment, combined with a commercial application
server such as Oracle AS, Weblogic or Websphere results could still bring a little better results.

GetMap
EAS + Tomcat6
100000

80000
Response [ms]

60000
AvgTotal
8x Core
20000 f(x) = 82.79 x^1.05 1x Core

R² = 0.91
0
0 50 100 150 200 250 300
Users


map server load is balancing on one and eight cores. The calculation is based on the average
value of "Response time" for one single user.

Number of requests processed, depending on the number of users
EAS + Tomcat6
280
240
200
160
Requests

OK
120 IOError
ServiceError
80
40
0
0 50 100 150 200 250 300
Users


EAS + Tomcat6
12
10
8
Requests / s

6 OK/s
IOError/s
4 ServiceError/s

2
0
0 50 100 150 200 250 300
Users


The following table show that all EAS server responses were successful.

EAS
Users Median Sum
1 1.87 33.00 460.00 33 0 0
10 6.81 142.00 605.00 1263 0 0
50 7.41 186.00 2021.00 8566 0 0
100 5.63 178.00 4195.50 16489 0 0
200 4.54 176.00 9478.50 32313 0 0
300 3.19 167.00 14850.00 44816 0 0
Table 2: Tests result conclusion

ERDAS Image Manager
Installation: MISE

Installation and configuration
− EAIM 9.3.2,
− JAVA 1.5.0_18-b02,
− Jboss 4.2.2 GA (NIO adapter),
− Authentication and authorization is turned on: If the authentication and authorization
for access to WMS services has been turned off, in a state of load (more than 5 users at
the same time) the server turned off and on from time to time returned.

Note
- The execution of each request requires an access to the database, which first search for
required layer or layers information and then verifies the access user’s rights to work
with the layer and its extent.

EAIM behavior showed considerable fluctuations, which resulted from the "Garbage collector"
work in the JVM and then from the greater memory load running the application server itself.

GetMap
EAIM + JBoss
100000

80000
Response [ms]

60000
AvgTotal
8x Core
20000 f(x) = 96.15 x^1.03 1x Core
R² = 0.85
0
0 50 100 150 200 250 300
Users



EAIM + JBoss
280
240
200
160
Requests

OK
120
80
40
0
0 50 100 150 200 250 300
Users


EAIM + JBoss
12
10
8
Requests / s

6 OK/s
IOError/s
4 ServiceError/s

2
0
0 50 100 150 200 250 300
Users


The following table show that all EAIM server responses were successful.

EAIM
Users Median Sum
1 1.55 27.00 557.00 27 0 0
10 4.31 99.00 825.50 909 0 0
50 6.08 168.50 2322.50 7725 0 0
100 6.76 190.00 4291.00 17297 0 0
200 4.19 171.00 8209.00 31538 0 0
300 2.77 150.00 16735.00 43430 0 0
Table 3: Tests results conclusion

ERDAS APOLLO Professional
Installation: MISE

− EAP built on 2009.11.12
− JAVA 1.5.0_20-b02
− JBoss AS 4.2.2.GA
− Configuration and its optimalization are completely identical with the setting made for
the installation EAIM 2009 (EAIM 9.3.2)

EAP behavior showed considerable fluctuations, which resulted from the "Garbage collector"
The most significant fluctuations always occurred in connection with the increased number of
requests in the disk I/O queue.

GetMap
EAP + JBoss
100000

80000
Response [ms]

60000
AvgTotal
f(x) = 79.11 x^1.1 8x Core
1x Core
20000 R² = 0.91

0
0 50 100 150 200 250 300
Users



EAP + JBoss
280
240
200
160
Requests

OK
120 IOError
ServiceError
80
40
0
0 50 100 150 200 250 300
Users


EAP + JBoss
12

10

8
Requests / s

OK/s
6
IOError/s
ServiceError/s
4

2

0
0 50 100 150 200 250 300
Users


EAP
Users Median Sum
1 1.28 23.00 676.00 23 0 0
10 6.51 117.50 736.50 1082 0 0
50 6.78 155.50 2371.00 7443 0 0
100 5.47 150.50 5420.00 14574 0 0
200 4.00 143.00 10514.50 26609 0 0
300 2.72 132.00 18218.00 36516 0 0

ERDAS APOLLO Professional (optimal)
Instalace: MISE

− EAP built on 2009.11.12
− JAVA 1.5.0_20-b02
− JBoss AS 4.2.2.GA
− Configuration is identical to the previous settings EAP and EAIM with the only
difference: on the basis of empirical tests, I came to the adjusting of the value for a
minimum and maximum number of RDS value instances: minimum number of instance
is 6 and maximum number of instance is 6.

EAP behavior showed considerable fluctuations, which resulted from the "Garbage collector"
The most significant fluctuations always occurred in connection with the increased number of
requests in the disk I/O queue.

GetMap
EAP + JBoss
100000

80000
Response [ms]

60000
AvgTotal
f(x) = 79.11 x^1.1 8x Core
1x Core
20000 R² = 0.91

0
0 50 100 150 200 250 300
Users


GetMap
EAP + JBoss
100000

80000
Response [ms]

60000
AvgTotal
8x Core
f(x) = 96.68 x^1.04 1x Core
20000
R² = 0.91
0
0 50 100 150 200 250 300
Users



EAP + JBoss
280
240
200
160
Requests

OK
120 IOError
ServiceError
80
40
0
0 50 100 150 200 250 300
Users


EAP + JBoss
12

10

8
Requests / s

OK/s
6
IOError/s
ServiceError/s
4

2

0
0 50 100 150 200 250 300
Users


EAP (optimal)
Users Median Sum
1 1.36 24.00 632.00 24 0 0
10 5.59 105.50 730.00 1011 0 0
50 6.43 148.00 2402.50 7156 0 0
100 5.43 148.00 5168.50 14449 0 0
200 4.58 148.00 10297.00 28721 0 0
300 3.78 146.00 16371.00 40041 0 0

Conclusion
The performed test is a compromise among the other proposed tests. This test is the best
approaching the real state from all of them. The objective of this test was to examine the
methodology of testing and technique of treatment results processing. During testing, we have
spent a lot of time tweaking the optimal settings of the test servers. AGS configurations
change its performance of about 5%, but in the case of ERDAS APOLLO it was about 200%.

ERDAS has better quality output at smaller output size
Resolution is the same and color depth for both was 24bit per pixel. Real number of colors:
[ERDAS, 49,462], [ESRI, 71,897].
File size of ESRI output is larger by about 60% (probably because of there is a greater number
of colors). Even if the ESRI output is larger, but according to technical parameters better it
looks "eyemetricaly" worse. ERDAS probably has better histogram functions.
ERDAS GIF output is also better (significantly better). ERDAS GIF compare to ESRI is
smaller and at the same color depth (8 bits as is custom for GIF) it gives a nice picture. It
might be caused by the fact that ESRI perhaps insufficiently recasts the color depth.

GIF output:

Picture 1: EAIM (GIF)
§

Picture 2: AGS (GIF)

PNG output:

Picture 3: EAIM (PNG)

Picture 4: AGS (PNG)

AGS has a balanced predictable behavior depending on CPUs number
AGS has a very balanced behavior, which is almost linear. This allows sufficient accuracy
in predicting the speed of responses and system resources in advance.
AGS is primarily written by ArcObjects, which are native COM objects and these are wrapped
with .NET interface.
The main reason for AGS balanced behavior with comparison to EA* is that AGS uses the all
capacity on the CPU levels soon and then no longer depends on the permeability of CPU (disk
I/O queue is almost empty during the test, the memory is used by the two thirds only). EA*
does the opposite, the CPU uses the capacity at 60% on average and processes are waiting for
processing requests from the disk I/O queue, application server (AS) process is limited by
JAVA 32bit, which does not allocate enough memory for large amount of clients => EA* is
more prone to speed of I/O operations and memory size.
Another reason for the uneven EA* behavior is that JAVA works with larger amount of memory
and sometimes the operating system had problems with the allocation of large blocks of
memory (384 megabytes).
Unbalanced performance of ERDAS APOLLO is also due to JAVA technology in which is written
and operated. JAVA unlike native code has a different approach to memory management, the
JVM free memory via Garbage Collector (GC), which from time to time, or if the free memory
is low, suspends the running program and finds all instances of objects having no further
reference and releases them.

AGS requires less system resources
Due to AGS has been installed in .NET version, than application server MS IIS was used, which

unlike the JAVA AS is written for native environment. This has the effect of minimum reduction
of the required resources, especially memory of the server.

AGS has an internal restriction causing a ServiceError
Implementation of AGS has probably an internal restriction, which causes that some queries
results are ServiceError. ERDAS APOLLO did not come back a single wrong answer during the
test.
These errors can be replicated outside the test as well - after consultation with VLAMA, LIKU
and TONO we agreed that AGS response ServiceError occurred in extreme cases where
requested extent has big difference between height and width. The ServiceError is caused by
unknown internal error "Unknown exception - Internal Server Error". This problem would
not theoretically occur in common operations => It does not seem to be important.

ERDAS shows less dependence on the number of users
The following table shows coefficients of dependency based on response time (Response time)
and the number of users, according to limit 300 users. Input values for calculations were the
300 users and a maximum value of the "Response time", taken from the regression curves
based on "Response time" and "the number of users" (bigger value indicates stronger
dependency).

Response time [s] Coefficient
AGS 50 0.17
EAIM 38 0.13
EAS 37 0.12
EAP 39 0.13

Table 6: Coefficient of dependency between response time and number of users

ERDAS is significantly faster
The following table shows the percentage performance comparison ERDAS APOLLO Servers
and ArcGIS Server for the given number of users (AGS means 100%). These results must be
treated deliberately, because the input values for median are taken from continuous
incremental test with a defined constant step.
It must be taken into account, that the results are related to the above described
dataset and in case of individual datasets (data formats) the results may vary.

To obtain more accurate results, it would be essential to prolong the period between addition
of requested user to get server time to stabilize for the current number of users. The time
between additions of users should be significantly longer than the response time "Response
time".

EAIM EAS
Users
Requests/s Requests Response time Requests/s Requests Response time
1 149.83 142.11 66.63 181.08 173.68 55.02
10 158.16 148.87 65.41 224.31 213.53 47.94
50 241.72 227.70 44.60 263.83 251.35 38.81
100 261.87 246.75 43.16 242.58 231.17 42.20
200 228.23 216.46 42.97 232.27 222.78 49.62
300 192.87 182.93 61.02 212.33 203.66 54.14
Table 7: ERDAS APOLLO 2009 percentage compared with ArcGIS Server

As shown above (Table 6) EAS is slightly more powerful than EAIM. The performance difference
can be explained by the fact that EAIM compared to EAS also tests included evaluation of the
ERDAS Catalog (user rights depend on layer and extent). The difference is also influenced by
the application server, for the EAS was chosen Tomcat6 servlet container and for EAIM it was
JBoss 4.2 with EJB3 support.

EAP – optimal [%] EAP [%]
Users
Requests/s Requests Response time Requests/s Requests Response time
1 132.12 126.32 75.60 123.65 121.05 80.86
10 168.62 158.65 57.84 196.40 176.69 58.36
50 225.12 200.00 46.14 237.29 210.14 45.53
100 225.36 192.21 51.99 227.21 195.45 54.52
200 261.53 187.34 53.91 228.18 181.01 55.04
300 268.72 178.05 59.69 192.97 160.98 66.42
Table 8: ERDAS APOLLO 2010 percentage compared with ArcGIS Server

Table 9:

Table 8 compares the results of the tests of servers ArcGIS and ERDAS APOLLO, where AGS is
taken as a reference and it represents 100% as well as in the previous chart. It says that as
for as a number of requests bigger is better, and regarding the response time smaller is better
on the other hand. The results show that EAP is not significantly different from those of his
predecessor EAIM and retains a significant performance advantage over AGS.

It is worth mentioning the percentage difference between the values of “Requests/s” and
“Requests”. Those differences mirror the load and direction of an application server. As the
number of requests to a server rises, the time required for an initialization of TCP connection
between the test client and application server is increasing as well. This as a direct result
means that the client in a given period of time does not manage to send so many requests to a
sever, as in case of small load (small number of clients asking at the same time), therefore
such a result increases with the number of users simultaneously.

GetMap: response time
Comparation
80000
70000
60000
50000
Response [ms]

EAP + JBoss
40000 EAIM + Jboss
AGS + IIS
30000 EAS + Tomcat6

20000
10000
0
0 50 100 150 200 250 300
Users

Graph 23: Comparison of speed responses depending on the number of users

GetMap: response time
Comparation
20000
18000
16000
14000
12000
Response [ms]

EAP + JBoss
10000 EAIM + Jboss
8000 AGS + IIS
EAS + Tomcat6
6000
4000
2000
0
0 10 20 30 40 50 60 70 80 90 100
Users

Graph 24: Comparison of speed responses depending on the number of users (detail)

Comparation
14
12
10
8
Requests / s

EAP + JBoss
6 EAIM + Jboss
AGS + IIS
4 EAS + Tomcat6

2
0
-2
0 50 100 150 200 250 300
Users

Graph 25: Comparing the average number of correct answers per second, depending on the
number of users

Comparation
14

12

10

8
Requests / s

EAP + JBoss
EAIM + Jboss
6 AGS + IIS
EAS + Tomcat6
4

2

0
0 10 20 30 40 50 60 70 80 90 100
Users

Graph 26: Comparing the average number of correct answers per second, depending on the
number of users (detail)

ERDAS 2010 versus ERDAS 2009
Based on the results published earlier there was a comparison of two product lines EAIM 2009
and EAP 2010. The results of that comparison are summarized in the following table, where as
the comparative criteria (100%) EAIM tests results were used. For values of “Request/s” and
“Requests” exceeding 100% it means that ERDAS 2010 surpassed ERDAS 2009. The remaining
columns “Response time”, IOError and ServiceError the opposite is true.
The results show that the performance of EAP exceeds the EAIM in case that we compare them
according the character “Requests/s” and especially if there is a large number of users (more
than 200). When comparing EAP and EAIM by the total number of processed requests for the
number of users, the power of EAP is approximately 10% worse than in case of EAIM 2009.

It is interesting to see the results minding the fact that the quicker are the responses per
second in case of EAP, the number of processed requests declined. Such a behavior is caused
by testing EAP 2010, where more intensive latency was identified during establishing and
terminating connections with this application server. It resulted in sending a smaller possible
maximum number of client requests to a server and resulting in a burden on the server, and
therefore faster processing of requests received. This increased latency might be caused by the
changes in behavior at side of:
− OS Windows 2003 as a result of installing security patches, servlets and their service in
AS
− AS JBoss and configuration changes beyond my adjustments compare to the original
versions
EAP 2010 / EAIM 2009 [%]
Users Median Sum
1 88.18 88.89 113.46 88.89 100.00 100.00
10 129.58 106.57 88.43 111.22 100.00 100.00
50 105.86 87.83 103.44 92.63 100.00 100.00
100 80.29 77.89 120.45 83.53 100.00 100.00
200 109.33 86.55 125.44 91.07 100.00 100.00
300 136.59 97.33 97.82 92.20 100.00 100.00
Table 10: ERDAS APOLLO 2010 percentage compared with ERDAS APOLLO 2009

The tested version of EAP compared to EAIM balanced better between the use of disk and CPU
time, which is positively reflected in the concordant behavior of the server. Further
improvement was seen in the memory performance EAP, where Memory Footprint applications
fell by about 15% in comparison with EAIM.

Perspective in the future steps
Concerning ERDAS APOLLO it is expected to be converted to Java6 in the future. It will have a
positive impact on improving performance in the areas of the memory management,
supporting of multiprocessor machines, and optimizing of byte code processing. Transition to
Java6 64bit will also break the limit for maximum amount of allocated memory for a single
JAVA process, which will in case you have enough memory, eliminate the effects of JAVA GC
and it will further increase the performance and it will harmonized load curves in general.
The power and concordant behavior of the server could be influenced positively by
implementation of rules to control the access to disk I/O operations that would prevent
overloading the disk I/O queue.
With the transition to Java6 64bit we can expect performance increases by about
10% -20% and it will harmonize performance excessively compared to the existing
set-up.

Map Server comparison, OGC WMS - Random Extent

More Related Content

Similar to Map Server comparison, OGC WMS - Random Extent (20)

More from GeoCommunity (15)

Recently uploaded (20)

Map Server comparison, OGC WMS - Random Extent