Axis2 client memory leak

Outline
• Issue & diagnosis
• Postmortem from technique point
– Axis2 1.4/1.4.1/1.5.6 client
– Implement & design issue

Issue

CServer exhaust 8G memory -> memory leak

diagnosis
jmap –histo pid

num #instances #bytes class name
----------------------------------------------
1: 1001744 246906600 [C //char array
2: 2855952 137085696 edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$Segment
3: 282338 115692192 [I // int array
4: 677362 102508648 [Ljava.util.HashMap$Entry;
5: 192117 99164392 [B // byte array
6: 2856122 91732120 [Ledu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$HashEntry;
7: 2855952 91390464 edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync
8: 1337498 53499920 java.lang.String
9: 433634 43130120 [Ljava.lang.Object;
10: 597908 38266112 java.util.HashMap

From the histogram, the memory leak is caused by Axis2.
1) The main change in this release is using web service client (axis2 1.4.1)
2) char[] / edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap
3) Google ‘axis2 1.4.1 concurrentHashMap memory leak‘

diagnosis
jmap –dump:file=cdump pid

• We got a 2.39G memory heap dump
– Very fast, less than 30 seconds (because 8G
memory in production)
• Problem : too big to open in dev box
– Need enough physical memory
– (May) need 64 bit OS
– Try Jhat/Jprofile/YJP/Jmat eclipse plugin
(windows/2G)
Open by Jmat standalone application

diagnosis

AxisConfiguration retain 540.9m heap

diagnosis

Hashtable allEndPoints holds more than 2k axis2 endpoint instances

diagnosis

All of the endpoint instances are about FavoriteService
1) Confirm in the code, only FavoriteService related did not call cleanup

Axis2 1.4 memory leak
• Reproduce

for( int i=0;i<count; i++) {
VersionStub stub = new VersionStub(configContext,null);
GetVersion request = new GetVersion();
stub.getVersion(request);
}

• Informal Model (OO design)

AxisConfiguration

Map allServices = new Hashtable();
Map allEndpoints = new Hashtable();

• Cause : Stub Initiation, add into AxisConfiguration

//Init, Add
allServices.put(serviceName, axisService);
..
allEndpoints.put(serviceName + "." + endpointName, axisService);

• Cause : Stub Finalize
//Stub
protected void finalize() throws Throwable {
super.finalize();
cleanup();
}

public void cleanup() throws AxisFault {
_service.getAxisConfiguration().removeService(_service.getName());
}

//AsixConiguration
public synchronized void removeService(String name) throws AxisFault {
AxisService service = (AxisService) allServices.remove(name);
if (service != null) {
AxisServiceGroup serviceGroup = service.getAxisServiceGroup();
serviceGroup.removeService(name);
log.debug(Messages.getMessage("serviceremoved", name));
}
}

• Cause : Client Finalize
//ServiceClient
super.finalize();
cleanup();
}

…
axisConfiguration.removeServiceGroup(serviceGroupName);
…
}

//AsixConiguration
public AxisServiceGroup removeServiceGroup(String serviceGroupName) throws AxisFault {
…
Iterator services = axisServiceGroup.getServices();
while (services.hasNext()) {
AxisService axisService = (AxisService) services.next();
allServices.remove(axisService.getName());
…
}
…

• Cause
//Init, Add
..
allEndpoints.put(serviceName + "." + endpointName, axisService);

//Cleanup(Finalize), Remove

• Memory Leak : allEndpoints

Axis2 1.4.1 fix
• Fix the bug AXIS2-3870
//AsixConiguration.removeServiceGroup

//removes the endpoints to this service
String serviceName = axisService.getName();
String key = null;
for (Iterator iter = axisService.getEndpoints().keySet().iterator(); iter.hasNext();){
key = serviceName + "." + (String)iter.next();
this.allEndpoints.remove(key);
}

Axis2 1.4.1 memory leak
• Reproduce

for( int i=0;i<count; i++) {
VersionStub stub = new VersionStub(configContext,null);
GetVersion request = new GetVersion();
stub.getVersion(request);
stub.cleanup();
}

Programmer: 1.4 has memory leak issue, so call cleanup

• Cause : Stub Finalize (no change)
//Stub
super.finalize();
cleanup();
}

_service.getAxisConfiguration().removeService(_service.getName());
}

//AsixConiguration
public synchronized void removeService(String name) throws AxisFault {
AxisService service = (AxisService) allServices.remove(name);
if (service != null) {
AxisServiceGroup serviceGroup = service.getAxisServiceGroup();
serviceGroup.removeService(name);
log.debug(Messages.getMessage("serviceremoved", name));
}
}

• Cause: Client Finalize (no change)
//ServiceClient
super.finalize();
cleanup();
}

…
axisConfiguration.removeServiceGroup(serviceGroupName);
…
}

• Cause: Client Finalize (no change)
//AsixConiguration
public AxisServiceGroup removeServiceGroup(String serviceGroupName) throws AxisFault {
…
Iterator services = axisServiceGroup.getServices();
while (services.hasNext()) {
AxisService axisService = (AxisService) services.next();
allServices.remove(axisService.getName());

…
for (Iterator iter = axisService.getEndpoints().keySet().iterator(); iter.
key = serviceName + "." + (String)iter.next();
this.allEndpoints.remove(key);
}
…

}

Servce are already removed from ServiceGroup in Stub cleanup
…
}
The while loop would never enter

• Cause
– Stub cleanup and ServiceClient cleanup have
dependency
– Can not call in below order

Stub.cleanup
ServiceClient.cleanup

• Two are two memory leak bugs in 1.4, 1.4.1 only
fix 1 bug

Axis2 1.5(.6) fix
• Fix the bug AXIS2-4007 AXIS2-4163
//Stub
super.finalize();
cleanup();
}

// _service.getAxisConfiguration().removeService(_service.getName());
_serviceClient.cleanup();
}

Implement issue
• Forget to cleanup
– Container object: add only, no remove
– Resource object: apply only, no return/close

• Cleanup dependency
– One object cleanup depend on other objects
– Two object cleanup/two method has order
dependency

Design Issue
• AxisConfiguration is a global shared object
– Usually only 1 instance even in client side.

• Purpose for put service/endpoint map in this
global object AxisConfiguration ?

Design Issue
• Message Dispatch in server side
// RequestURIBasedServiceDispatcher
public AxisService findService(MessageContext messageContext) throws AxisFault {

AxisConfiguration registry = configurationContext.getAxisConfiguration();

AxisService axisService = registry.getService(values[0]);

// If the axisService is not null we get the binding that the request came to add
// add it as a property to the messageContext
if (axisService != null) {
Map endpoints = axisService.getEndpoints();
if (endpoints != null) {
if (endpoints.size() == 1) {
messageContext.setProperty(WSDL2Constants.ENDPOINT_LOCAL_NAME,
endpoints.get(axisService.getEndpointName()));
} else {
String endpointName = values[0].substring(values[0].indexOf(".") + 1);
messageContext.setProperty(WSDL2Constants.ENDPOINT_LOCAL_NAME,
endpoints.get(endpointName));
}
}
}

return axisService;
}

Design Issue
• Service is ‘singleton’ in server side
// AxisConfiguration
public AxisService getService(String name) throws AxisFault {
AxisService axisService = (AxisService) allServices.get(name);
if (axisService.isActive()) {
return axisService;
} else {
throw new AxisFault(Messages
.getMessage("serviceinactive", name));
}
} else {
axisService = (AxisService) allEndpoints.get(name);
if (axisService.isActive()) {
return axisService;
} else {
throw new AxisFault(Messages
.getMessage("serviceinactive", name));
}
}
}
return null;
}

Design Issue
• Client side
– (Usually) New instance every method invocation
– No need for message routing
• Why still register in Axi2Configuration ?

axisConfig = configContext.getAxisConfiguration();
if (axisService == null) {
axisService = createAnonymousService();
}
this.axisService = axisService;
if (axisConfig.getService(axisService.getName()) == null) {
axisService.setClientSide(true);
axisConfig.addService(axisService);
} else {
…
}

…

Axis2 client memory leak

More Related Content

What's hot (20)

Similar to Axis2 client memory leak (20)

More from feng lee (6)

Recently uploaded (20)

Axis2 client memory leak