JEEConf 2017 - How to find deadlock not getting into it

How to find deadlock
not getting into it
Nikita Koval, Devexperts LLC
nkoval@devexperts.com
twitter.com/nkoval_

Other tracks are more interesting!
https://github.com/Devexperts/dlcheck
Slide 2/101. Copyright © 2017. Devexperts LLC. All rights reserved.

Real-life
example

interface Con { // Connection
void onRegister(ConnectionManager cm);
void close();
}
class ConnectionManager {
Set<Con> cons = new HashSet<>();
synchronized void register(Con c) {
cons.add(c);
con.onRegister(this);
}
synchronized void unregister(Con c) {
cons.remove(c);
}
...

interface Con { // Connection
void onRegister(ConnectionManager cm);
void close();
}
class ConnectionManager {
Set<Con> cons = new HashSet<>();
synchronized void register(Con c) {
cons.add(c);
con.onRegister(this);
}
synchronized void unregister(Con c) {
cons.remove(c);
}
...
// ... ConnectionManager
synchronized int getConCount() {
return cons.size();
}
synchronized void closeAll() {
for (Con c: cons)
c.close();
}
}

class ConnectionImpl implements Con {
ConnectionManager cm;
@Override
public synchronized void onRegister(ConnectionManager cm) {
this.cm = cm;
}
@Override
public synchronized void close() {
if (cm != null) {
cm.unregister(this);
cm = null;
}
}
}

Run functional test…
Works well!
Run stress test…

public void test() throws Exception {
ConnectionManager cm = new ConnectionManager();
new Thread(() -> {
while (true) {
Connection con = new ConnectionImpl();
cm.register(con);
con.close(); // lock con -> lock cm
}
}).start();
for (int i = 1; i < 100_000; i++) {
System.out.println("Iteration #" + i);
cm.closeAll(); // lock cm -> lock con
}
}

Java stack information for the threads listed above:
===================================================
"Thread-0":
at a.ConnectionManager.unregister(Sample.java:20)
- waiting to lock <0x0000000795b8ee60> (a a.ConnectionManager)
at a.ConnectionImpl.close(Sample.java:45)
- locked <0x0000000795d72fd8> (a a.ConnectionImpl)
at a.StressTest.lambda$test$0(StressTest.java:16)
at a.StressTest$$Lambda$1/1297685781.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
"main":
at a.ConnectionImpl.close(Sample.java:44)
- waiting to lock <0x0000000795d72fd8> (a a.ConnectionImpl)
at a.ConnectionManager.closeAll(Sample.java:29)
- locked <0x0000000795b8ee60> (a a.ConnectionManager)
at a.StressTest.test(StressTest.java:21) <27 internal calls>
Found 1 deadlock.

ConnectionManager cm
Connection con
1 MONITORENTER con
2 con.close()
5 MONITORENTER cm
6 cm.unregister(con)
3 MONITORENTER cm
4 cm.closeAll()
7 MONITORENTER con
8 con.close()

Connection con
1 MONITORENTER con
2 con.close()
5 MONITORENTER cm
3 MONITORENTER cm
4 cm.closeAll()
7 MONITORENTER con
8 con.close()

Connection con
1 MONITORENTER con
2 con.close()
5 MONITORENTER cm
3 MONITORENTER cm
4 cm.closeAll()
7 MONITORENTER con
8 con.close()
HERE IS A
DEADLOCK!

JEEConf 2017 - How to find deadlock not getting into it

Program analysis approaches
• Static analysis
• Model checking
• Dynamic analysis

Dynamic analysis: techniques
• Collecting execution traces (off-line)
- JCarder, MulticoreSDK
• Detection immediately (on-line)
- All context information is available
• Stacktrace, internal state
- VisualThreads

Lock hierarchy
Any two locks should be acquired in the same order in
the same thread
void transfer(Account from, Account to, int amount) {
lock(from)
lock(to)
// Do transfer
unlock(from, to)
}

Lock hierarchy
Any two locks should be acquired in the same order in
the same thread
void transfer(Account from, Account to, int amount) {
// Order lock acquisitions by ids
if (from.id < to.id)
lock(from, to)
else
lock(to, from)
// Do transfer
unlock(from, to)
}

Potential deadlock
• Lock hierarchy compliance – primary method to
avoid deadlocks
• Potential deadlock – this hierarchy violation

Lock graph
• Directed graph
• Vertex ↔ lock
• Edge 𝒂, 𝒃 means that 𝒂 is acquired before 𝒃
•Cycle = potential deadlock

Lock graph: example
Thread 1 Thread 2
1 a.lock()
2 b.lock()
3 unlock(a, b)
4 b.lock()
5 a.lock()
6 unlock(a, b)

Minimization principle: problem
• Two cycles may refer to the same error
Thread 1 Thread 2
1 a.lock()
2 c.lock()
3 b.lock()
4 unlock(a,b,c)
5 b.lock()
6 a.lock()
7 unlock(a,b)

Minimization principle: problem (1)
Thread 1 Thread 2
1 a.lock()
2 c.lock()
3 b.lock()
4 unlock(a,b,c)
5 b.lock()
6 a.lock()
7 unlock(a,b)

Thread 1 Thread 2
1 a.lock()
2 c.lock()
3 b.lock()
4 unlock(a,b,c)
5 b.lock()
6 a.lock()
7 unlock(a,b)

Minimization principle
•Shorter cycle is more useful
•Notify about the shortest one only

Algorithm layout
• Capture lock acquire and release operations
• On lock acquisition:
- Add lock to the set of locks held by the current thread
- Add new edges to the lock graph
- Report new cycles
• On lock release:
- Remove lock from the set of held locks

Lock acquisition and release analysis
Thread 1 Thread 2
4 a.lock()
5 c.lock()
7 b.lock()
8 a.unlock()
9 c.unlock()
10 b.unlock()
1 b.lock()
2 a.lock()
3 a.unlock()
6 b.unlock()
Thread#1Thread#2Slide 40/101. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock acquisition and release analysis (1)
Thread#1Thread#2
b
Thread 1 Thread 2
4 a.lock()
5 c.lock()
7 b.lock()
8 a.unlock()
9 c.unlock()
10 b.unlock()
1 b.lock()
2 a.lock()
3 a.unlock()
6 b.unlock()

Thread#1Thread#2
b
a
Thread 1 Thread 2
4 a.lock()
5 c.lock()
7 b.lock()
8 a.unlock()
9 c.unlock()
10 b.unlock()
1 b.lock()
2 a.lock()
3 a.unlock()
6 b.unlock()

Thread#1Thread#2
b
a
c
Thread 1 Thread 2
4 a.lock()
5 c.lock()
7 b.lock()
8 a.unlock()
9 c.unlock()
10 b.unlock()
1 b.lock()
2 a.lock()
3 a.unlock()
6 b.unlock()

Thread#1Thread#2
a
c
b
Thread 1 Thread 2
4 a.lock()
5 c.lock()
7 b.lock()
8 a.unlock()
9 c.unlock()
10 b.unlock()
1 b.lock()
2 a.lock()
3 a.unlock()
6 b.unlock()

Thread#2Thread#1
c
b
Thread 1 Thread 2
4 a.lock()
5 c.lock()
7 b.lock()
8 a.unlock()
9 c.unlock()
10 b.unlock()
1 b.lock()
2 a.lock()
3 a.unlock()
6 b.unlock()

Thread#1Thread#2
b
Thread 1 Thread 2
4 a.lock()
5 c.lock()
7 b.lock()
8 a.unlock()
9 c.unlock()
10 b.unlock()
1 b.lock()
2 a.lock()
3 a.unlock()
6 b.unlock()

Set of held locks
• Common lock usage patterns
synchronized(o) {
// Do something
}
synchronized void f() {
// Do something
}
l.lock()
try {
// Do something
} finally {
l.unlock()
}
• Locks are acquired and released in LIFO (stack) order

Set of held locks
• Stack with possibility of removing from the middle
• Commonly works in 𝑂 1
• Works in 𝑂(𝐵) at worst
- 𝐵 – number of locks acquired by the current thread

Topological order
• Constructed by the edges of the directed graph
• No topological order ⇔ graph has a cycle

Topological order
• The acyclic part of the lock graph is to be kept
• Edge (𝒂, 𝒃) leads to a topological order violation ⇒
the shortest path 𝒃 ↝ 𝒂 refers to the shortest cycle
• Incremental maintenance
- MNR algorithm suggested by Marchetti-Spaccamela et al.

Limitation: cycle minimization
void transfer(Account from,
Account to,
int amount) {
from.lock()
to.lock()
// Do transfer
unlock(from, to)
}
1 transfer(a, c, …)
2 transfer(c, b, …)
3 transfer(b, a, …)
4 transfer(a, b, …)

Limitation: cycle minimization (1)
Account to,
int amount) {
from.lock()
to.lock()
// Do transfer
unlock(from, to)
}

Account to,
int amount) {
from.lock()
to.lock()
// Do transfer
unlock(from, to)
}

Limitation: cycle minimization
• Adding edge (𝑎, 𝑏) creates new cycle
• Minimization of 𝑏, … , 𝑎 ⇔ minimization of 𝑏 ↝ 𝑎
- Can be implemented with the help of BFS
• Usually 2 or 3 vertexes in cycle
• Current implementation doesn’t support minimization

Limitation: independent cycles
• Logically independent cycles:
- 𝑎, 𝑐, 𝑏
- 𝑎, 𝑏
• Independent cycle is ignored due to minimization principle
• One useful cycle is better 

Limitation: single threaded cycle
Lock a, b
1 a.lock()
2 b.lock()
3 unlock(a, b)
4 b.lock()
5 a.lock()
6 unlock(a, b)

Limitation: guarded cycle
Thread 1 Thread 2
1 g.lock()
2 a.lock()
3 b.lock()
4 unlock(a, b)
5 g.unlock()
6 g.lock()
7 b.lock()
8 a.lock()
9 unlock(a, b)
10 g.unlock()

Dl-Check
• https://github.com/Devexperts/dlcheck
• Handy for testing
- Throws an exception if -Ddlcheck.fail=true
- Or write your own behavior
• Implemented as a Java agent
- java -javaagent:dlcheck.jar -jar your_app.jar
- Modifies byte-code within ASM framework

Byte-code transformation
Lock acquisition and release analysis:
• The first and the last action in synchronized method
• MONITORENTER and MONITOREXIT instructions
• java.util.concurrent.locks.Lock
- Method lock
- Successful invocation of method tryLock
- Method unlock

Synchronized method: in simple terms
// Do something
}
onLockAcquire(this,
2 /*location id*/);
try {
// Do something
} finally {
onLockRelease(this);
}
}
Lock acquisition analysis
Lock release analysis

Synchronized method: in simple terms
// Do something
}
synchronized void g() {
onLockAcquire(this,
2 /*location id*/);
try {
// Do something
} finally {
}
}
What about…
exceptions?

Synchronized method: exception
onLockAcquire(this, 2);
try {
// Do something
} finally {
}
}
try {
// Do something
} finally {
}
}
WRONG CORRECT

Synchronized method: byte-code
synchronized void g();
Code:
0: aload_0 // this
1: iconst_2 // location id
2: invokestatic #22 // onLockAcquire
// Do something
5: aload_0 // this
6: invokestatic #26 // onLockRelease
9: return
10: aload_0 // this
11: invokestatic #26 // onLockRelease
14: athrow
Exception table:
from to target type
0 9 10 any
synchronized void g() {
try {
// Do something
} finally {
}
}

Synchronized block
void f() {
synchronized (o) {
// Do something
}
}
void f() {
MONITORENTER o
try {
// Do something
} finally {
MONITOREXIT o
}
}
void f() {
MONITORENTER o
try {
onLockAcquire(o, 42);
// Do something
} finally {
MONITOREXIT o
onLockRelease(o);
}
}

Synchronized block
void f() {
synchronized (o) {
// Do something
}
}
Анализ взятия блокировки
Анализ отпускания блокировки
void f() {
MONITORENTER o
try {
// Do something
} finally {
MONITOREXIT o
}
}
void f() {
MONITORENTER o
try {
// Do something
} finally {
MONITOREXIT o
onLockRelease(o);
}
}
Not so
simple!

Synchronized block: backstage
void f() {
synchronized (o) {
// Do something
}
}
void f();
Code:
0: aload_0 // this
1: getfield #2 // monitor
4: dup
5: astore_1 // store monitor
6: monitorenter
// Do something
7: aload_1 // monitor
8: monitorexit
9: goto 17 // skip finally
12: astore_2 // exception
13: aload_1 // monitor
14: monitorexit
15: aload_2 // exception
16: athrow
17: return
Exception table:
from to target type
7 9 12 any
12 15 12 any trycatch

Synchronized block: exception
void f() {
MONITORENTER o
try {
// Do something
} finally {
MONITOREXIT o
try {
onLockRelease(o);
catch (Throwable t) {}
}
}
void f() {
MONITORENTER o
try {
// Do something
} finally {
MONITOREXIT o
onLockRelease(o);
}
}
WRONG CORRECT

void f() {
MONITORENTER o
try {
} catch (Throwable t) {
MONITOREXIT o
onLockRelease(o);
throw t;
}
try {
// Do something
...
Synchronized block: not a block
void f() {
MONITORENTER o
try {
// Do something
} finally {
MONITOREXIT o
}
}
Addtry-catch

j.u.c.l.Lock: common usages
void g() {
l.lock();
// Do something
l.unlock();
}
void f() {
l.lock();
try {
// Do something
} finally {
l.unlock();
}
}
void h() {
b = l.tryLock();
// Do something
l.unlock();
}

j.u.c.l.Lock: general usages
void g() {
l.lock();
// Do something
l.unlock();
}
void f() {
l.lock();
try {
// Do something
} finally {
l.unlock();
}
}
void h() {
b = l.tryLock();
// Do something
l.unlock();
}
It looks like unpaired
MONITORENTER
and MONITOREXIT!

j.u.c.l.Lock: tryLock
void h() {
b = l.tryLock();
// Do something
l.unlock();
}
void h() {
b = l.tryLock();
if (b) {
try {
onLockAcquire(l, 2);
} catch (Throwable t) {
l.unlock();
onLockRelease(l);
throw t;
}
}
// Do something
l.unlock();
onLockRelease(l);
}

ClassTransformer
Code template:
public final byte[] transform(..., classFileBuffer) {
ClassReader cr = new ClassReader(classFileBuffer);
ClassWriter cw = new ClassWriter(...);
ClassVisitor classTransformer = new MyTransformer(cw, ...);
cr.accept(classTransformer, EXPAND_FRAMES);
return cw.toByteArray();
}
Let’s try to run…

ClassTransformer: LinkageError
java.lang.LinkageError: loader (instance of
sun/misc/Launcher$AppClassLoader): attempted duplicate class
definition for name: "com/intellij/util/lang/UrlClassLoader"
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
...

java.lang.LinkageError: loader (instance of
sun/misc/Launcher$AppClassLoader): attempted duplicate class
definition for name: "com/intellij/util/lang/UrlClassLoader"
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
...
ClassTransformer: LinkageError
F***ing
ASM!

Problem ClassWriter
protected String getCommonSuperClass(String type1,
String type2) {
Class<?> c1, c2;
ClassLoader loader = getClass().getClassLoader();
try {
c1 = Class.forName(type1.replace('/', '.'), false, loader);
c2 = Class.forName(type2.replace('/', '.'), false, loader);
} catch (Exception e) {
throw new RuntimeException(e);
}
if (c1.isAssignableFrom(c2))
return type1;
...

Write it yourself!
@Override
protected String getCommonSuperClass(String type1,
String type2) {
ClassInfo c = ciCache.getOrBuildClassInfo(type1, loader);
ClassInfo d = ciCache.getOrBuildClassInfo(type2, loader);
if (c.isAssignableFrom(d, ciCache, loader))
return type1;
if (d.isAssignableFrom(c, ciCache, loader))
return type2;
...
ClassInfo is available BEFORE transformation

JAgent framework
• This problem is solved in JAgent
- https://github.com/Devexperts/jagent
• Makes java agents writing more simple

Dependencies conflict
• Several agents are used:
java -javaagent:agent1.jar -javaagent:agent2.jar
–jar my_app.jar
ASM 3 ASM 4
• Dependencies should be isolated

Isolation techniques
• Repackaging
- maven-shade-plugin (or analogue)
- Needs to know transitive dependencies and their
structure
- Sometimes doesn’t work as well (reflection)
• Separate ClassLoader
- Cannot be used for injected code

JAgent will help us
JAgent API
core:
+ premain
+ runtime
code
External libraries
JAgent Impl
transformer:
transformation
code
External libraries
Runs transformation
code via JAgent API
Inserts new code
JAgent ClassLoaderSystem ClassLoader

Benchmarks
• SpecJVM2008: Apache Derby (4 and 40 threads)
• DaCapo: Apache Lucene
• DaCapo: Banking application
• …

Performance impact
9.7
128.2
1.25
2.6
7.3
120.71
40.33
3.25
1
2
4
8
16
32
64
128
Derby 4T Derby 40T Lucene Bank
SLOWDOWNFACTOR
Jcarder MulticoreSDK

Performance impact
1.53 1.76
1.03
2.6
9.7
128.2
1.25
2.6
7.3
120.71
40.33
3.25
1
2
4
8
16
32
64
128
Derby 4T Derby 40T Lucene Bank
SLOWDOWNFACTOR
Dl-Check Jcarder MulticoreSDK

Memory usage impact
1
1.07 1.06 1
0.61
1.22
1 1
0.74
1.12
1.35
1
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
Derby 4 Derby 40T Lucene Bank
INCREASEFACTOR
Dl-Check Jcarder MulticoreSDK

Real-life
example

public void testClose() {
cm.register(con);
// lock cm
// lock con for con.onRegister()
assertEquals(1, cm.getConCount());
con.close();
// lock con
// lock cm for cm.unregister()
assertEquals(0, cm.getConCount());
}

==========================
!!! Potential deadlock !!!
==========================
### Cycle in lock graph: ###
Lock ConnectionImpl@d70c109 was acquired at:
a.ConnectionImpl.onRegister(Sample.java:39)
a.ConnectionImpl.close(Sample.java:44)
Lock ConnectionManager@2be94b0f was acquired at:
a.ConnectionManager.getConCount(Sample.java:24)
a.ConnectionManager.register(Sample.java:15)
a.ConnectionManager.unregister(Sample.java:20)
### Current lock stack: ###
Lock ConnectionImpl@d70c109 was acquired at:
a.ConnectionImpl.onRegister(Sample.java:39)
### Current stacktrace: ###
a.ConnectionManager.unregister(Sample.java:20)
a.FunctionalityTest.testClose(FunctionalityTest.java:22)
...

Evaluation in real projects
• Internal projects
• IntelliJ IDEA
• Kotlin
• …

Summary
• On-line algorithm for detecting potential deadlocks
• JAgent framework for writing Java agents
- https://github.com/Devexperts/jagent
• Dl-Check tool has been implemented
- https://github.com/Devexperts/dlcheck
• Further work:
- Contracts to describe lock acquisition rules

Thank you for attention!
Nikita Koval, Devexperts LLC
nkoval@devexperts.com
twitter.com/nkoval_

JEEConf 2017 - How to find deadlock not getting into it

More Related Content

What's hot (20)

Similar to JEEConf 2017 - How to find deadlock not getting into it (20)

Recently uploaded (20)

JEEConf 2017 - How to find deadlock not getting into it