[JEEConf-2017] RxJava as a key component in mature Big Data product

RXJAVA
AS A KEY COMPONENT
IN A MATURE
BIG DATA PRODUCT
by Igor Lozynskyi

HELLO!I am Igor Lozynskyi
I have 8 years of Java experience, currently work at
You can find me at @siromaha
2

Agenda
■ RxJava basics
■ Zoomdata (goals & architecture)
■ “Good” & “bad” RxJava experience
■ Some thoughts about Rx future
3

Disclaimer
■ This isn’t a sponsored talk!
■ This talk isn’t Zoomdata advertisement!
■ It’s just our experience.
4

WHAT IS
ZOOMDATA?
The Fastest Visual Analytics for Big Data
www.zoomdata.com
5

■ Data visualization software
■ Big Data
■ 30+ DB types
■ SQL, NoSQL, raw data (CSV)
■ Pretty charts
■ Platform!
Zoomdata (ZD):
6

■ Static is boring
■ Dynamic is interesting
Instant feedback
■ We do not like to wait
9

ZOOMDATA
ARCHITECTURE
very simplified view
10
very

[JEEConf-2017] RxJava as a key component in mature Big Data product

15
class QueryProcessor implements Observable.Transformer<QueryRequest, QueryResult> {
Observable<QueryResult> call(Observable<QueryRequest> requests) {
...
}
}

17
return requests
.takeWhile(req -> !isCancellation(req))
.map(this::validateRequest)
.compose(r -> requestProcessor(r))
.onErrorReturn(this::handleProcessingErrors)
}
Observable<QueryResult> requestProcessor(Observable<QueryRequestParsed> requests) {
return mainQuery(requests);
}
}

19
return requests
}
Observable<QueryResult> requestProcessor(Observable<QueryRequestParsed> requests) {
return merge(
mainQuery(requests),
intermediateResults(requests) // approximate result
);
}
}

21
return requests
}
Observable<QueryResult> requestProcessor(Observable<QueryRequestParsed> request) {
return merge(
mainQuery(requests),
intermediateResults(requests),
mainQueryProgress(requests) // based on main query
);
}
}

OUR RXJAVA
EXPERIENCE
Disclaimer:
■ Your mileage may vary!
24

Effortless mapping of
WebSocket to
Observable Stream
26

27
class RxWebSocketSession<T extends WebSocketMessage<?>> {
spring.WebSocketSession nativeSession;
Subject<T, T> inputSubject;
Subscriber<T> socketSubscriber;
public RxWebSocketSession(spring.WebSocketSession nativeSession) {
...
}
public Observable<T> getInput() {
return inputSubject;
}
public Subscriber<T> getOutput() {
return socketSubscriber;
}
...
}

28
class NativeWebSocketHandler extends TextWebSocketHandler {
IRxWebSocketHandler<TextMessage> sessionHandler;
Map<WebSocketSession, RxWebSocketSession<TextMessage>> sessions;
void afterConnectionEstablished(WebSocketSession session) {
RxWebSocketSession<TextMessage> rxSession
= new RxWebSocketSession<>(synchronizedSession(session));
sessions.put(session, rxSession);
sessionHandler.afterConnectionEstablished(rxSession);
}
void handleTextMessage(WebSocketSession session, TextMessage message) {
sessions.get(session).handleMessage(message);
}
void afterConnectionClosed(WebSocketSession session, CloseStatus status) {
RxWebSocketSession<TextMessage> rxSession = sessions.remove(session);
rxSession.close();
sessionHandler.afterConnectionClosed(rxSession, status);
}
}

29
class RxWebSocketHandler implements IRxWebSocketHandler<TextMessage> {
rx.Observable.Transformer<String, String> messageTransformer;
rx.Scheduler sendingIoScheduler;
public void afterConnectionEstablished(RxWebSocketSession<TextMessage> session) {
session
.getInput()
.map(TextMessage::getPayload)
.compose(messageTransformer)
.onBackpressureBuffer(OUTGOING_WS_BUFFER_CAPACITY,
() -> reportMessageDropped(session.getId()),
ON_OVERFLOW_DROP_OLDEST)
.observeOn(sendingIoScheduler)
.map(TextMessage::new)
.subscribe(session.getOutput());
}
}

It is easy to stream
intermediate results
to a client
30

It is easy to coordinate complex
& long running workflows
31

RxJava
allows to compose
dozens of
reactive streams
32

Query Cancellation
out-of-the-box
33

RxJava is
very powerful
Any stream
transformation
that you can imagine!
Image credits: Denys Nevozhai

Use TDD for Rx code!
Lack of unit tests
for Rx code
Many integration tests
instead
36Image credits: Matthew Henry

Stream code should not be too
complicated!
Rule: “10 lines max” for method
37

38
protected Observable<Map<QueryStream, ComputedTasks>> play(Map<QueryStream, ComputedTasks> initial) {
return concat(just(primaryTimeWindow), timeWindowStream)
.map(tw ->
playingReadRequests.columnKeySet().stream()
.collect(toMap(
calc -> calc,
calc -> calc.calculateTimewindow(tw)))
)
.buffer(2, 1)
.filter(contextsMaps -> contextsMaps.size() == 2)
.zipWith(checkPointsStream::iterator, Pair::of)
.scan(initialTasksByTimewindow, (tasksMap, actions) -> {
Map<TimewindowCalculator, List<ReadAction>> actionsByWindow = actions.getLeft();
Boolean doCheckpoint = actions.getRight();
Map<TimewindowCalculator, Map<QueryStream, ComputedTasks>> tasksByTimewindow = MapStream.of(actionsByWindow)
.map(e -> {
TimewindowCalculator timewindow = e.getKey();
List<ReadAction> timewindowActions = e.getValue();
Map<QueryStream, ComputedTasks> tasksOfTimewindow = tasksMap.get(timewindow);
Map<QueryStream, ComputedTasks> tasksByStream = calculateStreamTasksWithActions(timewindow,
tasksOfTimewindow, timewindowActions, doCheckpoint);
return Pair.of(timewindow, tasksByStream);
}).collect(toMap(Pair::getKey, Pair::getValue));
return tasksByTimewindow;
})
.map(tasksByTimewindow -> {
Collection<Map<QueryStream, ComputedTasks>> tasksByStreamMaps = tasksByTimewindow.values();
return mergeTasksMapsByStream(tasksByStreamMaps);
})
.doOnNext(this::rememberLastTasksReference)
.skip(1);
}

39
.map(tw ->
playingReadRequests.columnKeySet().stream()
.collect(toMap(
calc -> calc,
calc -> calc.calculateTimewindow(tw)))
)
.buffer(2, 1)
.filter(contextsMaps -> contextsMaps.size() == 2)
.scan(initialTasksByTimewindow, (tasksMap, actions) -> {
Map<TimewindowCalculator, List<ReadAction>> actionsByWindow = actions.getLeft();
Boolean doCheckpoint = actions.getRight();
Map<TimewindowCalculator, Map<QueryStream, ComputedTasks>> tasksByTimewindow = MapStream.of(actionsByWindow)
.map(e -> {
TimewindowCalculator timewindow = e.getKey();
List<ReadAction> timewindowActions = e.getValue();
Map<QueryStream, ComputedTasks> tasksOfTimewindow = tasksMap.get(timewindow);
Map<QueryStream, ComputedTasks> tasksByStream = calculateStreamTasksWithActions(timewindow,
tasksOfTimewindow, timewindowActions, doCheckpoint);
return Pair.of(timewindow, tasksByStream);
}).collect(toMap(Pair::getKey, Pair::getValue));
return tasksByTimewindow;
})
.map(tasksByTimewindow -> {
Collection<Map<QueryStream, ComputedTasks>> tasksByStreamMaps = tasksByTimewindow.values();
return mergeTasksMapsByStream(tasksByStreamMaps);
})
.skip(1);
}

40
.map(this::calculateTimeWindows)
.buffer(2, 1)
.filter(this::contextMapsOfSizeTwo)
.scan(initialTasksByTimewindow, this::updateJobResultsForNewTimeWindow)
.map(tasksByTimewindow -> mergeTaskMapsByStream(tasksByTimewindow.values()))
.skip(1);
}

41
.map(this::calculateTimeWindows)
.buffer(2, 1)
.filter(this::contextMapsOfSizeTwo)
.map(this::transitionWindow)
.scan(initialTasksByTimewindow, this::updateJobResultsForNewTimeWindow)
.map(tasksByTimewindow -> mergeTaskMapsByStream(tasksByTimewindow.values()))
.skip(1);
}

It is easy to forget
about one very
important thing
42
Remember next slide
Image credits: Oscar Sutton

Remember not to overflow
your client!
Your client will not be happy
47

Remember not to overflow
yourself!
48
Slow subscriber may cause data
buffering
Buffering may be costly

Rx
silently introduces
a notion of time
into your API
50

Track your stream lifetime!
GroupBy operator
52

54
Child streams live along with
parent!

Unpleasant when child streams hold
resources
56

Subscription takes some time
Be careful not to miss messages
57

Reactive transformations are
time dependent
59
Zoomdata had an issue due to this

https://github.com/ReactiveX/RxJava/issues/3522 (fixed in 1.0.15)
61

Hard to code complicated
context-dependent workflow
■ Loop vs flatMap
■ State should be incorporated into the stream!
62

Try not to mutate objects
inside Rx streams
■ Rx stream is a concurrent system
■ In concurrent systems mutation is evil
■ Zoomdata had such issues
63

Think twice before providing
pull API over Rx stream
64

■ Pull push hard & risky
■ Pull push mutability disaster
Think twice before providing
pull API over Rx stream
■ Push pull ok
65

It is better to control
execution threads
on your own
■ Doing CPU in IO thread
■ Doing IO in CPU thread
66Image credits: Haiku Deck

Care about concurrency
■ No ThreadLocal tricks!
■ flatMap/zip/merge may move execution into
unexpected (for you) thread
■ Immutability is the only reasonable way to go
67

Complexity is
still there!
69Image credits: sciencemag.org

■ WebSocket to Rx stream mapping: 3 files
■ ZD query processing is almost entirely RxJava code
■ Most algorithms live in 10 files (~1000 loc)
□ playing, sharpening, progress, timeline, push/pull data
strategies, main query, timeline, and much more
■ RxJava is complicated, but very powerful
■ Zoomdata benefits a lot from RxJava usage!
Is RxJava good enough for ZD?
71

■ Java 9 & Spring 5 will spread reactive programming
■ We expect big shift towards Project Reactor
■ We will have more reactive and proactive software!
□ Zoomdata is already proactive
Some thoughts about Rx future
72

THANKS!
Contact me:
@siromaha
aigooor [at] gmail [dot] com
Presentation template by SlidesCarnival
Image credits: Denys Nevozhai

[JEEConf-2017] RxJava as a key component in mature Big Data product

More Related Content

What's hot (20)

Similar to [JEEConf-2017] RxJava as a key component in mature Big Data product (20)

Recently uploaded (20)

[JEEConf-2017] RxJava as a key component in mature Big Data product