Alternatives for Systems Integration in the NoSQL Era - NoSQL Roadshow 2013

© Talend 2013 “Alternatives for Systems Integration in the NoSQL Era” by Kai Wähner
Alternatives for Systems Integration
in the NoSQL Era
Kai Wähner
kwaehner@talend.com
@KaiWaehner
www.kai-waehner.de
9/16/2013

Kai Wähner
Consulting
Developing
Coaching
Speaking
Writing
Main Tasks
Requirements Engineering
Enterprise Architecture Management
Business Process Management
Architecture and Development of Applications
Service-oriented Architecture
Integration of Legacy Applications
Cloud Computing
Big Data
Contact
Email: kwaehner@talend.com
Blog: www.kai-waehner.de/blog
Twitter: @KaiWaehner
Social Networks: Xing, LinkedIn

What is the key message?

Key messages
NoSQL cannot be avoided, and must be integrated!
NoSQL integration is already possible!
Different APIs, Frameworks and Products helps a lot!

Agenda
1) Introduction to NoSQL
2) Systems Integration
3) API
4) Integration Framework
5) Enterprise Service Bus
6) Integration Suite
7) Custom Connectors
Live Demos

Agenda
3) API

The evolving database landscape

NoSQL
Big Data
Cloud
Complementary concepts

Agenda
3) API

A new era: NoSQL

➜„Spaghetti communication“
What is the problem?

What is the problem?
• Applications
• Interfaces
• Technologies
• Products
Growth

Solution: Systems integration
All Roads lead
to Rome ...

Wishes for integrators
• Standardized Modeling
• Efficient Realization
• Automatic Testing

Systems integration in the NoSQL era

Complexity
of Integration
Enterprise
Service Bus
Integration Suite
Low High
Integration
Framework
IncludesIncludes
API

API
• Proprietary API
• Vendor API
• Generic API
• Web Service Interface
• REST Web Service
• SOAP Web Service

APIs
➜ Vendor APIs

Example: Amazon AWS S3 Key Value Store (Java API)
AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials(
S3Sample.class.getResourceAsStream("AwsCredentials.properties")));
String bucketName = "my-first-s3-bucket-" + UUID.randomUUID();
String key = "MyObjectKey";
try {
s3.createBucket(bucketName);
s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile()));
S3Object object = s3.getObject(new GetObjectRequest(bucketName, key));
ObjectListing objectListing = s3.listObjects(new ListObjectsRequest()
.withBucketName(bucketName)
.withPrefix("My"));
s3.deleteObject(bucketName, key);
s3.deleteBucket(bucketName);
} catch (AmazonServiceException ase) {
// error handling...
} catch (AmazonClientException ace) {
}

Example: Amazon AWS S3 Key Value Store (Ruby API)

APIs
➜ Generic Storage APIs

jClouds (Generic API)
Generic API for IaaS

Blobstore API

Several different
NoSQL providers
supported

jClouds (Generic API) – AWS S3 Blobstore (Java)
Use another blobstore?
Just change this line!

APIs
➜ Generic NoSQL specific APIs
Kundera

EclipseLink (JPA) NoSQL Support
http://wiki.eclipse.org/EclipseLink/Examples/JPA/NoSQL
As of EclipseLink 2.4, EclipseLink has added JPA support for NoSQL databases, initially
with support for MongoDB and Oracle NoSQL.
EclipseLink's NoSQL support allows the JPA API and JPA annotations/xml to be used with
NoSQL data. EclipseLink also supports several NoSQL specific annotations/xml including
@NoSQL that defines a class to map NoSQL data.

Hibernate OGM (Object/Grid Mapper)
http://www.hibernate.org/subprojects/ogm.html

Kundera
https://github.com/impetus-opensource/Kundera

APIs
➜ REST / SOAP APIs

SaaS – Salesforce (REST API)
• SOAP
• REST

Complexity
of Integration
Enterprise
Service Bus
Integration Suite
Low High
Integration
Framework
INTEGRATION
Connectivity
Routing
Transformation

Integration
Integration Frameworks

Enterprise Integration Patterns

Standalone
Application
Server
Web Container
Cloud
Spring
Container
Deployment

AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials(
S3Sample.class.getResourceAsStream("AwsCredentials.properties")));
String bucketName = "my-first-s3-bucket-" + UUID.randomUUID();
String key = "MyObjectKey";
try {
s3.createBucket(bucketName);
s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile()));
S3Object object = s3.getObject(new GetObjectRequest(bucketName, key));
ObjectListing objectListing = s3.listObjects(new ListObjectsRequest()
.withBucketName(bucketName)
.withPrefix("My"));
s3.deleteObject(bucketName, key);
s3.deleteBucket(bucketName);
} catch (AmazonServiceException ase) {
} catch (AmazonClientException ace) {
}
No longer „glue code“

// Producer
from(“ftp:toS3")
.setHeader(S3Constants.KEY, simple(“order.txt"))
.to("aws-s3://myBucket?accessKey=" + a+ "&secretKey= " + s)
// Consumer
from(„salesforce://orders__c?user=dummy1“)
.filter(„attributeType==‚dvd‘“)
.to(“ibm-database:orderData")
Domain Specific Language (Camel)

NoSQL with an Integration Framework

Document-oriented database

Document-oriented database
• 10gen
• stores structured data as JSON-like
documents with dynamic schemas
• REST API and several SDKs (Java, .NET, Ruby,
PHP, Python, etc.)
• Ad hoc queries, indexing, replication, load
balancing
• Powerful, but also easy to use and flexible
• Example: Disney persists state information of
online games in a common object repository.

Code example: MongoDB Java Driver

Code example: camel-mongodb component
// Producer
from(“jms:FlightDocumentQueue")
.to("mongodb:myDb?database=flights
&collection=tickets
&operation=insert");
// Consumer
from("mongodb:myDb?database=flights
&collection=cancellations
&tailTrackIncreasingField=departureTime")
.to(“jms:CancelledFlightsQueue");

Live demo
Integration of NoSQL with an integration framework in action...

Some more...

Key-Value database

Code example: camel-aws component
// Producer
from(„jms:toS3Queue")
.setHeader(S3Constants.KEY, simple(“order.txt"))
.to("aws-s3://myBucket?accessKey=" + a + "&secretKey= " + s)
// Consumer
from("aws-s3://myBucket?accessKey=“ + a + "&secretKey=" + s)
.to("log:S3logging")

Code example: camel-jclouds component
from("direct:toJcloudsAwsS3")
.setHeader(JcloudsConstants.BLOB_NAME, "jclouds-demo-tutorial.txt")
.setHeader(JcloudsConstants.CONTAINER_NAME, "kw-s3-data")
.to("jclouds:blobstore:aws-s3")
from("direct:toJcloudsMicrosoftAzure")
.setHeader(JcloudsConstants.BLOB_NAME, "jclouds-demo-tutorial.txt")
.setHeader(JcloudsConstants.CONTAINER_NAME, "kw-s3-data")
.to("jclouds:blobstore:azureblob")

In-memory database

Code example: camel-hazelcast component
// Producer
from("direct:add")
.setHeader(HazelcastConstants.OPERATION, „add“)
.to("hazelcast:queue:foo");
// Consumer
from("hazelcast:queue:foo")
.log(“content of object foo: ${body}");

Graph-oriented database

Code example: camel-neo4j component
// Producer
from("jms:createNewNeo4jNode")
.to("neo4j:http://Neo4jServer:7474/data");
// Consumer
from(„neo4j://todo)...
Not implemented in current Camel release (2.12)  Maybe 2.13?
 Use Camel‘s REST Connectors (shown in some minutes...)

Column-oriented database

Code example: camel-hbase component

Connectivity
Routing
Transformation
INTEGRATION
Tooling
Monitoring
Support
Complexity
of Integration
Enterprise
Service Bus
Integration Suite
Low High
Integration
Framework
+
What is an Enterprise Service Bus (ESB)?

Many important players ...
Gartner Quadrant 2013 for Application Integration

Oracle Service Bus

Three IBM ESBs
 Rebranded: WebSphere Integration Bus

Mule ESB

Talend ESB

Three ESBs (JBoss ESB, Switchyard, Fuse ESB)
Two BPMs (jBPM, Polymita)
No unified platform (yet)
Red Hat / JBoss

NoSQL with an Enterprise Service Bus

Example: Talend ESB

TalendAdministrationCenter
TalendESBStudio
TalendESB
Service Development
Mediation &
Integration
Testing
Build & Deploy
Performance &
Availability
Web Services Stack
Mediation &
Integration
Message Broker
Service Container
Loadbalancing & High
Availability
Security
Business Rules
Management
Development Runtime Operation
Documentation & Examples
Professional Services Certified Partners
Training & Certification24x7 Support Indemnification
Maintenance
Project
Repository
Configuration
Deployment
Repository
Example: Talend ESB

Example: Talend ESB
Eclipse STP/WTP
soapUI
Route Designer
Mediation
Service Designer
Integration
Apache Maven
Build & Deploy
Apache CXF
REST & Web Services
Apache Camel
Mediation
Apache ActiveMQ
Message Broker
Apache Karaf / Cellar
OSGi / Clustering
Eclipse Equinox
OSGi
Service Locator &
Service Act.
Monitoring
Distributed Registry / Tracking
Secure Token Server
Security
Service Activity Monitoring
and Service Locator UI
Management,
Configuration &
Monitoring
Repository
Metadata & Projects
Development Runtime Operation
Documentation & Examples
vFabric
Hyperic HQ
Apache Archiva
Artifact repository
TalendESBStudio
TalendESBRuntime
TalendAdministrationCenter
Professional Services Certified Partners
Training & Certification24x7 Support Indemnification
Maintenance

Route Builder
 Endpoints
 EIPs
 Processors
 Custom Connectors
Configuration
 Connectors
 Endpoints
Code Generation
 100% Java
 Camel Code
 Packaged as OSGi
Bundles
Execution in the IDE
 Debugging
 Live statistics
 Short dev cycles
Example: Talend ESB Studio

Live demo
Integration of NoSQL with an Enterprise Service Bus in action...

Agenda
3) API
4) Enteprise Service Bus

Connectivity
Routing
Transformation
Complexity
of Integration
Enterprise
Service Bus
Integration Suite
Low High
Integration
Framework
INTEGRATION
Tooling
Monitoring
Support
+
BUSINESS PROCESS MGT.
BIG DATA / MDM
REGISTRY / REPOSITORY
RULES ENGINE
„YOU NAME IT“
+
What is an Integration Suite?

Oracle Platform

IBM WebSphere Stack

WSO2 Carbon Platform

Data
Quality
Data
Integration
MDM ESB
Big
Data
Big Data
 Open source license
 Free of charge
 Optional support
 Based on open source
projects such as
Eclipse or Apache
Camel, CXF, Hadoop
 Commercial license
 Subscription model
 Support included
Data
Quality
Data
Integration MDM ESB BPM
Talend Unified Platform

Integration of NoSQL with an integration suite in Action...
Live Demo

Open Source
Proprietary
ESB Vendor == Integration Suite Vendor

ESB BPM Data Quality
ESB Vendor == Integration Suite Vendor

• A lot of glue code
• Testing
• Bugfixing
• No support
Some other people already had
the problems you would have!
Custom Combination of ESB, BPM, etc.

Agenda
3) API
5) Enteprise Service Bus

Custom NoSQL Connectors
Easy to realize for all
integration alternatives *
• Integration Framework
• Enterprise Service Bus
• Integration Suite
* At least for open source solutions

Live demo (Example Salesforce Connector)
Custom NoSQL Connectors in action...

Alternative for custom NoSQL Connectors
• SOAP
• REST

Code example: REST API for Salesforce object store
// Salesforce Query (SOQL) via REST API
from("direct:salesforceViaHttpLIST")
.setHeader("X-PrettyPrint", 1)
.setHeader("Authorization", accessToken)
.setHeader(Exchange.CONTENT_TYPE, "application/json")
.to("https://na14.salesforce.com/services/data/v20.0/query?q=SELECT+name+from
+Article__c")
// Salesforce CREATE via REST API
from("direct:salesforceViaHttpCREATE")
.setHeader("X-PrettyPrint", 1)
.setHeader("Authorization", accessToken)
.setHeader(Exchange.CONTENT_TYPE, "application/json“)
.to("https://na14.salesforce.com/services/data/v20.0/sobjects/Article__c")

Did you get the key message?

Thank you for your attention. Questions?
kwaehner@talend.com
www.kai-waehner.de
LinkedIn / Xing
@KaiWaehner

Alternatives for Systems Integration in the NoSQL Era - NoSQL Roadshow 2013

More Related Content

What's hot (7)

Viewers also liked (7)

Similar to Alternatives for Systems Integration in the NoSQL Era - NoSQL Roadshow 2013 (20)

More from Kai Wähner (20)

Recently uploaded (20)

Alternatives for Systems Integration in the NoSQL Era - NoSQL Roadshow 2013