Develop modern apps using Spring ecosystem at time of BigData
2 likes1,397 views
This document discusses using Spring Boot and Spring XD for developing modern applications. It provides an overview of Spring Boot's capabilities for rapid application development. Spring XD is introduced as a platform for building data ingestion, processing, and analytics pipelines. The document also includes demonstrations of creating simple applications with Spring Boot and Spring XD.
![[DEMO:]
 Tweets
 Boot
 Data
 REST
 app
Â
1.⯠Basic
 5
 mins
 Boot
 Data
 Rest
 app
 which
 simply
 works!
Â
2.⯠Lets
 add
 logging
 and
 external
 properMes
 support
Â
3.⯠Lets
 add
 integraMon
 tests
Â
4.⯠Lets
 add
 Tweet
 lifecycle
 event
 listeners
Â
5.⯠Lets
 add
 Actuator
 monitoring
 support
Â
6.⯠Lets
 add
 much
 more
 ïŹelds
 to
 Tweet
 enMty
Â
7.⯠Lets
 add
 Tweet
 compact
 projecMon
Â
8.⯠Lets
 add
 validaMon
Â
9.⯠Lets
 add
 addiMonal
 stuïŹ
 to
 Tweet
 HATEOUS
 json
 view
Â
10.âŻLets
 add
 more
 custom
 queries
Â
11.âŻBut
 beVer
 do
 not
 try
 to
 make
 Java
 8
 data
 and
 Mme
Â
types
 work))
Â
Â](https://image.slidesharecdn.com/modernappswithspring-151107165800-lva1-app6892/85/Develop-modern-apps-using-Spring-ecosystem-at-time-of-BigData-7-320.jpg)
![[DEMO:]
 Spring
 XD
 toy.
 Play
 with
 me!
Â
1.⯠Show
 XD
 bootstrap
 in
 singlenode
 mode
 and
 deploy
 simple
Â
streams.
 Show
 XD
 shell
 and
 XD
 UI
 usages.
Â
2.⯠Show
 XD
 distributed
 mode
 conïŹguraMon
 with
 external
Â
Zookeeper,
 DB
 and
 Redis.
 Show
 XD
 meta
 data
 in
Â
Zookeeper.
 Deploy
 same
 simple
 streams
 on
 cluster
 and
Â
show
 XD
 cluster
 fault
 tolerance
 and
 recovery.
Â
3.⯠Show
 example
 of
 predeïŹned
 XD
 module
 conïŹguraMon
Â
(e.g.
 hdfs)
Â
4.⯠Create
 custom
 module
 and
 upload
 it
 into
 XD
Â
5.⯠Show
 DEMO
 for
 stream
 reading
 tweets
 from
 TwiVer
 and
Â
posMng
 them
 into
 REST
 client
 app
 developed
 before
 via
Â
HTTP
 and
 wriMng
 them
 into
 HDFS
 in
 parallel
Â](https://image.slidesharecdn.com/modernappswithspring-151107165800-lva1-app6892/85/Develop-modern-apps-using-Spring-ecosystem-at-time-of-BigData-16-320.jpg)
Develop modern apps using Spring ecosystem at time of BigData
- 1. Develop  modern  apps  using  Spring  ecosystem  at  BigData  era Â
- 2. LEAD Â SOFTWARE Â ENGINEER Â AT Â EPAM Â SYSTEMS. Â PATIONATE Â DEVELOPER, Â SPEAKER, Â ACTIVE Â MEMBER Â OF Â KIEV Â JUG. Â PARTICIPATE Â IN Â DIFFERENT Â EDUCATIONAL Â INITIATIVES, Â ENGINEERING Â EVENTS Â AND Â JCP/ADOPTJSR Â PROGRAMS. Â Â OLEG TSAL-TSALKO ABOUT ME
- 3. Spring   for  a  long  Mme   no  more  than   just  a  Framework Â
- 6. Spring  Boot  âąâŻ General  purpose  plaRorm  for  Web  app  and  client  app  development  which  makes  complex  apps  development  look  easy   âąâŻ Rapid  development  of  diïŹ Â type  applicaMons  based  on  solid  Spring  foundaMon  âąâŻ No  boilerplate  code.  You  write  only  needfull/ funcMonal  code.  Spring  knows  beVer  how  make  it  all  work.  âąâŻ Nevertheless  easy  to  customize  and  extend  âąâŻ Simple  and  robust  dependency  management Â
- 7. [DEMO:]  Tweets  Boot  Data  REST  app  1.⯠Basic  5  mins  Boot  Data  Rest  app  which  simply  works!  2.⯠Lets  add  logging  and  external  properMes  support  3.⯠Lets  add  integraMon  tests  4.⯠Lets  add  Tweet  lifecycle  event  listeners  5.⯠Lets  add  Actuator  monitoring  support  6.⯠Lets  add  much  more  ïŹelds  to  Tweet  enMty  7.⯠Lets  add  Tweet  compact  projecMon  8.⯠Lets  add  validaMon  9.⯠Lets  add  addiMonal  stuïŹ Â to  Tweet  HATEOUS  json  view  10.âŻLets  add  more  custom  queries  11.âŻBut  beVer  do  not  try  to  make  Java  8  data  and  Mme  types  work))  Â
- 9. Spring  XD  âąâŻ General  purpose  plaRorm  for  developing  BigData  apps  for  data  ingesMon,  data  stream  and  batch  processing,  data  export  and  data  analyMcs  âąâŻ Scalable  and  fault  tolerant  plaRorm  in  your  service  âąâŻ DeclaraMve  processing  workïŹow  development,  deployment  and  management  âąâŻ Huge  number  of  predeïŹned  available  modules  that  can  be  used  straight  away  to  bring  your  processing  pipelines  in  life  in  seconds  âąâŻ Easy  to  extend  plaRorm  and  pluggable  custom  modules  âąâŻ Build  on  top  of  rich  and  robust  Spring  foundaMon Â
- 15. Real  world  example  for  XD  Raw  data  is  captured  from  mobile  phones  in  JSON  format  by  a  Spring  XD  cluster   where  several  processes  are  performed.   The  data  is  then  stored  in  an  HDFS  cluster  where  Spring  XD  batch  jobs  use  SQL  via  the  HAWQ   interface  to  HDFS  and  store  the  calculated  reports  in  Redis.   Spring  Boot  is  then  used  with  Angular.js  and  D3.js  to  show  analyMcs  to  end  users. Â
- 16. [DEMO:]  Spring  XD  toy.  Play  with  me!  1.⯠Show  XD  bootstrap  in  singlenode  mode  and  deploy  simple  streams.  Show  XD  shell  and  XD  UI  usages.  2.⯠Show  XD  distributed  mode  conïŹguraMon  with  external  Zookeeper,  DB  and  Redis.  Show  XD  meta  data  in  Zookeeper.  Deploy  same  simple  streams  on  cluster  and  show  XD  cluster  fault  tolerance  and  recovery.  3.⯠Show  example  of  predeïŹned  XD  module  conïŹguraMon  (e.g.  hdfs)  4.⯠Create  custom  module  and  upload  it  into  XD  5.⯠Show  DEMO  for  stream  reading  tweets  from  TwiVer  and  posMng  them  into  REST  client  app  developed  before  via  HTTP  and  wriMng  them  into  HDFS  in  parallel Â
- 18. Spring  XD  (out  of  scopeâŠ)  âąâŻ Show  XD  job  running  over  Hadoop  âąâŻ Show  RabbitMQ  integraMon  âąâŻ Show  DEMO  on  running  either  MapReduce  job  using  Cascading  or  Pig  over  HDFS  tweets  data  OR  running  Spark  app  over  HDFS  tweets  data Â
- 19. Spring  Boot/XD  enable  twelve  factor  apps  I.  Codebase  One  codebase  tracked  in  revision  control,  many  deploys  II.  Dependencies  Explicitly  declare  and  isolate  dependencies  III.  ConïŹg  Store  conïŹg  in  the  environment  IV.  Backing  Services  Treat  backing  services  as  aVached  resources  V.  Build,  release,  run  Strictly  separate  build  and  run  stages  VI.  Processes  Execute  the  app  as  one  or  more  stateless  processes  VII.  Port  binding  Export  services  via  port  binding  VIII.  Concurrency  Scale  out  via  the  process  model  IX.  Disposability  Maximize  robustness  with  fast  startup  and  graceful  shutdown  X.  Dev/prod  parity  Keep  development,  staging,  and  producMon  as  similar  as  possible  XI.  Logs  Treat  logs  as  event  streams  XII.  Admin  processes  Run  admin/management  tasks  as  one-ÂâoïŹ Â processes Â
- 20. Everything  wasnât  so  smooth  though⊠ âąâŻ SPEL  doesnât  work  on  ïŹlter  with  JSON  elements  access  implicitly  as  it  works  with  transformer  module.  To  make  ïŹlter  work  you  need  either  to  convert  JSON  to  Tuple  or  to  use  JSON  Xpath  SPEL  syntax  âąâŻ When  transform  JSON  into  Tuple  and  back  WARNings  are  thrown  on  Jackson  transformaMon???  âąâŻ In  order  implicit  converMon  to  Tuple  works,  you  need  to  specify  contentType  MIME  header  in  HTTP  request  otherwise  it  canât  automaMcally  cast  text/plain  MIME  message  to  Tuple  âąâŻ  If  you  deploy  hVp  module  on  port  which  is  occupied  it  wonât  fail  on  deploy  but  your  stream  will  simply  silently  not  work  âąâŻ SomeMmes  container  gets  broken  and  smth  stops  to  work  like  Taps  in  my  case  for  example  and  no  errors  in  logs.  Only  container  restart  helps.  âąâŻ Spring  data  rest  doesnât  expose  enMty  IDs  in  JSON  body  by  default  âąâŻ When  Mongo  enMty  save  operaMon  failing   as  a  result  of  HTTP  call  no  excepMons  visible  in  logs,  just  500  ERROR  response  to  client  without  proper  stack  trace,  however  if  you  call  MongoTemplate  directly  youâll  get  proper  stack  trace.  âąâŻ If  HTTP  GET  request  contentType=applicaMon/json  then  response  will  be  rendered  as  applicaMon/x-Ââ spring-Ââdata-Ââcompact+json,  when  by  default  it  will  be  HAL  document  âąâŻ By  default  HVp  POST  request  set  contentType=text/plain  which  failed  to  be  transformed  into  enMty  âąâŻ hVp://wiki.fasterxml.com/JacksonFAQDateHandling  âąâŻ hVps://github.com/FasterXML/jackson-Ââdatatype-Ââjsr310  âąâŻ Mongo  doesnât  support  Java  8  Date  and  Time  types  naMvely  âąâŻ hVps://jira.spring.io/browse/DATAREST-Ââ524  âąâŻ hVp://stackoverïŹow.com/quesMons/25072967/spring-Ââxd-Ââredis-Ââmessage-Ââbus-Ââremoving-Ââheaders-Ââ from-Ââthe-Ââmessage  issue  with  headers  passing  in  Redis  âąâŻ SpringXD  doesnât  support  packages  starMng  with  âjavaâ))!!! Â
- 21. Links  âąâŻ Spring  -Ââ  hVp://spring.io  âąâŻ Zookeeper  -Ââ  hVp://zookeeper.apache.org/doc/trunk/zookeeperStarted.html  âąâŻ Hadoop  -Ââ  hVp://hadoop.apache.org/docs/r2.7.1/hadoop-Ââproject-Ââdist/hadoop-Ââ common/SingleCluster.html  âąâŻ XD  -Ââ  hVp://docs.spring.io/spring-Ââxd/docs/current-ÂâSNAPSHOT/reference/html  âąâŻ Boot  -Ââ  hVp://docs.spring.io/autorepo/docs/spring-Ââboot/1.2.0.M2/reference/ htmlsingle/  âąâŻ DataRest  -Ââ  hVp://docs.spring.io/spring-Ââdata/rest/docs/2.3.2.RELEASE/reference/ html/  âąâŻ Redis  -Ââ  hVp://redis.io/documentaMon  âąâŻ RestTemplate  -Ââ  hVps://spring.io/blog/2009/03/27/rest-Ââin-Ââspring-Ââ3-ÂâresVemplate  Â
- 22. Thank  you!  TwiVer:  @tsaltsol  Skype:  oleg.tsalko  GitHub:  olegts   Code  shown:  hVps://github.com/olegts/ SpringXDCustomTweetsTransformer  hVps://github.com/olegts/TweetsBootDataRestApp  Â