7450A - CRONOS helping ENGIE adopting Private Cloud with Bluemix Local System
- 2. Agenda • Who is Cronos • Client & Project Context • Solution • Technical Challenges • Conclusion & Lessons Learned
- 3. The Cronos Group - fact figures +4500 People +800 New hires +300 Competence centers +450M Revenue 18% Yearly Growth International
- 5. iBizz fact figures +70 People +40 New hires 2018 8 Competence centers 5M Revenue 100% Yearly Growth IBM rev 3,2M
- 6. COMARK GROUP WAVE Integration Designers OPTIS CRAFTWORKZ OSWALD IBIZZ – anno 2017 I8C IS4U iPerform Portico A unique combination - the power of open source, the solutions of IBM and innovative minds Optis JAVA & JAVAScript, DEVOPS, MBF, PURE ID comark craftworkz groupwave Integration Marketing automation prototyping as a service, AI Messaging & Collaboration IBIZZ IBM licences THELEDGER. TheLedger. Blockchain RACCOONS RACCOONS IBM WATson Portico IBM DX
- 7. Agenda • Who is Cronos • Client & Project Context • Solution • Technical Challenges • Conclusion & Lessons Learned
- 8. The Client • ENGIE / ELECTRABEL • is the largest producer and supplier of electricity in Belgium; • sells electricity, natural gas, energy products and energy services to corporate and consumer clients. • Business Unit = IS M&S Belux • Information System Marketing and Sales BeNeLux
- 9. Project Context • The Belgian energy market is about to evolve remarkably in 2018. • new federal clearing house • new energy market process model MIG 6 • significant impact on the data exchange and the market processes structuring the electricity and gas market today • Goal of MIG6 • simplify the data exchange between participants in the energy market • incorporate the latest technologies, such as the availability of smart meters and decentralized production.
- 10. Project Context • “Enterprise Application Integration” platform (EAIv2) • Key integration platform in energy market data exchange • Critical with growing importance • EAIv2 infrastructure and application architecture • IBM System p / AIX based • WebSphere Message Broker v7 and WebSphere MQ v7 • Single server architecture • Existing EAIv2 platform not suitable for MIG6 • MQ/IIB version lack features • Not fitting future SLA requirements • Not able to support fast/agile development cycles incl. server deployments
- 11. Business challenges • Early availability of platform to support development for MIG6 • Server infrastructure • Upgrade WebSphere Message Broker v7 and WebSphere MQ v7 to version IIB v10 and MQ v8 • Need for new business monitoring solution based on the ELK stack (Elasticsearch, Logstash and Kibana) • The implementation of IBM PureApplication System as private cloud platform for ENGIE M&S • Private cloud platform needs to be managed
- 12. Project Approach • A IBM private cloud solution was chosen by ENGIE • Faster time to market for new project (independent from ENGIE IT) • Lowering overall TCO (running cost / infra / licenses) • Autonomous decisions (self service …) • To support a new agile operating model • Cronos was chosen as a partner to • Provide 24/7 managed services to operate the private cloud • Strong commitment from IBM • Assistance with architecture design and performance tuning • Onboarding of skills from IBM Cloud Technical Engagement
- 13. Agenda • Who is Cronos • Client & Project Context • Solution • Technical Challenges • Conclusion & Lessons Learned
- 14. What is a Private Cloud? Private cloud is a type of cloud computing that delivers similar advantages to public cloud, including scalability and self-service. Unlike public clouds, a private cloud is dedicated to a single organization and provides higher control and security. Definition
- 15. On a Pre-Integrated System POWER or x86 On Your Own Infrastructure Bring your own hardware On SoftLayer Available in SoftLayer data centers worldwide A hybrid cloud application platform for cloud enabling applications and middleware with enterprise grade qualities of service Off-Premises On-PremisesOn-Premises PureApplication Platform Family
- 16. Virtual System Patterns in PureApplication • Virtual System Patterns are a simple mechanism to quickly and consistently deploy infrastructure • Virtual Machines • Operating System • Software Components • Script Packages • IBM provides a rich catalog of Patterns for IBM software • Patterns can be extended and modified by Script Packages
- 17. PureApplication System Private Cloud Platform at ENGIE • Pre-integrated hardware includes • Model W2500-32 • 32 Intel Xeon Ivy Bridge Cores • 1024 GB of memory • IBM V7000s with 43 TB capacity • Pre-integrated software includes • PureApplication Software • Red Hat Enterprise Linux • VMWare ESX and vSphere • IBM Tivoli Monitoring • IBM Spectrum Scale • IBM WebSphere Application Server and WebSphere Liberty • IBM DB2 BLU • Today IBM sells IBM Bluemix Local System as the equivalent • W3500 iSCSI • W3550 Flash
- 18. Solution outline • Datacenter services • Dual dataroom setup • IT services • Networking • LAN/VLAN • Firewall • NTP, DNS, SMTP • LDAP • Outbound proxy • Storage: NFS • TieBreaker VM’s • GPFS • Multi-system subdomain Site 1 Site 2 Datacenter Services
- 19. Solution outline • PureApplication provides automation through Shared Service to install and configure • Red Hat Enterprise Satellite Server • IBM Tivoli Monitoring • IBM Spectrum Scale (GPFS) Datacenter Services PureApplication Shared Services Site 1 Site 2
- 20. Elasticsearch, Logstash and Kibana (ELK) Datacenter Services PureApplication Shared Services IIB / MQ Workload ELK Workload Site 1 Site 2
- 21. Elasticsearch, Logstash and Kibana (ELK) • Non IBM workload • No pattern available • Kick started during bootcamp • Virtual System Pattern • Deploy Virtual Machines • Install RHEL 7.2 • Add additional storage • Install and configure ITM for Linux OS agent • Ansible • Install and configure ElastichSearch, Logstash and Kibana • Install and configure Agent Builder ITM custom agent Kibana and Elasticsearch pattern
- 22. IBM Integration Bus and IBM MQ External shared services PureApplication Shared Services IIB / MQ Workload ELK Workload Site 1 Site 2 Site 1 Site 2
- 23. IBM Integration Bus and IBM MQ • IBM workload • IIB and MQ pattern available • Virtual System Pattern • Deploy Virtual Machines • Install RHEL 6.8 • Install and configure GPFS client, MQ and IIB • Install and configure ITM for Linux OS agent • Install and configure ITCAM for IIB agent • Ansible • Install and configure ITCAM for MQ agent (workaround) IIB and MQ development pattern
- 24. Agenda • Who is Cronos • Client & Project Context • Solution • Technical Challenges • Conclusion & Lessons Learned
- 25. Technical Challenges - IBM Tivoli Monitoring • IBM Tivoli Monitoring can automatically be installed and configured • Agents are automatically installed and configured for: Linux OS, IBM MQ, IBM Integration Bus, etc • Licenses are included • We established a simple mechanism to monitor ELK • ITM supports building and installing a custom agent using “Agent Builder • Fully supported by IBM on PureApplication • We had to work with IBM Support to get the following agents working • ITCAM for MQ agent • Agent Builder agent
- 26. IBM Spectrum Scale (GPFS) – Introduction • GPFS cluster • GPFS filesystem 1 • GPFS fileset 1 • GPFS fileset 2 • GPFS filesystem 2 • GPFS fileset 1 • GPFS fileset 2 • GPFS clients 1. Bootstrap and get authorisation from a GPFS manager (not shown here) 2. Mount one or more GPFS filesystem within the GPFS cluster 3. Get one or more GPFS fileset within these filesystems assigned 4. Optionally use a symbolic link to access each fileset within each filesystem for convenience • Note that a single GPFS client always has authorisation on all filesystems and their fileset!
- 27. IBM Spectrum Scale (GPFS) – external Tiebreaker best practices • Active-active GPFS cluster requires external tiebreaker • We used a RHEL 6.8 VM, refer to this link for supported OS versions • Each RHEL distribution release version supports a specific version of GPFS! • Version of GPFS in PureApplication varies as documented here • Version of GPFS installed and configured depends on version of GPFS pattern type and PureApplication firmware version • GPFS tiebreaker requires 256 MB of storage for each filesystem in the GPFS cluster as documented here
- 28. IBM Spectrum Scale (GPFS) – Filesystem Blocksize IO Type Application Examples Blocksize Large Sequential IO Scientific Computing, Digital Media,File based Analytics 1MB to 16MB Relational Database DB2, Oracle, Small Files on ESS 512KiB Small I/O Sequential General File Service ,Email, Web Applications 256KiB Special* Special 16KB-64KiB • GPFS supports a number of different blocksizes for its filesystems, as shown in the table here (source) • IBM GPFS Virtual Application Pattern allows the user to specify the blocksize for the filesystem • We found that blocksizes in excess of 1 MB were not working as expected and addressed this through APAR IT18456
- 29. IBM Spectrum Scale (GPFS) – Soft and Hard Quota for Filesets • GPFS on PureApplication requires the user to specify a “Storage Maximum Limit” size for each new GPFS Fileset • User can update both the “Storage Minimum Limit” and “Storage Maximum Limit” for each existing GPFS Fileset • Select Operations > GPFS-Manager > Change Fileset from Instance Console • Minimum denotes the “Soft Limit” for the fileset quota; this can be exceeded for up to 7 days by default. • Maximum denotes the “Hard Limit” for the fileset quota; this cannot be exceeded. • Sometimes users might prefer or require to avoid quotas altogether for their filesets • On PureApplication one cannot specify “0M” for the “Storage Maximum Limit” (APAR IT18904) • We found that Fileset Quotas do not apply for root by default • Option to enforce fileset quota for root to be delivered in 2.2.3.0 firmware
- 30. IBM Spectrum Scale (GPFS) – Understanding and Tuning Recovery Times • We performed extensive tests with GPFS clients when losing a GPFS node • Hard shutdown of a VM hosting a GPFS node as described here • GPFS client read/write operations can be temporarily paused but are ultimately successful • Number of singleton services run within GPFS cluster • Loss of Cluster Manager interrupts GPFS client read/write operations • Recovery time is up to 70 seconds by default on PureApplication • In a 2 node GPFS cluster the recovery time can safely be reduced to 20 seconds by using mmchconfig and set: • failureDetectionTime=10 (seconds) • leaseRecoveryTime=10 (seconds) • Contact IBM Support for guidance
- 31. Technical Challenges - VMWare • Bluemix Local System and PureApplication System come with VMWare ESX/vSphere 6.0 • Installed and configured behind the scenes • Licenses included • Solution for VMWare ESX 6.0 ”storageHeap” memoryleak • We were unable to deploy Virtual Machines with Block Storage Volumes • Confirmed by IBM Support to be caused by a known issue in VMWare ESX 6.0 • Patch made available by IBM Support to review the percentage used in “storageHeap” on all Compute Nodes (ESX 6.0 hosts) • Firmware version 2.2.3.0 includes APAR IT19336 , a more recent build of ESX 6.0 that is no longer vulnerable to this memory leak • VMWare ESX and vSphere 6.0 licenses • When installing Bluemix Local/PureApplication System you need to register licenses with VMWare • Once registered you need to install the license key on the system, effectively replacing the “not-for-resale” VMWare license that IBM ships out of the factory • Use the information here to confirm what licenses are installed WARNING! When the “not-for-resale” license expires, within 24 hours Compute Nodes (ESX 6.0 hosts) will go into ”quiesced” state! You can no longer deploy new VMs, nor move deployed VMs from another Compute Node!
- 32. Agenda • Who is Cronos • Client & Project Context • Solution • Technical Challenges • Conclusion & Lessons Learned
- 33. Conclusion • Private Cloud platform delivered for ENGIE M&S • Number of challenges have had to be overcome during implementation • Private Cloud is foundation for EAIv3 platform • Will limit interaction with traditional IT organisation within ENGIE M&S • Cronos operates this Private Cloud • Operations of platform on behalf of ENGIE M&S • Partner for ENGIE implement new patterns for new projects • Private Cloud allows for new ways for working for ENGIE M&S • ENGIE M&S can operate more or less independently (self-service model) • This includes new projects, proof-of-concepts, etc
- 34. Lessons Learned • Client underestimated complexity of new solution • Implementing a private cloud is a complex undertaking • Strategic one-time activity • Planning and design is crucial • Client • Cronos • IBM Cloud Technical Engagement team • Technical skills at the right time are key to success • Combination of IBM Support and IBM Cloud Technical Engagement delivered the results we needed
- 35. Questions?