AAI-3218 Production Deployment Best Practices for WebSphere Liberty Profile
3 likes2,485 views
This document provides best practices for configuring and deploying WebSphere Liberty Profile in a production environment. It discusses profile configuration, topologies and practices including standalone, collective, and z/OS integration. It also covers application deployment using server packages, build pipelines, and upgrades. Management configuration topics include high availability, security, dynamic routing, auto-scaling, and Admin-metadata. Finally, it discusses sizing considerations and when multiple collectives may be needed.
AAI-3218 Production Deployment Best Practices for WebSphere Liberty Profile
- 1. © 2015 IBM Corporation Production Deployment Best Practices for WebSphere Liberty Profile Erin Schnabel WebSphere Liberty Profile Lead Developer schnabel@us.ibm.com Christopher Vignola WebSphere System Management Architect cvignola@us.ibm.com
- 2. Topics • Profile Configuration • Topologies & Practices • Application Deployment • Management Configuration • Numbers and Sizes 2
- 4. Configuring Data Sources • Use top-level libraries • Allows multiple data sources and applications to share the same class loader <library id="DB2JCC4Lib"> <fileset dir="C:/DB2/java" includes="db2jcc4.jar db2jcc_license_cisuz.jar"/> </library> <dataSource id="db2" jndiName="jdbc/sampleDB"> <jdbcDriver libraryRef="DB2JCC4Lib"/> <properties.db2.jcc databaseName="SAMPLEDB" serverName="localhost" portNumber="50000"/> </dataSource> <application location="myApp.ear"> <classloader commonLibraryRef="DB2JCC4Lib"/> </application> 4
- 5. Configuring Data Sources: Connections • Connection pooling and timeouts • Reduce connectionTimeout to confirm that max pool size is being exceeded • Increase maxPoolSize if you see connection timeouts • If you don’t need two phase commit: • Use ConnectionPoolDataSource rather than XADataSource • Data sources can enlist in a global transaction as a one-phase resource even if they are not XA-capable. 5
- 6. Configuring Data Sources: Connection sharing • Set isolation level property instead of programmatically • Use isolationLevel property on datasource • Set in resource reference binding/extension • Declared isolation level allows for better matching/sharing of connections • Use containerAuthData instead of user/password • Prevents applications using res-auth=Application from accessing the data source with container credentials 6
- 7. Threading and Thread pools • Auto-tuning thread pool • Optimizes for executor throughput (measured every 1.5 sec) • Dynamically adjusts between coreThreads and maxThreads – Default coreThreads = (2*processorCores) – Default maxThreads = MAX_INT • Not usually necessary to tune the thread pool • Some workloads may warrant increasing coreThreads • Long-running / Outbound-to-self • Measure performance before and after adjustment 7
- 8. Separate App and Admin HTTPS traffic • Configure Virtual Hosts to isolate the application from other (internal/administrative) traffic. 8 <httpEndpoint id=“appEndpoint” … /> <!– Restrict access to default_host: only accessible via default endpoint --> <virtualHost id=“default_host” allowFromEndpointRef=“defaultHttpEndpoint”/> <!-- define an application-specific virtual host --> <virtualHost id=“applicationHost” allowFromEndpointRef=“appEndpoint”> <hostAlias>*:${app.http.port}</hostAlias> <hostAlias>*:${app.https.port}</hostAlias> </virtualHost> <!-- configure plugin to route to the app-specific endpoint --> <pluginConfiguration httpEndpointRef=“appEndpoint” /> server.xml: <virtual-host name=”applicationHost" /> Ibm-web-bnd.xml Virtual host binding required in the application
- 9. SSL Certificates and KeyStores • Use only officially signed SSL certificates • Use a separate keystore for inbound vs. outbound • Encode or encrypt passwords in your configuration <ssl id="defaultSSLConfig" keyStoreRef="defaultKeyStore" trustStoreRef="defaultTrustStore" /> <keyStore id="defaultKeyStore" location="${server.config.dir}/key.jks" type="JKS" password="{aes}..." /> <keyStore id="defaultKeyStore" location="${server.config.dir}/trust.jks" type="JKS" password="{aes}…" /> 9
- 10. Security - General • Harden your environment – extensive resources: http://www.ibm.com/developerworks/websphere/techjournal/1210_lansche/1210_lansche.html http://www.ibm.com/developerworks/websphere/techjournal/1303_lansche/1303_lansche.html • Avoid vulnerabilities – keep service current. • Register for support notifications at ibm.com/support . 10
- 12. Topologies • Standalone • collective Assisted Lifecycle • Collective
- 13. When to Use Which Topology? • Standalone – small, isolated, or situational deployments • collective Assisted Lifecycle • Small/Med deployments • Existing admin skill • multi-server QoS • Collective • Small/Med/Large deployments • Lightweight environment • Multi-server QoS 13
- 14. Universal Practices • Server Package Deployment • High Availability – Rule of 3 • Automated deployment • Secure environment 14
- 15. QoS Practices (Optional) • Dynamic Routing • Auto-scaling • z/OS integration 15
- 16. Gold Standard – HA Collective 16 Collective Controller Replica Set CC CC CC • Include AdminCenter • Use Dynamic Routing • Use Auto-scaling Machine Boundary AppServer AppServerLiberty Profile Per App IHS IHS IHS Per Collective
- 17. z/OS Integration • Use read-only mount point for servicing runtime • Use PROCs for running servers • Use START/MODIFY/STOP commands for server lifecycle • Use SAF Registry support • Exploit platform integration as necessary: • zosWLM • zosTransaction • zosLocalAdapters 17
- 19. Build & Deploy • Use server package for deployment – 3 models • Developers direct deploy • Developers hand off app, admin packages/deploys • Developer direct deploy w/packaging automation to ensure “approved server config” >> Establish build/deploy pipeline • Automate – use script, Chef, Urbancode, etc 19
- 21. Server Package Deployment: Managing Config 21 $WLP_USER_DIR/MyServer/ server.xml apps/ MyApp.war deploy (apply target overrides) $WLP_USER_DIR/MyServer/ server.xml configDropins/ overrides/ prod.xml apps/ MyApp.war
- 22. Server Package Update • Update through build process • Blue/Green deploy • Dual install locations • Ripple start (stop old/start new, JVM by JVM) • Delete old instance at your convenience 22
- 23. Upgrade – 1 of 5 23 host1.com http/s: 9080/9443 status: STARTED /wlp-blue/usr/servers/prod1 /wlp-blue/bin /wlp-blue/lib host2.com http/s: 9080/9443 status: STARTED /wlp-blue/usr/servers/prod1 /wlp-blue/bin /wlp-blue/lib
- 24. Upgrade – 2 of 5 24 host1.com http/s: 9080/9443 status: STARTED /wlp-blue/usr/servers/prod1 /wlp-blue/bin /wlp-blue/lib host2.com http/s: 9080/9443 status: STARTED /wlp-blue/usr/servers/prod1 /wlp-blue/bin /wlp-blue/lib host1.com http/s: 9080/9443 status: STOPPED /wlp-green/usr/servers/prod1 /wlp-green/bin /wlp-green/lib host2.com http/s: 9080/9443 status: STOPPED /wlp-green/usr/servers/prod1 /wlp-green/bin /wlp-green/lib
- 25. Upgrade – 3 of 5 25 host1.com http/s: 9080/9443 status: STOPPED /wlp-blue/usr/servers/prod1 /wlp-blue/bin /wlp-blue/lib host2.com http/s: 9080/9443 status: STARTED /wlp-blue/usr/servers/prod1 /wlp-blue/bin /wlp-blue/lib host1.com http/s: 9080/9443 status: STARTED /wlp-green/usr/servers/prod1 /wlp-green/bin /wlp-green/lib host2.com http/s: 9080/9443 status: STOPPED /wlp-green/usr/servers/prod1 /wlp-green/bin /wlp-green/lib
- 26. Upgrade – 4 of 5 26 host1.com http/s: 9080/9443 status: STOPPED /wlp-blue/usr/servers/prod1 /wlp-blue/bin /wlp-blue/lib host2.com http/s: 9080/9443 status: STOPPED /wlp-blue/usr/servers/prod1 /wlp-blue/bin /wlp-blue/lib host1.com http/s: 9080/9443 status: STARTED /wlp-green/usr/servers/prod1 /wlp-green/bin /wlp-green/lib host2.com http/s: 9080/9443 status: STARTED /wlp-green/usr/servers/prod1 /wlp-green/bin /wlp-green/lib
- 27. Upgrade – 5 of 5 27 host1.com http/s: 9080/9443 status: STARTED /wlp-green/usr/servers/prod1 /wlp-green/bin /wlp-green/lib host2.com http/s: 9080/9443 status: STARTED /wlp-green/usr/servers/prod1 /wlp-green/bin /wlp-green/lib
- 29. High Availability Services • Collective scope • Built-in to collective controller replica set • Host scope (e.g. scalingMember-1.0 feature) • Uses local port 29
- 30. Collective SSL • Collective root, member root • First controller establishes “true root” • Must be copied to subsequent replicas • Trust between • Controllers and members • Members and controllers • Members on same host • Certificates • Signers • Identity 30
- 31. Security • Use external user registry – e.g. • Ldap • SAF • Multiple registries allowed – e.g. as granular as per cluster. • Collective Controller Replica set must use same registry. 31
- 32. Highly Available Collective Controller • Three controllers minimum. Odd numbers only. • Up to 2000 member per controller Recommended: Max(Members/2000,3) +1(for controller failure) +1(for network partition) • Use configDropIns directory – it’s replicated! • Configure members with controller failover addresses. 32 CC CC CC
- 33. Dynamic Routing • Requires IHS (or Apache). • Use dynamicRouting-1.0 feature in controllers. • Double-layer IHS to simplify firewall management. • Terminate SSL at earliest opportunity. 33 CC CC CC IHS IHS IHS
- 34. Auto-scaling • Put scalingController-1.0 feature in at least 3 controllers. • Set hostSingleton port for vertical scaling. • Set min instances based on average demand. 34 AppServer AppServer AppServer Elastic Resources CC CC CC
- 35. Auto-scaling Policy • Leave headroom on max settings (e.g. • CPU <= 90% • Heap <= 90% • Memory <=90% • Include scaling policy in server package • scaling-metadata.xml 35
- 36. Use Admin-metadata • Owner • Contacts • Note • Tags • admin-metadata.xml – part of server package • assignable to host, server, application, cluster, runtime • There is also an API • Used in AdminCenter to search and set views 36
- 38. 38 Liberty Profile Collective Size Design Considerations • Collectives are design for large scale • What limits the size of a WebSphere Collective? • Breadth and currency for shared Information across controller JVMs • Communication and coordination of shared information • Product features scale differently with large collective size • Most affected : collectiveController, scalingController • It is possible to create a large topology collective But will it function to your requirements? • Collective works well with defaults settings. Some environments may require tuning.
- 39. 39 When do you need multiple collectives? • Isolation • Development vs testing vs production • Critical applications • Backup site • Lines of business • Each funding area may have different policies for when to apply fixes, when to upgrade • Geography: • Controllers can span data centers with qualifying* network and config • Members can span data centers with tuning • Large collectives require planning to avoid “urban sprawl”
- 40. 40 How Large a collective can I create? • No hard limit – trust, but verify • Lab tested: • Tested 10,000 members • 5 controllers • 50 VMs • ~200 members per VM • Controller VMs: 20GB memory+6 CPUs • Member VMs: 64GB+16 CPUs
- 41. 41 How Many Host OS Instances per collective • No Design Limit • Typical large topology up to hundreds of hosts. • Practical Limits • Operations may take longer: – Configurations, server deployment • Notifications flowing back to collective controllers • Load on controllers with concurrent operations • 50 hosts per collective controller guideline
- 42. 42 How Many Application Servers per Host OS? • Keep WAS JVMs completely within physical memory • Allow for overhead • Process footprint about 1.25 to 1.5X maximum heap size for 32 bit heaps • 1.6x to 1.8x for 64-bit heaps • If App Server gets swapped out, add more memory or else move to different host OS. • Ensure sufficient CPU is allocated • Especially for hypervisor • Avoid CPU starvation
- 43. 43 How Many Applications Per Server • Balance between resource usage and isolation • With one application per server • One bad application does not bring down all other applications • Easier to tune each application in isolation • With more than one application per server • Less resources: cost of application server runtime amortized across multiple applications • Smaller topology to manage • Configure as much isolation as you can afford • If you have 300 applications clustered on 3 nodes – Complete isolation 900 JVMs – Complete sharing 3 JVMs
- 44. 44 System Management Best Practices • Set and track your performance goals • Measure performance of your most commonly used operations • Track changes as you increase collective size • Track changes over time to identify new issues • Use Scripting (or your own Java framework) • Automatable, repeatable, testable • Don’t overload collective controllers • Give enough memory + CPU • Run AdminCenter in multiple controllers to spread load • Don’t co-locate with resource intensive processes, e.g., application servers with heavy load
- 45. Wrap Up
- 46. Covered • Topologies & Practices • Profile Configuration • Application Deployment • Management Configuration • Numbers and Sizes 46
- 47. Notices and Disclaimers Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.
- 48. Notices and Disclaimers (con’t) Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. • IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
- 49. Thank You Your Feedback is Important! Access the InterConnect 2015 Conference CONNECT Attendee Portal to complete your session surveys from your smartphone, laptop or conference kiosk.