Next Generation Enterprise Architecture
Download as PPTX, PDF33 likes13,775 views
The document outlines the evolution and business implications of a new enterprise architecture called Zeta, emphasizing the need for dynamic resource allocation and integration of existing systems. It highlights the issues with traditional isolated workloads, aiming for improved resource utilization, business continuity, and streamlined processes. The Zeta architecture promotes real-time data processing and aims to provide a competitive advantage through effective resource management and operational efficiencies.
Next Generation Enterprise Architecture
- 1. The Next Generation Enterprise Architecture © 2014 MapR Techno©lo 2g0ie1s4 MapR Technologies 1 Resistance is Futile:
- 2. © 2014 MapR Technologies 2 Agenda • Current State – History – Moving Forward • The Next Enterprise Architecture • Business Implications • Concrete Implementations
- 3. © 2014 © 201 M4 aMpaRp RTe Tcehcnhonloogloiegsies 3 Current State
- 4. © 2014 MapR Technologies 4 Study History to Prepare for the Future • A data center was built • The servers were statically partitioned • If we want to break the cycle we have to break the partitions and become dynamic
- 5. © 2014 MapR Technologies 5 Understanding the Why’s • Isolation of resources – Assists in troubleshooting – Prevents the analytics team from impacting production • Maximum throughput of an application – Guaranteed volume (maximum): compute, memory and storage • Business Continuity – We know exactly what is backed up, when, and where – Difficult to perfect and to test
- 6. © 2014 MapR Technologies 6 Issues with Isolated Workloads • Segregated servers lead to under utilized hardware – Wasted capacity and energy • Complicated processes to move data to the required processing servers – Operational impact, including extra monitoring – Time delays moving data (not real-time) – Troubleshooting time when there are issues • Difficult to thoroughly test DEV vs. QA vs. Production – Environments have different shapes and sizes – They will not have identical configurations
- 7. © 2014 MapR Technologies 7 Goals Moving Forward • Leverage all existing hardware • Create isolation in a different way • Improve production operational processes • Fix process of moving from DEV to QA to Production • Support real-time business continuity
- 8. The Next “Last” Enterprise Architecture © 2014 © 201 M4 aMpaRp RTe Tcehcnhonloogloiegsies 8
- 9. The Next Generation Enterprise Architecture Enterprise Applications Global Resource Management © 2014 MapR Technologies 9 • Dynamic compute resources • Common storage platform • Real-time application support • Flexible programming models • Deployment management • Solution based approach • Applications to operate a business * This is a pluggable architecture Distributed File System
- 10. © 2014 MapR Technologies 10 Technologies That Work Web Servers Business Applications Enterprise Applications Global Resource Management Mesos + YARN Distributed File System MapR-FS S3 HDFS
- 11. We Will Call This Architecture… © 2014 MapR Technologies 11
- 12. © 2014 MapR Technologies 12 What’s in a Name • The letter Z is the last letter in the English alphabet, but Zeta is not the last letter of the Greek alphabet – But this is the last generalized architecture you will need. • Sixth letter of the Greek alphabet – Hexagon represents the 6 surrounding pieces • Zeta represents the number 7 – 7 total components in this architecture – Components work with a global resource manager
- 13. © 2014 MapR Technologies 13 Origin Story of the Zeta Architecture • Cultivated by Jim Scott – Created the pretty diagrams – Put a nice name on it – Documented the concepts • Not really a new concept – Google pretty much pioneered these technology concepts – They have never really discussed it cohesively in this way
- 14. © 2014 MapR Technologies 14 Zeta Architecture at Google HTTP Servers Enterprise Applications Global Resource Management Borg & Omega Distributed File System GoogleFS GMail
- 15. © 2014 © 201 M4 aMpaRp RTe Tcehcnhonloogloiegsies 15 Concrete Implementations
- 16. © 2014 MapR Technologies 16 Web Server Logs • Web server generates logs • Land on local disk – Logs periodically rotated • Shipped to other servers • Run jobs on logs
- 17. © 2014 MapR Technologies 17 Web Server Logs • Web server generates logs • Land on DFS – Logs still rotate – Logs now tolerant of a server failure prior to rotation – Logs are instantaneously available for computation • Run jobs on logs – Data locality
- 18. © 2014 MapR Technologies 18 Advertising Platform
- 19. © 2014 MapR Technologies 19 Advertising Platform - Simplified
- 20. © 2014 MapR Technologies 20 Advertising Platform on Zeta
- 21. © 2014 © 201 M4 aMpaRp RTe Tcehcnhonloogloiegsies 21 Business Implications
- 22. © 2014 MapR Technologies 22 Integration of Existing Systems • Use standards like NFS to connect existing systems • Pluggable security models fit into your companies current standards • Not everything works well in this model – Oracle, DB2, SQL Server, PostgreSQL, MySQL • They tend to not support being resource managed, containers or other DFS • Applications in this architecture can still use them • If they start supporting these technologies then things change
- 23. © 2014 MapR Technologies 23 Rethink the Data Center • All Servers – Run Mesos – Participate in the Distributed File System • Dynamic Allocation of Resources – Spin up more web servers – Custom Business Applications – Big Data Analytics • Data Locality – No more shipping data – Store and process the data where it was created
- 24. © 2014 MapR Technologies 24 Simplified Architecture • Less moving parts – Less things to go wrong • Better resource utilization – Scale any application up or down on demand • Common deployment model (new isolation model) – Repeatability between environments (dev, qa, production) • Shared file system – Get at the data anywhere in the cluster – Simplifies business continuity
- 25. Production Research © 2014 MapR Technologies 25 Business Continuity • Resilience – Redundancy – High Availability – Spare Capacity • Recovery – Snapshots – Disaster Recovery • Contingency – Protect against the unforeseen – Multisite Capability Production WAN Datacenter 1 Datacenter 2 WAN EC2
- 26. © 2014 MapR Technologies 26 Platform-wide Security and Compliance • Authentication, Authorization, Auditing – Users and jobs – All tiers • Data protection – Wire-level encryption between servers – Masking • Regulatory Compliance – Automatic expiration of “old” data – Data locality supported by distributed file system
- 27. © 2014 MapR Technologies 27 Net Benefit • Reduced operating expenses (OPEX) – Better utilization of available capacity and data center space • Reduced capital expenses (CAPEX) – Less total hardware needed • Improves time to market – Streamlined deployments – Environments become consistent and predictable • Delivers a competitive advantage – Via platform scaling – Performance improvements
- 28. © 2014 MapR Technologies 28 Recap • Saves valuable time and money • Enables stronger business continuity capabilities • Google has been doing this for years – Real-time is the crux of everything Google does • Time for the rest of us to operate at Google scale – The technologies are there and they play together nicely – Process changes must occur internally to achieve this architecture • This approach will become the “traditional” way of thinking – Don’t get beat to it by your competitors
- 29. Go Forth and Implement the Zeta Architecture © 2014 MapR Technologies 29
- 30. © 2014 MapR Technologies 30 Q & A Engage with us! @kingmesal maprtech jscott@mapr.com MapR maprtech mapr-technologies
Editor's Notes
- #5: Static partitioning was used to create isolation. This enabled people to know that when a problem occurred what was causing the issue. This is something containers can fix. Additionally, when we statically partition, we cannot fully leverage all the resources for all the most important work. Most server types are busy at different times of day, or can be. Which work is the most important? Whichever is deemed the most important now!
- #6: The problem with this business continuity story is that you are still limited to generally scheduled backups and not real-time.
- #10: Pluggable, because we need an architecture that platforms can be modeled after. If we have to rethink an enterprise architecture every time the next greatest thing comes out we end up redoing a lot of work.
- #11: These technology concepts are now mature enough that they will stick around and the specific implementation is flexible.
- #13: NOTE: The solution architecture can integrate remote platforms and is a conceptual model and thus isn’t necessarily managed by a global resource manager
- #14: “Google is living a few years in the future and sends the rest of us messages” –Doug Cutting, Hadoop Founder
- #17: Server crashes before a file is rolled and the data is lost.
- #18: Leveraging the distributed file system means that the data is NOT lost. Scales without the pain of figuring out how to move that data, the platform handles it for you. From the solution architecture perspective this is likely going to be utilized in monitoring the operations of an environment, or perhaps revenue management.
- #20: These parts are expensive. They are difficult to test because production is often drastically different from dev and qa.
- #21: NOTE: The billing database is an external entity, likely an RDBMS and is still a part of the solution architecture to deliver the business objectives
- #23: As pointed out in the advertising example. The billing system is still in an RDBMS. It still integrates via the solution architecture, but the RDBMS is not running on this platform architecture.
- #29: The benefits are plentiful They rely heavily on Borg and Omega They perform 2 billion container deployments per week
- #31: Follow me on twitter to get updates for documentation on the Zeta Architecture