CTE Phase III
Download as PPTX, PDF1 like738 views
The document provides an overview of Phase III of the Continuous Transformation Environment (CTE) implementation, including technical goals, focus areas, use cases, and next steps. Key points include: - Phase III goals are to establish a large-scale testbed capable of emulating commercial cloud systems at scale and integrating high-speed networking. - Initial focus areas are the CTE lab infrastructure, connectivity to satellite facilities, and ingesting initial technology partners. - Use cases demonstrate capabilities like rapid private cloud deployment, large-scale video analytics, social media command centers, and advanced data analytics visual tools. - Next steps are to continue integrating systems, implementing high-speed interfaces, and establishing complex application provision
CTE Phase III
- 1. Continuous Transformation Environment Phase III Implementation, Prototypes, and T&E-based Use Cases
- 2. Overview • Phase III Technical Goals • Initial CTE Technology Focus Areas • Overview of the CTE Technology ingest process • Use-Case Approach to T&E • GPU Investigations • Current CTE Use Cases • Next Steps 2
- 3. Phase III Technical Goals • Establish a large-scale testbed capabilities that enable emulation of commercial and cloud systems at scale – Implement a hybrid public/private/multi-tenant cloud – 1024 public IP addresses – enough to support the largest computing system built so far • Begin the CTE system integration with high-speed interfaces – All the systems have 1G interfaces, will implement 10G, 40G Ethernet and Fibre Channel by the end of Q3 FY2014 • Integrate: – All the systems into the central management, provisioning, and configuration control services – Machine provisioning with software defined networking (SDN) • Implement the ability to provision virtual and physical machines with complex applications and databases pre-loaded • Enable a vastly simplified complex system/architecture implementation-integration-interoperability capability 3
- 4. Initial CTE Technology Focus Areas • Establish the central CTE lab infrastructure at the QTS Richmond facility • Support connectivity and integration with the satellite facilities • Demonstrate high performance facility integration • Ingest an initial set of technology partners to explore a variety of cloud computing infrastructure hardware and software • User focused demonstrations 4
- 5. CTE Technology Ingest Process • Technology provider interviews – Organization and key person assessment – Technology overview, typical use cases, existing or envisioned customer base – Standards compliance – Open Systems/Open APIs • Use Case Mapping • Modifications and/or Interoperability Implementation • Component/System Integration • Use Case Test/Demonstration 5
- 6. Use-Case Approach to T&E • Isolated technology tests are of limited value • Test both the technology/service and the providing organization • Provide quantifiable value to participants – Metrics of various types – Reference third party installation and test – Neutral test, integration, and demonstration capability – New partners and business opportunities 6
- 7. GPU-based Investigations • HP-nVidia Partnership Testing • Video and Imagery Analytics • Computational Astronomy • Computational Cryptography • Computational Neuroscience • Cloud-based Robotics Control and Management • Immersive Stereoscopic Visualization
- 8. 8 CTE Prototypes and Use Cases Number Name Sponsors Short Description CTE 13-1 DIY Cloud Infrastructure: Rapid Deployment of a Private and Public Cloud with Amazon-style Ease-of-Use and Accounting HP, Convergence HP Demonstration of the Cloud Matrix for IaaS, Virtual Desktop demo with Convergence user devices CTE 13-2 Intercontinental, Large-scale VIDINT Command/Analysis Center: Distribution, Storage, Full Field of View Tracking, and full video and image stream search with arbitrary image(s) Pixia, Aspera, Signal Innovations, pixLogic, Flexanalytics, Cloud Front Group, L3, HP, M3 Com, Ciena, Infinite Dimensions, Terramajic Show full field of view tracking and analytics demo with metadata transfer to Autonomy advanced analytics CTE 13-3 Tapping the Firehose: Large-scale Social Media Analytics and Command Center InTTENSITY, Optensity, Infinite Dimensions Working with data sources such as the main Twitter feed is challenging. A social media command center will be demonstrated, providing insight into a variety of topics. CTE 13-4 Making Big Data Pay: Visual Programming of Advanced Analytics and Manipulation of Data Optensity, Flexanalytics, Pixlogic Demonstrate an easy-to-use visual programming tool for advanced analytics. Integrate Flexanalytics/Pixlogic video analytics. CTE 13-5 Beyond the OODA Loop: Collect-Correlate-Alert-Act (C2A2) Attivio, Ringtail A demonstration of decision data is drawn from multiple sources, correlated, and alerts generated and sent for actions to be taken in the context of shipping port management and security. CTE 13-6 Multi-INT Enterprise Service "Dial tone" and Collect-Correlate- Alert-Act MarkLogic, Ringtail A multi-INT scenario illustrating the Collect-Correlate-Alert-Act (C2A2) paradigm. CTE 13-7 Next-Generation Cloud-based C4ISR Visualization Ringtail A next-generation command center, used by the Chairman of the Joint Chiefs of Staff , in terms of both software and hardware. CTE 13-8 Big Data Storage and Analytics Oracle, Platfora, MapR, Cloudera, Hortonworks, Sqrrl, IBM, Security First Data cloud protection, visual tools and analytics for Hadoop-based systems in multiple facilities CTE 13-9 Object Storage and Analytics Hitachi Large scale object store and manipulation CTE 13-10 iCommand Dell, AAI, Textron Next Generation Command Center using Chairman of the Joint Chiefs of Staff visualization system CTE 13-11 Video Management KSI Video Social Media System for Video and Image Analysis CTE 13-12 Asteroid Detection/Computational Astronomy UN Space Programme II, NASA, ESA The CTE supported an international team in an asteroid detection contest. The CTE team discovered a previously unknown Main Belt asteroid, a rare achievement. CTE 14-1 Large-scale Immersive Training and Evaluation Private sector investors, 5 NFL Teams, OSD P&R Integrate government founded software development kits (SDKs), such as the University of Southern California Institute of Creative Technology's Virtual Human Toolkit, and create a next-generation immersive training environment, with improved SDKs. CTE 14-2 COTS Test and Evaluation NRO and others Formally test COTS technology and provide evaluations to interested government agencies. NRO has started working with the CTE to evaluate the high-speed message acceleration hardware employed by large-scale stock and bond trading organizations. CTE 14-3 Bitcoin Mining: Computational Cryptography CTE Finding bitcoins, a process refered to as bitcoin mining, is an exercise in computational cryptography. The CTE is strtudying the algrothms and technology associated with bitcoin mining as community innovations in gryptography, and real experience with bitcoins. Bitcoins are used by criminals and others who wish to obscure financial transactions. CTE 14-4 Automating Identification FBI, Verizon, Infinite Dimensions Efforts such as the FBI's Next Generation Ientification (NGI) program need to utilize a multi-source data to ID a variety of entities and concepts. This project evealuates and integrates technology to provide insight into promising technologies that are relevant to this task.
- 9. DIY Cloud Infrastructure • Show how easy it was to integrate various cloud systems through a single and unified provisioning and cost accounting system • Provision both virtual and physical machines (for performance) • Permits a better match between task and execution architecture • Eliminate vendor lock-in 9
- 10. Features • Infrastructure-as-a-Service (IaaS) for private and hybrid cloud environments • Self-service infrastructure portal for quick auto- provisioning – Integrated billing • Support of heterogeneous environments – Cloud-bursting to a variety of public cloud providers – Supported hypervisors include VMware, Microsoft HyperV, Red Hat KVM, and Integrity VMs 10
- 11. Large-Scale VIDINT, IMINT “Netflix for the DoD/IC” • Used the software transport software used by Netflix to move the video and imagery • Managed video and imagery using Pixia’s system • Employed Project Guppy’s long-distance, high performance networking – Aspera’s London lab to QTS Richmond • Integrated: – Video and imagery analytics on ingest using Signal Innovation Group’s software to track all moving elements of a video or image stream – Video and imagery search using pixLogic’s system • Integrated into HP Autonomy’s correlation and alerting engine 11
- 12. Performance Metrics • No observable latency in the transmission – Phase III hypothesis: Project Guppy team can provide affordable high performance transoceanic circuits that can deliver high volume and velocity content anywhere • Days for the key integrations: – The Pixia, SIG, and pixLogic integrations were facilitated via STANAG 4559, NATO Standard Image Library Interface – Open API’s facilitated the integration into HP’s Autonomy – Phase III hypothesis: Other video and image analytics, correlation, and alerting engines can be easily integrated 12
- 13. Large-Scale Social Media Analytics and Command Center • Integrates full Twitter Firehose, Facebook, etc. • Data is injected into a real-time ingestion and orchestration engine that allows creation of distinct processing pipelines to filter • Performs specific Natural Language Processing (NLP) based enrichment of the social media data of interest 13
- 14. Visual Programming of Advanced Analytics – Video Analytics Integration • Rapidly compose and execute data analytics across multiple clouds • Graphical composition for analytic workflows, virtualizes the composed workflows (or Apps) • Executes the Apps across multiple clouds • Integrated with piXlogic – Provides visual search solutions that automatically analyze and index the contents of images and video files 14
- 15. Beyond the OODA Loop Collect-Correlate-Alert-Act (C2A2) • Integrated Tibco’s Spotfire analytics capability with Ringtail’s Common Operational Picture visualization use by the Chairman of the Joint Chiefs of Staff (CJCS) – Integrated open source data about ship movement between and in ports to Spotfire – Correlation calculations triggered Alerts in Spotfire – Alert messages from Spotfire sent to the CJCS Command System 15
- 16. Performance Metrics • Spotfire integrated and collected multiple open source data feeds into the correlation process • The Spotfire-CJCS COP integration was performed in less than one week • The Java Message Service (JMS) was leveraged to provide interoperability • Open APIs also facilitated integration 16
- 17. Multi-INT Enterprise Service "Dialtone" and C2A2 • Experimented with the notion of an enterprise service registry • Implemented via the Representational State Transfer (REST) construct • Integrated multiple data sources and presented them to an enterprise search function – Both unstructured and structured – Demonstration employed open source government data 17
- 18. Next-Generation Cloud-based C4ISR Visualization • A more complete demonstration of Ringtail’s Common Operational Picture visualization use by the Chairman of the Joint Chiefs of Staff (CJCS) • Data and capability integration and interactivity are supported via: – Well-defined JMS-based messaging – REST interface support • Video and collaboration integration to overall package was interesting 18
- 19. Big Data Storage and Analytics • Showcased Data Cloud Protect participants • Included Hadoop vendors and analytics partners • Provided a venue to educate the audience on the operational issues • Highlighted initial collaborative efforts and results 19
- 20. Object Storage and Analytics • Installed and implemented Hitachi Data Systems prototype Object Store highlighting: – Multiple Entry Points: The system must allow for multiple independent applications simultaneously performing operations such as read and write. – Global Namespace: An object store should present a global namespace (GNS) to the client. – Access Protocol: The access protocol should work equally well over a WAN (such as the Internet) as over a LAN. • Therefore, the access protocol cannot be chatty, as most network file system protocols tend to be • Further, it should support mobile devices, such as smartphones, as clients – Unstructured Data: The system must be designed to optimize for unstructured data as this will be the predominant type – Hardware Agnostic: Hardware changes frequently. This includes servers, storage subsystems, and even the storage medium 20
- 21. Social Media and Video Management • Cloud-based video and associated “Big Data” management services • Focus is video/data management and sharing from “Remote Sensing Platforms” (e.g., unmanned aircraft, boats, underwater vehicles) • Social media-like interface and approach to content management, and analytics integration 21
- 22. VIDINT Command Center • COTS-based approach to a Video Intelligence Command Center • Focused around cameras and facility physical security • Integrates analytics capable of license plate recognition (LPR) with the camera feeds – Employs commercial video formats and interchange standards 22
- 23. Summary • Covered quite a variety of systems, both hardware and software • Managed to establish many noteworthy performance metrics for partners to tout • Illustrated the value and use of standards, open systems, and agile development methods • Set the stage for Phase III and expanded integration of the CTE infrastructure 23
- 24. Background
- 25. Continuous Transformation Environment Overview and Current Direction “A Cloud Computing Range”
- 26. • CTE Objectives – Prototype development and experimentation: Innovation in a collaborative environment by industry, government, and academia in an open systems collaborative environment – Integration, verification, test, and release of Commercial-Off-The-Shelf (COTS) products for government use – Rapidly prove operational utility of high technology solutions – Open systems and standards compliance evaluation, documentation, and capabilities matrix • Solve GAO identified big integrator problem1 Organizing principle: “Give innovation a chance.” • Consortium of large and small technology companies and facilities partners CTE at a Glance 261 Government Accounting Office-09-326SP, http://www.gao.gov/new.items/d09326sp.pdf
- 27. CTE Network 27 Quality Technology Services (QTS) and Verizon (VZW) are the current facilities providers OCONUS Sites are VZW CONUS VZW: Engelwood, CO; Culpeper, VA, Miami, FL Remaining CONUS Sites are QTS, in particular the 1.3M sq. ft. Richmond, VA site
- 28. Main CTE Experimentation Lab Location Richmond-High Density Multi-DataCenter Campus 500,000 Sq Ft of Planned Raised Floor Multiple Distinct Data Center Buildings (Current Basis of Design) 1 3 2 Office Space 1.3 Million Sq Ft Campus
- 29. Use Case Scenarios 1. Industry product for government use 2. Government product for industry use 3. Industry/government collaboration for government use 4. Standards Evaluation for government and/or standards body 5. Industry Compliance and Confidence testing 6. Government interoperability testing of industry products to select replacements or build partnerships 7. Government operational evaluations/ttp development of cloud solutions 8. Discovery and reuse through collaboration of communities of interest
- 30. Use Case Scenario Development • Find Opportunities from Suppliers and Users • Develop and Evaluate Opportunities • Supply Confidence/Knowled ge to Consumer • Field and Use
- 31. Government Opportunities • Resource seed prototypes, enabling rapid industry driven cloud solutions to DoD problems • Establish DoD CRaDA with the CTE that enables mutual benefit in: – Open System, Standards, and Content – Compliance testing results/methods – Open system industry demonstrations for DoD consideration – Open Government data/systems for industry consideration • Establish Open Source/Open Systems Charter foundation for DoD, including repository functions 31
- 32. Mutual Benefit Industry/CTE Gain • Direct Rewards to Innovators • Provide the Opportunity for Agile Partnerships • Access Government Solutions/Information (e.g. GOTS Software) • Improve Organizational Focus and Strategy Execution • Support Ongoing Innovation Government Gain • Implement Agile Development and Contracting Strategies • Speed New Applications and Systems from Concept to Use • Enhance Efficiency in Fielding and Support • Outpace Emerging Threats • Optimize Resource Employment
- 33. Way Forward/Next Steps • Invite Government Leaders back to CTE for workshop – November (OSD, Joint Staff) • Develop Proposal for Prototypes – December (CTE) • Develop Draft CRaDA – December (OSD, CTE) • Develop White Paper on Open System/Open Source Function – January (CTE, OSD) • Seed into field activities (AF ACC, AF/Joint Training, TRMC S&T) – May (CTE, OSD) 33
- 34. Summary • Continue to evolve the capacity and capability of the CTE with industry partners • Form government partnership to create a gateway for government-industry collaboration • Establish a series of rapid prototypes for DoD and non-DoD solutions • Demonstrate open systems/open source expertise 34
- 36. • QTS • HP • Hitachi Data Systems • Verizon • Oracle • NetApp • Solace • Ciena • IBM • ISC8 • Quantum • Spectralogic • Blackridge • EMC • Nicera • Juniper 36 CTE Infrastructure Access to over: 50,000+ Processing Cores 100,000 reserve 100+ Petabytes of Storage Over 30 Sites Worldwide Classified/Unclassified Facilities And Growing… Main facility in Richmond, VA: 1.3M square feet 110MW of power from 2 grid elements Full onsite backup & UPS 25,000 tons of chilled water generation capacity per day
- 37. CTE Software Participants • Cloud Infrastructure – Amazon, Dell, EMC, HP, IBM, Oracle, Quantum, Red Hat, Aspera • Data Storage and Manipulation – Accumulo, Cloudera, Hortonworks, MapR, Sqrrl, Vertica/HP • Document Manipulation and Analytics – HyLighter, IxReveal, Digital Reasoning • Video Management and Analytics – Pixia, SIG, Aspera, Pixlogic, Flexanalytics, Terramajic, KSI Video • Advanced Analytics Tools – AgileX, HP, Attivio, IBM, Optensity, Oracle, Solace • Security – Blackridge, CyberPoint, IBM, ISC8, Red Hat, Security First, Verizon 37
- 38. Current Technical Emphasis • Complete the CTE System Integration with High- Speed Interfaces – All the systems have 1G interfaces, will implement 10G by the end of Q1 FY14 • Integrate all the systems into the central provisioning service • Implement the ability to provision virtual and physical machines with complex applications and databases pre-loaded • Integrate machine provisioning with software defined networking (SDN) • Enable a vastly simplified complex system/architecture implementation capability 38