Mapping presentation THAG big data from space
Download as PPTX, PDF0 likes394 views
The document discusses the capabilities and projects of Outsourcing Partner in the field of big data, particularly in collaboration with the European Space Agency. It highlights several successful projects across various sectors, including telecommunications and retail, showcasing the technologies used and the outcomes achieved. The document also raises points regarding the need for security policies, interoperability, and the unlocking of data potential in collaboration with industry stakeholders.
Mapping presentation THAG big data from space
- 1. Delegation / Organisation Logo Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 1 Big Data From the Space 2017 Cycle 1st Mapping Meetings Outsourcing Partner Sp. z o.o. Bartosz Szkudlarek Piotr Zaborowski
- 2. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 2 We are Outsourcing Partner, a technology company, specialized in custom software development and Big Data. Outsourcing Partner capabilities on Big Data
- 3. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 3 What can we bring? Proven technology experience with common Big Data technologies. Outsourcing Partner capabilities on Big Data
- 4. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 4 Outsourcing Partner capabilities on Big Data Our experience Six projects in Big Data domain, which use Hadoop, Apache Spark and other technologies. Two projects for ESA where the point was to integrated and visualize massive data.
- 5. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 5 Outsourcing Partner capabilities on Big Data Project name Project subject Technologies Numbers European Space Agency GEOSS Web Portal Data hub portal with search functionality. Objective of this project was to integrate two different data sources on one visualisation platform HTML5, maps, microservices More than 1 mln resuls Two different data sources. European Space Agency The EO Web – the new website Proof of concept for new content architecture of new Earth Observation website which collects all information from domain services. The primary purpose of this project to identify and unify content elements from all EO websites and to provide efficient mechanism for harvesting, indexing, categorising and searching content. HTML5, Elastic Search, Kibana, Google Analytics More than 50 websites with technical documentation about missions instruments and other information connected with the area, over the 500k resources identified. Operational, constant dev Proof of concept Operational, complete
- 6. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 6 Outsourcing Partner capabilities on Big Data Project name Project subject Technologies Numbers Telecommunication sector T-Mobile Messaging broker Communication exchange between operator and customer is crucial. We implement communication broker for text messages (SMS, push notifications, etc..) which allows to monitor: • message efficiencies (how many reminders are needed for force user to pay delayed payments, what message force user to buy additional internet limit), • message rules ( the system can not send information about available internet package if user order package though any channel). Casandra, Apache Hadoop The system handled 15 mln customers, 3 mln message per day. Telecommunication sector T-Mobile Customer self-service system To provide services for customers, the telecommunication company needs to have many backend systems to support operations. The aim of this project was to implement the mechanism for collecting information about user activities in one repository. Except massive amount of data the challenge was to unify information from many domains systems. ELC stack (Elastic Search, Kibana, Logstash) Operational, constant dev Proof of concept Operational, complete
- 7. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 7 Outsourcing Partner capabilities on Big Data Project name Project subject Technologies Numbers Betterware Retail company Sale support prediction mechanism Together with Betterware, we analyzed the sales data and singled the sets of products which are frequently bought by consumers. Apache Sparx, Apache Hadoop, Tableau Software 8 500 customers, 1 k orders dally, machine learning algorithms train on 1 mln operations (5 years of history data). Insurance company Integration of customer databases The aim of the project was to integrate data about customers and their operations stored and managed by four different domain systems. The scope of the project contains:- data analysis and providing integrated domain model, - ETL transformations programming, - visualization of data based on Tableau Software Tableau Software, Amazon AWS 4 domain system, more than 30 unified domain objects. Operational, constant dev Proof of concept Operational, complete
- 8. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 8 Outsourcing Partner capabilities on Big Data Project name Project subject Technologies Numbers Electoral Committee Candidate for President of the Republic Media monitoring During the presidential election in 2015 in Poland we monitored social media (Facebook, Twitter, Youtube) and digital newspapers. From data fetched from social media we prepared reports of popularity of particular candidates, sentiment of comments connected with candidates and leaders of communities (blog authors, influencers), we built algorithm estimates trending phrases for political domain. Apache Hadoop, Apache Spark, HTML5 reports Operational, constant dev Proof of concept Operational, complete
- 9. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 9 Comments on Big Data from Space (OSP) • Security and legal recommendations should be defined if applicable • 4.4 Services and data location with legal consequences policy is not referenced. Harmonisation should clarify strategy and policy towards data localisation and promoted licensing models technologies. • Services reliability • 4.5.4.6 suitable services reliability or reproducibility for industrial development. Availability model should be applied (like in the Ground Segment) for platforms exposed to crowdsource/industry to secure its business models • Openness to other data sources • 4.5.4.1 Some proven decision support solutions base on combining satellite data and other data sinks, thus architecture supporting data integration should be considered.
- 10. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 10 Comments on Big Data from Space (OSP) • Consider exchangeability aspect • 4.5.X.1 Interoperability and exchangeability can be one of the strategy dimension in cross domain data flow. • Consider architectural influence of data organisational spread on usability (technical) • 4.5.2.1 For data organisation (like CDM) shredding policy should be aligned to current and potential requirements. Solution should enable generic interfaces be build in awareness of underlying data distribution while not infrastructure. • Openness vs predictability on provided platforms • 4.5.3.1 orchestration and prioritisation: in shared environment extensive experiments may coexists with operational periodic/stream analytics that should not be depredated.
- 11. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 11 OSP suggestions for Big Data from Space Roadmap Apart from precise needs and solutions mapping we suggest consideration of following. • Standardisation advisory body constituted for new/ongoing initiatives would enable natural alignment to process and consider new approaches. • Services and technologies catalogue of state of the art, recommended and applying setup for members and industry review. • Layered architecture of systems should be proposed and adopted with common interfaces to enable interoperability, relocations, third party added value services development - with respect of blurred borders and dependencies. • Federalisation tactics should be consolidated. • Industry-related, legal and security policies and strategies should be defined.
- 12. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 12 Conclusions on Big Data from Space from OSP The most valuable Big Data projects came from interdisciplinary teams which can juggle data from many different data sources
- 13. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 13 Conclusions on Big Data from Space (OSP) Data Scientists are mostly mathematicians and physics. Significant part of them start experiments from sample databases such us IRIS or Lena. Why can't they use the Agency resources?
- 14. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 14 Conclusions (OSP) As SME with long SW and big data domain we recognise following challenges in unlocking data potential according to 5.2 European Strategic Interests: • High entry threshold - data is closed for non-domain industry companies and research units. • Current ESA big data exploitation projects are silo – there is no collaboration and competition, no place for processing workflow, • There is (possibly) evaluation gap – resources managed by the Agency are valuable but unevaluated, there are no (not many) mechanism for collecting community feedback and evolve, • Great data and services are of undefined reliability and partly unpredictible
- 15. Outsourcing Partner Big Data from the Space | 21st of February 2017 | Slide 15 Conclusions (OSP) Useful tools to deal with pitfalls of Big Data exploitation: • Focusing on the potential customers the Agency should put an effort promoting and exposing the value of the data, • Data platform should be as open & simple as possible – the Open Data principle, • Implement mechanisms of collaboration; define subsets, rate&evaluate, share: ideas, experiments, results, extend, finally create processing chain, • Deliver reliable services meeting industry needs or enable commercial federalisation/transition to business of value added services