![7CONFIDENTIAL
HUE with CDH-5
[desktop]
#database_logging=True
django_debug_mode=True
collect_usage=False
use_new_editor=True
use_new_side_panels=True
app_blacklist=spark,zookeeper,search,indexer,sqoop,pig,jobsub,rdbms
[[auth]]
backend=desktop.auth.backend.PamBackend,desktop.auth.backend.AllowFirstUserDjangoBack
end
pam_service=sudo sshd login
#idle_session_timeout=120
[[session]]
expire_at_browser_close=True
[hbase]
#hbase_conf_dir=/etc/hbase/conf
hbase_conf_dir={{HBASE_CONF_DIR}}
hbase_cluster=(Peach|ip-172-31-46-118.eu-west-1.compute.internal:9090)
[impala]
[[ssl]]
enabled=true
validate=false
[beeswax]
hive_server_host=ip-172-31-46-119.eu-west-1.compute.internal
[[ssl]]
enabled=false
validate=false](https://image.slidesharecdn.com/dmitriyyatsyuk-170607133348/85/ITsubbotnik-Spring-2017-Dmitriy-Yatsyuk-Hadoop-Big-data-AWS-Cloudera-CDH-5-x-7-320.jpg)
![9CONFIDENTIAL
LET’S Encrypt implementation
# renew_before_expiry = 30 days
version = 0.13.0
archive_dir =
/etc/letsencrypt/archive/cdm.aptest.CLIENT.com
cert =
/etc/letsencrypt/live/cdm.aptest.CLIENT.com/cert.pem
privkey =
/etc/letsencrypt/live/cdm.aptest.CLIENT.com/privkey.pe
m
chain =
/etc/letsencrypt/live/cdm.aptest.CLIENT.com/chain.pe
m
fullchain =
/etc/letsencrypt/live/cdm.aptest.CLIENT.com/fullchain.
pem
# Options used in the renewal process
[renewalparams]
authenticator = standalone
installer = None
account = 69eadfb4d56ff298317fea965987659a
standalone_supported_challenges = http-01](https://image.slidesharecdn.com/dmitriyyatsyuk-170607133348/85/ITsubbotnik-Spring-2017-Dmitriy-Yatsyuk-Hadoop-Big-data-AWS-Cloudera-CDH-5-x-9-320.jpg)
ITsubbotnik Spring 2017: Dmitriy Yatsyuk "Готовое комплексное инфраструктурное решение для Hadoop Big data и AWS с Cloudera CDH 5.x"
- 1. 1CONFIDENTIAL Real complex infrastructure solution for Hadoop Big data and AWS with Cloudera CDH 5.x May, 2017
- 2. 2CONFIDENTIAL CLIENT • Epam is responsible for system engineering of enterprise data lake. based on the Hadoop technology stack. • Epam is responsible for ETL implementation from other internal system as well as external data providers • Epam is responsible for development and maintainance one of the key areas – Customer Value Management flows. • Epam developed flexible store level dashboards to provide real-time insight into sales process based on Tableaus CLIENT turned to Epam in order to resolve the performance and stability issues linked to the growing volumes of data. Also CLIENT has business requirements to implement advanced analytics and advanced techniques of sales. BUSINESS PROBLEM IMPLEMENTATION
- 5. 5CONFIDENTIAL EMR vs CLOUDERA CDH 5.x
- 7. 7CONFIDENTIAL HUE with CDH-5 [desktop] #database_logging=True django_debug_mode=True collect_usage=False use_new_editor=True use_new_side_panels=True app_blacklist=spark,zookeeper,search,indexer,sqoop,pig,jobsub,rdbms [[auth]] backend=desktop.auth.backend.PamBackend,desktop.auth.backend.AllowFirstUserDjangoBack end pam_service=sudo sshd login #idle_session_timeout=120 [[session]] expire_at_browser_close=True [hbase] #hbase_conf_dir=/etc/hbase/conf hbase_conf_dir={{HBASE_CONF_DIR}} hbase_cluster=(Peach|ip-172-31-46-118.eu-west-1.compute.internal:9090) [impala] [[ssl]] enabled=true validate=false [beeswax] hive_server_host=ip-172-31-46-119.eu-west-1.compute.internal [[ssl]] enabled=false validate=false
- 9. 9CONFIDENTIAL LET’S Encrypt implementation # renew_before_expiry = 30 days version = 0.13.0 archive_dir = /etc/letsencrypt/archive/cdm.aptest.CLIENT.com cert = /etc/letsencrypt/live/cdm.aptest.CLIENT.com/cert.pem privkey = /etc/letsencrypt/live/cdm.aptest.CLIENT.com/privkey.pe m chain = /etc/letsencrypt/live/cdm.aptest.CLIENT.com/chain.pe m fullchain = /etc/letsencrypt/live/cdm.aptest.CLIENT.com/fullchain. pem # Options used in the renewal process [renewalparams] authenticator = standalone installer = None account = 69eadfb4d56ff298317fea965987659a standalone_supported_challenges = http-01
- 11. 11CONFIDENTIAL Chef server implementation name 'users' maintainer 'Chef Software, Inc.' maintainer_email 'cookbooks@chef.io' license 'Apache 2.0' description 'Creates users from a databag search' long_description IO.read(File.join(File.dirname(__FILE__), 'README.md')) version '1.8.3' recipe 'users::default', 'Empty recipe for including LWRPs' recipe 'users::sysadmins', 'Create and manage sysadmin group' %w( ubuntu debian redhat centos fedora freebsd mac_os_x scientific oracle amazon ).each do |os| supports os end source_url 'https://github.com/chef-cookbooks/users' if respond_to?(:source_url) issues_url 'https://github.com/chef-cookbooks/users/issues' if respond_to?(:issues_url)
- 16. 16CONFIDENTIAL CLIENT Server deployment diagram Key features: Internet Client pc’s Bastion Hive2 Data Node Node man. NAT Hue Resource Manager Cloudera Zabbix Jenkins Chef- Server Hue LB Tableau LB Tableau DC1 Tableau DC2 Tableau Prod Tableau Worker1 Tableau Worker2 Tableau Backup Name Node Oozie Peach-Cluster-n1 ….. Peach-Cluster-n4 Test nodes Jupyter HIveLB R-Studio Peach Hue LB Peach HIve2 LB Peach CHD5 LB - Production and staging environments - HA mode(Hadoop and Tableau) - DMZ configuration - Centralized configuration and monitoring - Dedicated analytics server DMZ configuration( Jupyter and R studio) - CM env
- 17. 17CONFIDENTIAL • Data integration projects of implementing new platform into enterprise fabric based on Enterprise Data Hub • Developed Security Model for Big Data solution (Kerberos) • Implemented production and staging environment • Rapid ETL Development by using Python, Pig, Hive, MapReduce over Hadoop • >30 ETL jobs • Bringing unstructured and semi-structured data • Integration with Enterprise Infrastructure • Using Tableau for rich data visualization CLIENT: BIG DATA EXPERIENCE KEY TECHNICAL HIGHLIGHTS
- 18. 18CONFIDENTIAL EPAM BIG DATA COMPETENCY CENTER Big Data Architecture Design, Implementation, and SupportBig Data CC value for clients Deep expertise with cutting edge technologiesTop Facts • Data Strategy, Data Governance Consulting • Data Hub/Lake architecture • Advanced Solutions Development, Predictive & Prescriptive Analytics • Infrastructure Implementation and Integration with Enterprise Security • Bi Data Solutions & Platform Support • 300+ Engineers, Architects and Consultants • 50+ Successfully delivered Big Data and HPC solutions • 10+ years of BI product development history (for SAP, Oracle, Pentaho) • We understand business and how to make BI & BigData technology work • Our design is straightforward while following industry best practices • EPAM process is interactive, iterative and highly effective • Quick development and timely implementation • Proven delivery approach
- 19. 19CONFIDENTIAL EPAM OFFERINGTOP MESSAGES AND KEY FACTS TECHNOLOGIES EPAM Data Science includes Data Scientists and Senior Solution Architects with MS/PhD Degrees in Applied Math, Physics, Computer Science, & Predictive Analytics. Along with a strong mathematical background, the group has vertical expertise in multiple industries and extensive practical software development skills related to Big Data and conventional DW technologies • 100+ Data Scientists • 20+ Data Modelers • 10+ Data Strategists • 50+ successfully delivered projects TOP STORIES DATA SCIENCE: DISCOVER AND PREDICT Services • Large scale information solutions design • Predictive model building and validation • Customer segmentation analysis • Data profiling and preparation • Dimension reduction techniques Mathematical foundation • Probability, statistics, and stochastic processes • Supervised and unsupervised machine learning techniques • Numerical methods and implementations • Optimization theory Predictive models: linear and logistic regression, decision trees, clustering, naïve bayes, support vector machines, neural networks, kernel estimation, panel data analysis, survival/duration analysis, and time series analysis
Editor's Notes
- #2: Готовое комплексное инфраструктурное решение для Hadoop Big Data на Amazon AWS на базе CDH 5.x Пользоваться выхлопом данного решения будут бизнесс аналитики на примере Для компании по розниченой продаже брэндовой одежды и парфюмерии На основании экономия ресурсов Безоасность Удобство использования и поддержки
- #3: Школа ремонта С чем пришел клиент Что все плохо и падает и вылезли из задницы И клиент доволен
- #18: А теперь почему это получилось потому что у нас есть епам компетенси центр
- #19: Мы понимаем бизнес и то, как работает технология BI & BigData Наш дизайн прост и соответствует лучшим отраслевым практикам Процесс EPAM является интерактивным, итеративным и высокоэффективным Быстрая разработка и своевременное внедрение Подтвержденный метод доставки Стратегия данных, Консалтинг по управлению данными Архитектура центра данных / озера Разработка передовых решений, предварительная и предписывающая аналитика Внедрение инфраструктуры и интеграция с корпоративной безопасностью Bi Data Solutions и поддержка платформ
- #20: EPAM Data Science включает ученых-данных и старших архитекторов решений со степенями MS / PhD по прикладной математике, физике, информатике и аналитической аналитике. Наряду с сильной математической базой группа обладает опытом работы в разных отраслях и имеет обширные практические навыки разработки программного обеспечения, связанные с большими данными и традиционными технологиями DW Разработка крупномасштабных информационных решений Прогнозирование построения моделей и их валидация Анализ сегментации клиентов Профилирование и подготовка данных Методы уменьшения размеров Кто хочет хочет поучаствовать ….. Коллеги спасибо вам за внимание и я буду рад если кому-то это понадобится и использует в своей работе и карьере