![[TOPIC]
We will cover…
1. Baseline architecture
2. New requirements for the cloud use-case
3. Deep dive into the Solution
• App Services
• Event Hubs
• Service Fabric
• ElasticSearch in Azure
| On the way to Azure: monitoring and analytics based on Elastic stack](https://image.slidesharecdn.com/mdtazurebootcamp-170427084953/85/On-the-way-to-Azure-monitoring-and-analytics-based-on-Elastic-stack-2-320.jpg)
![[TOPIC]
Baseline architecture: Problem statement
The products under the hood
1. Black boxes
2. High complexity
3. Few diagnostics capabilities
What clients see
1. “Irrelevant” new features
2. High regression rate
3. Issue denial by Support
Key facts about products to monitor:
1. 10000+ tables, 70000+ indexes and 10000+ unique queries in MS SQL Server
2. 100+ on-premise installations
3. Regression testing takes 2-3 months – never run for hot fixes in the old versions
4. Still brings a lot of money to our customer
| On the way to Azure: monitoring and analytics based on Elastic stack](https://image.slidesharecdn.com/mdtazurebootcamp-170427084953/85/On-the-way-to-Azure-monitoring-and-analytics-based-on-Elastic-stack-3-320.jpg)
![[TOPIC]
Baseline architecture: Considered Options
Products (free/commercial)
1. Zabbix / Nagios / OpenNMS /
New Relic
2. Azure application insights
3. Elastic stack
Major driving requirements
1. OOTB features
2. Flexibility
3. Performance, Availability and
Scalability
| On the way to Azure: monitoring and analytics based on Elastic stack](https://image.slidesharecdn.com/mdtazurebootcamp-170427084953/85/On-the-way-to-Azure-monitoring-and-analytics-based-on-Elastic-stack-4-320.jpg)
![[TOPIC]
Baseline architecture: Solution Context View
Monitoring System
Support EngineerSupport Engineer
QAQA
How latest fix
affected performance?
What is this process
at all doing?
What are the most important
user processes?
What is User A doing?
Performance metrics Application logs
Configuration changes Analytics
Monitored Product #1
Monitored Product #N
Metrics and data
Metrics and data
Legend
System Function
DeveloperDeveloper
Product ManagerProduct Manager
Why is my
process slow?
User AUser A
| On the way to Azure: monitoring and analytics based on Elastic stack](https://image.slidesharecdn.com/mdtazurebootcamp-170427084953/85/On-the-way-to-Azure-monitoring-and-analytics-based-on-Elastic-stack-5-320.jpg)
![[TOPIC]
New requirements: limitations and drivers
| On the way to Azure: monitoring and analytics based on Elastic stack
1. Monitored products targeting Azure
2. No way to do cross-customer analysis
3. Support costs optimization
4. Higher availability
5. Higher interoperability](https://image.slidesharecdn.com/mdtazurebootcamp-170427084953/85/On-the-way-to-Azure-monitoring-and-analytics-based-on-Elastic-stack-6-320.jpg)
![[TOPIC]
New requirements: cloud-native options
| On the way to Azure: monitoring and analytics based on Elastic stack
1. Application Insights
2. Operations Management
Suite
3. Azure Search
1. Provide similar
experience for cloud and
on-prem
2. Maintain single codebase
vs.](https://image.slidesharecdn.com/mdtazurebootcamp-170427084953/85/On-the-way-to-Azure-monitoring-and-analytics-based-on-Elastic-stack-7-320.jpg)
![[TOPIC]
The solution: Overview
| On the way to Azure: monitoring and analytics based on Elastic stack
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS](https://image.slidesharecdn.com/mdtazurebootcamp-170427084953/85/On-the-way-to-Azure-monitoring-and-analytics-based-on-Elastic-stack-8-320.jpg)
![Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
[TOPIC]
The solution: Data collection
| On the way to Azure: monitoring and analytics based on Elastic stack
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Agent Service
Web Jobs
Monitoring Service
Web Jobs
Application
Insights
Stream Analytics
HTTPS
HTTPS
1. Azure Functions
2. Azure Batch
3. Service Fabric
4. Cloud Services
5. IaaS VMs
6. Web Jobs](https://image.slidesharecdn.com/mdtazurebootcamp-170427084953/85/On-the-way-to-Azure-monitoring-and-analytics-based-on-Elastic-stack-9-320.jpg)
![Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
[TOPIC]
The solution: Staging area
| On the way to Azure: monitoring and analytics based on Elastic stack
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Event Hubs
Staging area
Event Hub
1. Queue
2. Service Bus
3. Event Hubs](https://image.slidesharecdn.com/mdtazurebootcamp-170427084953/85/On-the-way-to-Azure-monitoring-and-analytics-based-on-Elastic-stack-10-320.jpg)
![Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
[TOPIC]
The solution: Services
| On the way to Azure: monitoring and analytics based on Elastic stack
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Availability set (Windows VMs)
subnet /28
User
HTTPS
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Thresholds
Jobs
Pentaho
Jobs
Ingest
Jobs
1. Azure Functions
2. Azure Batch
3. Web Jobs
4. Cloud Services
5. IaaS VMs
6. Service Fabric
1. Kibana (per tenant)
2. ETLs
3. Reverse Proxy
4. Ingest Jobs (new)](https://image.slidesharecdn.com/mdtazurebootcamp-170427084953/85/On-the-way-to-Azure-monitoring-and-analytics-based-on-Elastic-stack-11-320.jpg)
![Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
[TOPIC]
The solution: ElasticSearch
| On the way to Azure: monitoring and analytics based on Elastic stack
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
1. Elastic Cloud
2. AWS ElasticSearch
3. Private ElasticSearch hoster
4. IaaS VMs
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Backup Storage
Azure
Diagnostics
Elasticsearch
data
Elasticsearch
master
Load balancer
Load balancer](https://image.slidesharecdn.com/mdtazurebootcamp-170427084953/85/On-the-way-to-Azure-monitoring-and-analytics-based-on-Elastic-stack-12-320.jpg)
![[TOPIC]
The solution: Summary
| On the way to Azure: monitoring and analytics based on Elastic stack
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS
Common Virtual Network
Microsoft
Azure
Availability set (Linux VMs)
subnet /29
Availability set (Linux VMs)
subnet /28
TLS
Event Hubs
Staging area
Event Hub
HTTPS
Availability set (Windows VMs)
subnet /28
HTTPS
User
HTTPS
Availability set
subnet /??
Web Servers
+Agents
Azure SQL
database
TLS
TLS
HTTPS
HTTPS
Azure
Diagnostics
HTTPS
Backup Storage
Azure
Diagnostics
Service
Fabric
Kibana
Azure AD
Reverse proxy
Key Vault
HTTPS
Agent Service
Web JobsThresholds
Jobs
Elasticsearch
data
Elasticsearch
master
SendGrid
Monitoring Service
Web Jobs
Load balancer
Pentaho
Jobs
Ingest
Jobs
Load balancer
Application
Insights
Stream Analytics
HTTPS
HTTPS
HTTPS](https://image.slidesharecdn.com/mdtazurebootcamp-170427084953/85/On-the-way-to-Azure-monitoring-and-analytics-based-on-Elastic-stack-13-320.jpg)
On the way to Azure: monitoring and analytics based on Elastic stack
- 1. On the way to Azure: monitoring and analytics based on Elastic stack Artsiom Baranovskiy, Solution Architect at EPAM Systems Artsiom_Baranovskiy@epam.com April 22, 2017
- 2. [TOPIC] We will cover… 1. Baseline architecture 2. New requirements for the cloud use-case 3. Deep dive into the Solution • App Services • Event Hubs • Service Fabric • ElasticSearch in Azure | On the way to Azure: monitoring and analytics based on Elastic stack
- 3. [TOPIC] Baseline architecture: Problem statement The products under the hood 1. Black boxes 2. High complexity 3. Few diagnostics capabilities What clients see 1. “Irrelevant” new features 2. High regression rate 3. Issue denial by Support Key facts about products to monitor: 1. 10000+ tables, 70000+ indexes and 10000+ unique queries in MS SQL Server 2. 100+ on-premise installations 3. Regression testing takes 2-3 months – never run for hot fixes in the old versions 4. Still brings a lot of money to our customer | On the way to Azure: monitoring and analytics based on Elastic stack
- 4. [TOPIC] Baseline architecture: Considered Options Products (free/commercial) 1. Zabbix / Nagios / OpenNMS / New Relic 2. Azure application insights 3. Elastic stack Major driving requirements 1. OOTB features 2. Flexibility 3. Performance, Availability and Scalability | On the way to Azure: monitoring and analytics based on Elastic stack
- 5. [TOPIC] Baseline architecture: Solution Context View Monitoring System Support EngineerSupport Engineer QAQA How latest fix affected performance? What is this process at all doing? What are the most important user processes? What is User A doing? Performance metrics Application logs Configuration changes Analytics Monitored Product #1 Monitored Product #N Metrics and data Metrics and data Legend System Function DeveloperDeveloper Product ManagerProduct Manager Why is my process slow? User AUser A | On the way to Azure: monitoring and analytics based on Elastic stack
- 6. [TOPIC] New requirements: limitations and drivers | On the way to Azure: monitoring and analytics based on Elastic stack 1. Monitored products targeting Azure 2. No way to do cross-customer analysis 3. Support costs optimization 4. Higher availability 5. Higher interoperability
- 7. [TOPIC] New requirements: cloud-native options | On the way to Azure: monitoring and analytics based on Elastic stack 1. Application Insights 2. Operations Management Suite 3. Azure Search 1. Provide similar experience for cloud and on-prem 2. Maintain single codebase vs.
- 8. [TOPIC] The solution: Overview | On the way to Azure: monitoring and analytics based on Elastic stack Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS
- 9. Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS [TOPIC] The solution: Data collection | On the way to Azure: monitoring and analytics based on Elastic stack Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS HTTPS HTTPS Azure Diagnostics HTTPS Agent Service Web Jobs Monitoring Service Web Jobs Application Insights Stream Analytics HTTPS HTTPS 1. Azure Functions 2. Azure Batch 3. Service Fabric 4. Cloud Services 5. IaaS VMs 6. Web Jobs
- 10. Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS [TOPIC] The solution: Staging area | On the way to Azure: monitoring and analytics based on Elastic stack Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS Event Hubs Staging area Event Hub 1. Queue 2. Service Bus 3. Event Hubs
- 11. Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS [TOPIC] The solution: Services | On the way to Azure: monitoring and analytics based on Elastic stack Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS Availability set (Windows VMs) subnet /28 User HTTPS Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Thresholds Jobs Pentaho Jobs Ingest Jobs 1. Azure Functions 2. Azure Batch 3. Web Jobs 4. Cloud Services 5. IaaS VMs 6. Service Fabric 1. Kibana (per tenant) 2. ETLs 3. Reverse Proxy 4. Ingest Jobs (new)
- 12. Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS [TOPIC] The solution: ElasticSearch | On the way to Azure: monitoring and analytics based on Elastic stack Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS 1. Elastic Cloud 2. AWS ElasticSearch 3. Private ElasticSearch hoster 4. IaaS VMs Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Backup Storage Azure Diagnostics Elasticsearch data Elasticsearch master Load balancer Load balancer
- 13. [TOPIC] The solution: Summary | On the way to Azure: monitoring and analytics based on Elastic stack Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS Common Virtual Network Microsoft Azure Availability set (Linux VMs) subnet /29 Availability set (Linux VMs) subnet /28 TLS Event Hubs Staging area Event Hub HTTPS Availability set (Windows VMs) subnet /28 HTTPS User HTTPS Availability set subnet /?? Web Servers +Agents Azure SQL database TLS TLS HTTPS HTTPS Azure Diagnostics HTTPS Backup Storage Azure Diagnostics Service Fabric Kibana Azure AD Reverse proxy Key Vault HTTPS Agent Service Web JobsThresholds Jobs Elasticsearch data Elasticsearch master SendGrid Monitoring Service Web Jobs Load balancer Pentaho Jobs Ingest Jobs Load balancer Application Insights Stream Analytics HTTPS HTTPS HTTPS
- 14. Thank you! Your Questions are Welcome! Artsiom_Baranovskiy@epam.com | On the way to Azure: monitoring and analytics based on Elastic stack