Why advanced monitoring is key for healthy
2 likes344 views
The document discusses the importance of monitoring in data virtualization, highlighting its benefits such as improved support, security, and capacity planning. It details the features of Denodo's monitoring tools, which provide real-time insights, error logging, and diagnostics capabilities, as well as integration with enterprise monitoring tools for historical data analysis. Proper implementation of these monitoring strategies is essential for maintaining healthy enterprise deployments and facilitating growth.
Why advanced monitoring is key for healthy
- 1. DATA VIRTUALIZATION PACKED LUNCH WEBINAR SERIES Sessions Covering Key Data Integration Challenges Solved with Data Virtualization
- 2. Why Advanced Monitoring is Key for Healthy Enterprise Deployments Pablo Alvarez-Yanez Director of Product Management, Denodo
- 4. 4 Q: Do I Need Monitoring? A: YES
- 5. 5 But Why? • Get/Provide better support • Detailed information to simplify resolution of issues and support cases • Improved security • Audit trails and cyber-attack alerts • Keep users happy • Proactive detection of potential issues: e.g. sources are down • Be prepared for the future • Base information for capacity planning, detect infrastructure bottlenecks • Reduce infrastructure costs • Scale down based on access patterns Setting up, using and maintaining a proper monitoring infrastructure require some effort. However, the benefits derived from a better understanding of the platform usage are unquestionable:
- 6. 6 And How? Monitoring can be see from many different angles and usually there is no "one tools fits all" approach. For example, we can think of monitoring as: • Real time monitoring • Detailed insights on the current status of the server • Diagnostics of issues • Go “back-in-time” to understand why an • Auditing access • Examine log in attempts and queries based on different criteria • Historical analysis • Identification of trends and patterns that can only be seen with analysis over a longer period of time • Alerts • Notifications when certain situations are problematic
- 8. 8 Query Execution ▪ Details of the query execution can be reviewed during execution time, both from the design studio and the monitoring tool ▪ These execution traces can be saved and loaded in the design studio for detailed examination. ▪ In addition, these query traces can be also generated and stored for all queries run on the server. ▪ Although this is costly, it’s sometimes useful to debug some issue. ▪ Do not have this option always enabled in production! ▪ This capability can be enabled with the following command: ▪ SET 'com.denodo.vdb.interpreter.execution.saveTrace' = 'ALL_FILE' ▪ To log only erroneous queries, use the value ‘ERROR_FILE’ instead. Use ‘DISABLED’ to go back to normal logging settings
- 9. 9 Denodo Monitoring Tool Web-based tool that provides detailed information on the current status of the Denodo server. In addition, with the right privileges, allows to act on the system. For example, cancel queries and sessions. Its main capabilities are: ▪ Distributed with the Denodo Platform ▪ Integrated with the system authentication, easy to use ▪ Access to very granular information about the current status of the system ▪ Users and sessions (possibility to close them from the tool) ▪ Queries (possibility to see the execution plan, and cancel queries) ▪ Cache processes ▪ Status of data sources (status, ping, connection pools) ▪ Detailed thread information
- 11. 11 Server Error Logs ▪ The different components of the Denodo platform log errors in the <denodo>/logs folders ▪ These files are the most valuable resource to understand server errors, as they contain the error trace generated during execution ▪ The logs are generated with log4j with a rolling policy: ▪ Current log file plus an archive of 7 files of 10MB each ▪ The rolling settings, output files and format of the output can be changed in the logj2.xml configuration files located in <denodo>/conf/<component>/logj2.xml
- 12. 12 Monitor Logs (CPU, queries, sessions, cache, etc.) Denodo can automatically log a lot of detailed information using a monitoring agent, called “Denodo Monitor”. ▪ The information is retrieved via a JMX subscription to the server. ▪ Since the agent can run externally (normally in the Solution Manager server), it adds minimal overhead to the actual Denodo server. ▪ It can be launched graphically from the Solution Manager, or with shell scripts. ▪ You can find more details in the documentation ▪ It offers multiple storage options: ▪ Log to CSV files (default) ▪ Log to database (Oracle, SQLServer, MySQL, Postgres) ▪ Log to S3
- 13. 13 Data Source Status Logs ▪ Denodo can also ping data sources in time intervals, to make sure that the sources are up and running ▪ Since end users interact directly with Denodo, understanding when errors actually come from non responsive data sources is fundamental to understand many issues ▪ These settings can be configured in the configuration file of the Denodo monitor in <denodo>/tools/monitor/denodo- monitor/conf or <solution-manager>/conf/solution-manager- monitor/denodo-monitor/ /conf/solution-manager/denodo-monitor
- 14. Diagnostics of Individual Issues 14
- 15. 15 Denodo Monitoring and Diagnostics Tool In addition to real time monitoring, Denodo’s monitoring tool can load Denodo logs for a particular timeframe to “go back in time” to help identify and diagnose an individual issue. Some of it’s capabilities are: ▪ Possibility to "zoom in" into smaller time frames to narrow down the scope ▪ Access to very granular information about the status of the system during a particular time frame ▪ Integration with VDP errors stack trace ▪ Users and sessions ▪ Queries ▪ Cache processes ▪ Status of data sources (status, ping, connection pools) ▪ Detailed thread information This tool load the information from the logs in the cache database. It is intended for specific timeframes (e.g. 1 hour, 1 day). It is not intended for historic monitoring of larger timeframes
- 16. Auditing 16
- 17. 17 Monitor Logs – Auditing details Logging attempts, data accessed by a particular user, or sessions opened from a particular attempt, are typical examples of access audits. The information required to address those questions is available in the monitor logs mentioned in the previous section However, if these questions are frequently asked, it’s recommended to log directly into a relational database This way, any reporting tool using standard SQL can be used to easily report on those metrics
- 19. 19 Historical Monitoring - Needs Monitoring a solution over a long period of time is important to draw conclusions on average usage, peaks and capacity planning However, the volume of data generated by logs can grow out of control very easily. For example: • We see around 5-6 concurrent queries running in the server. They take from 0.5 to 1 second • This is: 5 x 60 secs x 60 min x 24 hours = ~450k queries per day • ~160 million queries per year As we can see, logging relatively small loads will produce a lot of data to manage and report on Fortunately, there are enterprise log management products specialized on this type of scenarios
- 20. 20 Enterprise Monitoring Tools Enterprise monitoring tools normally provide the following capabilities 1. A log gathering agent, that runs in each one of the servers 2. A processing pipeline to parse and format the data in the logs 3. A time-series storage and processing engine, capable of storing large volumes and efficiently run queries based on time intervals 4. A dashboarding / reporting tool focused on monitoring There are many tools in the market that cover this segment, for example Splunk, the ELK stack (Elastic, Logstash, Kibana), CloudWatch in AWS, etc. Some other products specialize on the dashboarding part only, for example Grafana
- 21. 21 Enterprise Monitoring Tools and Denodo Integrating these tools with Denodo is quite straightforward 1. If necessary, install the agent 2. Depending on the log you are monitoring, specify the corresponding pattern 1. Many are just CSV files, others will need a regular expression 3. Ingest and store the logs 4. Build a dashboard to show the metrics you are interested on In the Denodo community site you can already find detailed information for Splunk and CloudWatch
- 22. Alerts 22
- 23. 23 Alerts Another interesting feature of enterprise monitoring tools is the generation of alerts Based on the incoming logs, and a set of pre-configured parameters, these tools can generate an alert that can trigger some custom action, normally sending an email to an administrator For example, alerts can be useful for: • System is not responding • Data sources are down • Number of concurrent requests is higher than a safety limit In the articles for Splunk and CloudWatch in the Denodo Community you can also find details on how to configure alerts
- 24. Summary 24
- 25. 25 Summary ▪ Good monitoring is the foundation of a healthy deployment and sets the path for trust and growth ▪ Denodo offers multiple logs and tools that provides a detailed view of the status of the server ▪ Enterprise monitoring tools can integrate easily to provide additional capabilities like historical monitoring and alerts ▪ Together, this ecosystem simplifies resolution of issues, prevents them from happening, and paves the path for new projects
- 26. Q&A
- 27. 27 Next Steps Access Denodo Platform in the Cloud! Take a Test Drive today! www.denodo.com/TestDrive GET STARTED TODAY
- 28. Thank you! © Copyright Denodo Technologies. All rights reserved Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.