Introduction to cloudforecast
3 likes1,623 views
CloudForecast is a system monitoring and visualization tool that uses Perl and RRDTool to collect data from servers and generate graphs. It collects metrics like CPU usage, network traffic, and Gearman worker status. Data is stored in RRD files and a SQLite database. A radar component collects data and a web interface is used to view graphs generated from the collected data.
![package CloudForecast::Data::Gearmanstarter;
use CloudForecast::Data -base;
use IO::Socket::INET;
=pod
host_config)
resources:
- gearmanstarter[:port]
eg)
- gearmanstarter # 9000
- gearmanstarter:9005 # port
=cut](https://image.slidesharecdn.com/intro-cloudforecast-100726094115-phpapp01/85/Introduction-to-cloudforecast-55-320.jpg)
![rrds map { [$_,'GAUGE'] } qw/busy idle/;
# rrds ‘key’, ‘TYPE’ or rrds [‘key’,‘TYPE’], []..](https://image.slidesharecdn.com/intro-cloudforecast-100726094115-phpapp01/85/Introduction-to-cloudforecast-56-320.jpg)
![fetcher {
my $c = shift;
my $host = $c->address;
my $port = $c->args->[0] || 9000;
my $sock = IO::Socket::INET->new(
PeerAddr => $host,
PeerPort => $port,
Proto => 'tcp',
);
my $raw_stats;
$sock->sysread( $raw_stats, 8192 );
my ($busy, $idle);
foreach my $line ( split /[rn]+/, $raw_stats ) {
if ( $line =~ /^Busy.+: (d+)/ ) {
$busy = $1;
}
if ( $line =~ /^Idle.+: (d+)/ ) {
$idle = $1;
}
}
return [$busy,$idle];
};](https://image.slidesharecdn.com/intro-cloudforecast-100726094115-phpapp01/85/Introduction-to-cloudforecast-61-320.jpg)
![fetcher {
my $c = shift;
my $host = $c->address;
my $port = $c->args->[0] || 9000;
my $sock = IO::Socket::INET->new(
PeerAddr => $host,
PeerPort => $port,
Proto => 'tcp',
);
my $raw_stats;
$sock->sysread( $raw_stats, 8192 );
my ($busy, $idle);
foreach my $line ( split /[rn]+/, $raw_stats ) {
if ( $line =~ /^Busy.+: (d+)/ ) {
$busy = $1;
}
if ( $line =~ /^Idle.+: (d+)/ ) {
$idle = $1;
}
}
return [$busy,$idle];
};](https://image.slidesharecdn.com/intro-cloudforecast-100726094115-phpapp01/85/Introduction-to-cloudforecast-62-320.jpg)
Introduction to cloudforecast
- 1. Introduction to CloudForecast Yokohama.pm #6.x @kazeburo
- 2. • • @kazeburo • 2010 5 • 2010 6 • Web
- 3. CloudForecast • • • perl rrdtool sysadmin (by tokuhirom)
- 4. • •
- 5. (1)
- 6. (1) • ping
- 8. (1) • ping • HTTP,SSH • Disk
- 9. (2) • • • • Nagios • monit
- 10. (1)
- 11. (1) •
- 12. (1) • • CPU
- 13. (2) • • • • Cacti • munin • Ganglia • CloudForecast
- 14. CloudForecast • • YAML • Web • Perl!
- 15. CPAN • Plack + Starlet • Gearman • Parallel::Prefork • DBI DBD::SQLite • YAML::Syck • Text::Xslate • Data::Section::Simple • local::lib
- 16. • RRDTool RRDs.pm • SNMP.pm • Shirahata(WAF)
- 17. CloudForecast 1 RRD Radar Web SQLite
- 18. CloudForecast 1 (1) RRD Radar Web SQLite
- 19. CloudForecast 1 (1) RRD Radar Web SQLite (2)
- 20. CloudForecast 1 (1) RRD Radar Web SQLite (2) (3)
- 21. CloudForecast 1 (1) RRD Radar Web SQLite (2) (3)
- 22. CloudForecast 2 fetcher worker RRD update Radar Web worker SQLite
- 23. CloudForecast 2 Gearman Worker fetcher worker RRD update Radar Web worker SQLite
- 24. CloudForecast 2 Gearman Worker RRD update Radar Web worker SQLite
- 26. 1 # ubuntu $ sudo apt-get install librrds-perl libsnmp-perl # CentOS rrdtool EPEL $ sudo yum install net-snmp-perl $ sudo yum install rrdtool-perl
- 27. 2 $ git clone git://github.com/kazeburo/cloudforecast.git $ cd cloudforecast $ cpanm -l extlib --installdeps . $ ls Changes cloudforecast_web Makefile.PL docs README host_config cf_devstarter htdocs cf_fetcher_worker lib cf_updater_worker server_list_sample.yaml cloudforecast_radar t cloudforecast_sample.yaml
- 28. # $ cp cloudforecast_sample.yaml cloudforecast.yaml # $ cp server_list_sample.yaml server_list.yaml
- 29. (server_list.yaml) --- #LOCALHOST servers: - config: basic.yaml hosts: - 127.0.0.1 server1 my great localhost --- #My Servers servers: - config: basic.yaml hosts: ....
- 30. (server_list.yaml) --- #LOCALHOST servers: - config: basic.yaml hosts: - 127.0.0.1 server1 my great localhost --- #My Servers servers: - config: basic.yaml hosts: ....
- 31. (server_list.yaml) --- #LOCALHOST servers: - config: basic.yaml hosts: - 127.0.0.1 server1 my great localhost --- #My Servers servers: - config: basic.yaml hosts: ....
- 32. (server_list.yaml) --- #LOCALHOST servers: - config: basic.yaml hosts: - 127.0.0.1 server1 my great localhost --- #My Servers IP servers: - config: basic.yaml hosts: ....
- 33. (server_list.yaml) --- #LOCALHOST servers: - config: basic.yaml hosts: - 127.0.0.1 server1 my great localhost --- #My Servers IP servers: - config: basic.yaml hosts: ....
- 34. (server_list.yaml) --- #LOCALHOST servers: - config: basic.yaml hosts: - 127.0.0.1 server1 my great localhost --- #My Servers IP servers: - config: basic.yaml hosts: ....
- 35. (basic.yaml) --- component_config: resources: - basic - traffic:eth0 basic = eth0 CPU TCP
- 36. (basic.yaml) --- component_config: resources: - basic - traffic:eth0 basic = eth0 CPU TCP
- 37. (basic.yaml) --- component_config: resources: - basic - traffic:eth0 basic = eth0 CPU TCP
- 38. • Basic(CPU TCP ) • Traffic • Disk Usage • DiskIO Count • Apache Status • Nginx Status • Squid • Mysql • Innodb
- 42. # $ ./cloudforecast_radar -c cloudforecast.yaml -l server_list.yaml # Web $ ./cloudforecast_web -p 5000 -c cloudforecast.yaml -l server_list.yaml
- 43. Web ( )
- 44. Web ( )
- 47. •
- 48. • •
- 49. • • •
- 50. # site-lib $ mkdir -p site-lib/CloudForecast/Data
- 51. Gearman Worker gearman worker process http://blog.nomadscafe.jp/2010/07/gearman-worker-process.html
- 52. % ./gearman-starter.pl --max-prcess 8 -s 127.0.0.1:7004 --scoreboard-dir /var/run/worker --port 7005 MyWorker % telnet localhost 7005 Trying 127.0.0.1... System: gearman_servers: 127.0.0.1:7004 class: MyWorker BusyWorkers: 0 IdleWorkers: 8 -- pid Status Counter Comment 1630 . 0 1631 . 0 1632 . 0 1633 . 0 1634 . 0 1635 . 0 1636 . 0 1637 . 0
- 53. % ./gearman-starter.pl --max-prcess 8 -s 127.0.0.1:7004 --scoreboard-dir /var/run/worker --port 7005 MyWorker % telnet localhost 7005 Trying 127.0.0.1... System: gearman_servers: 127.0.0.1:7004 class: MyWorker BusyWorkers: 0 IdleWorkers: 8 -- pid Status Counter Comment 1630 . 0 1631 . 0 1632 . 0 1633 . 0 1634 . 0 1635 . 0 1636 . 0 1637 . 0
- 54. • •
- 55. package CloudForecast::Data::Gearmanstarter; use CloudForecast::Data -base; use IO::Socket::INET; =pod host_config) resources: - gearmanstarter[:port] eg) - gearmanstarter # 9000 - gearmanstarter:9005 # port =cut
- 56. rrds map { [$_,'GAUGE'] } qw/busy idle/; # rrds ‘key’, ‘TYPE’ or rrds [‘key’,‘TYPE’], []..
- 57. • COUNTER • • • ) Traffic ( octet ) • GAUGE • • • )
- 58. (1) graphs 'status' => 'worker status'; # graphs ‘key1’ => ‘label’; # graphs ‘key2’ => ‘the great graph’;
- 59. (2) graphs 'status' => 'worker status'; .. __DATA__ @@ status DEF:my1=<%RRD%>:busy:AVERAGE DEF:my2=<%RRD%>:idle:AVERAGE AREA:my1#00C000:Busy GPRINT:my1:LAST:Cur: %4.1lf GPRINT:my1:AVERAGE:Ave: %4.1lf GPRINT:my1:MAX:Max: %4.1lf GPRINT:my1:MIN:Min: %4.1lfc STACK:my2#0000C0:Idle GPRINT:my2:LAST:Cur: %4.1lf GPRINT:my2:AVERAGE:Ave: %4.1lf GPRINT:my2:MAX:Max: %4.1lf GPRINT:my2:MIN:Min: %4.1lfc
- 60. (2) graphs 'status' => 'worker status'; .. key __DATA__ @@ status DEF:my1=<%RRD%>:busy:AVERAGE DEF:my2=<%RRD%>:idle:AVERAGE AREA:my1#00C000:Busy GPRINT:my1:LAST:Cur: %4.1lf GPRINT:my1:AVERAGE:Ave: %4.1lf GPRINT:my1:MAX:Max: %4.1lf GPRINT:my1:MIN:Min: %4.1lfc STACK:my2#0000C0:Idle GPRINT:my2:LAST:Cur: %4.1lf GPRINT:my2:AVERAGE:Ave: %4.1lf GPRINT:my2:MAX:Max: %4.1lf GPRINT:my2:MIN:Min: %4.1lfc
- 61. fetcher { my $c = shift; my $host = $c->address; my $port = $c->args->[0] || 9000; my $sock = IO::Socket::INET->new( PeerAddr => $host, PeerPort => $port, Proto => 'tcp', ); my $raw_stats; $sock->sysread( $raw_stats, 8192 ); my ($busy, $idle); foreach my $line ( split /[rn]+/, $raw_stats ) { if ( $line =~ /^Busy.+: (d+)/ ) { $busy = $1; } if ( $line =~ /^Idle.+: (d+)/ ) { $idle = $1; } } return [$busy,$idle]; };
- 62. fetcher { my $c = shift; my $host = $c->address; my $port = $c->args->[0] || 9000; my $sock = IO::Socket::INET->new( PeerAddr => $host, PeerPort => $port, Proto => 'tcp', ); my $raw_stats; $sock->sysread( $raw_stats, 8192 ); my ($busy, $idle); foreach my $line ( split /[rn]+/, $raw_stats ) { if ( $line =~ /^Busy.+: (d+)/ ) { $busy = $1; } if ( $line =~ /^Idle.+: (d+)/ ) { $idle = $1; } } return [$busy,$idle]; };
- 63. •
- 65. IDLE BUSY
- 66. TODO/Plan • Web • RRD SQLite •
- 67. • CloudForecast Perl • •