![Tuck In! Future and Past
Moo Interception
package Foo:: Role :: Database :: CMDB;
use Moo:: Role;
requires "log";
has cmdb => (
is => "ro",
required => 1,
handles => "Foo:: Role :: Database",
isa => sub {
_INSTANCE_OF($_[0], "Foo:: Helper :: CMDB") and $_[0]-> DOES("Foo:: Role :: Dat
and return;
die " Insufficient initialisation parameter for cmdb";
},
coerce => sub {
_HASH($_ [0]) and return Foo :: Helper ::CMDB ->new (%{$_ [0]});
$_ [0];
},
);
role can be consumed by any class needing access to CMDB
transform hash initializer into object
Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 31 / 44](https://image.slidesharecdn.com/cmdb-driven-by-perl-handout-180406124723/85/A-CMDB-Driven-by-Perl-27-320.jpg)
![Tuck In! Future and Past
Hard work
package Foo:: Helper :: CMDB;
use Moo; extends "Foo:: Helper :: DatabaseClass ";
has config _tables => (is => "lazy", ...);
has history_tabl es => (is => "lazy", ...);
around deploy => sub { ...
my @tables = @{$self -> config_tables };
foreach my $tbl (@tables) {
my @h ist_coldefs =
map { my $default = defined $_ - >[4] ? " DEFAULT $_ - >[4]" : "";
$pure_cols{$_ - >[1]} ? ("$_ - >[1] $_ - >[2] $default")
: ("old_$_ - >[1] $_ - >[2] $default", "new_$_ - >[1] $_ - >[2] $default")
} @table_info ;
}
unshift @hist_coldefs , "${base_name}_hist_id INTEGER PRIMARY KEY";
my $hist_defs = join("n , ", @hist_coldefs );
my $hist_tbl = <<EOCHT;
CREATE TABLE ${base_name}_hist (
${hist_defs}
);
EOCHT
that are all tables with trailing t in their names
create history table for each config table
memoizing old and new values on updating payload
Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 32 / 44](https://image.slidesharecdn.com/cmdb-driven-by-perl-handout-180406124723/85/A-CMDB-Driven-by-Perl-28-320.jpg)
![Tuck In! Features
”UPSERT”
MERGE INTO tablename USING table_reference ON (condition)
WHEN MATCHED THEN
UPDATE SET column1 = value1 [, column2 = value2 ...]
WHEN NOT MATCHED THEN
INSERT (column1 [, column2 ...]) VALUES (value1 [, value2 ...]);
SQLite
Unsupported by SQLite
INSERT OR REPLACE deletes before insert
→ Kills UPDATE Trigger
Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 36 / 44](https://image.slidesharecdn.com/cmdb-driven-by-perl-handout-180406124723/85/A-CMDB-Driven-by-Perl-32-320.jpg)
A CMDB Driven by Perl
- 1. CMDB Driven by Perl Road to a Perl ”driven” Configuration Management Database Jens Rehsack Niederrhein Perl Mongers The Perl Conference in Amsterdam Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 1 / 44
- 2. Introduction Motivation Motivation Progress Progress isn’t made by early risers. It’s made by lazy men trying to find easier ways to do something. (Robert A. Heinlein) Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 3 / 44
- 3. Introduction Motivation Motivation Efficiency Business success is because of Perl. It enables us to deliver right solutions in days instead of months. (Elizabeth Mattijsen) Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 4 / 44
- 4. Introduction Motivation Goal Automation Full flavoured systems management Installation without Administrator interaction Control sensors and alarming Ensured system state by actual-theoretical comparison Faster reaction in emergency cases by organized component moving Have an up-to-date ”Operation Handbook” as well as archiving them Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 5 / 44
- 5. Beginning Taking over Begin with reporting Begin with reporting In the beginning was the (Installation-)Report Technical Sales defined an XML Document for Change Requests and Status Reports Based on work of forerunner a 70% solution could be delivered Document Definition lacks entity-relations Document Definition misses technolgy requirements ⇒ Appears to be a dead end Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 7 / 44
- 6. Beginning After reporting From reporting to . . . Mind the goal Alice: Would you tell me, please, which way I ought to go from here? The Cheshire Cat: That depends a good deal on where you want to get to. (Lewis Carroll) Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 8 / 44
- 7. Beginning Mine Sweeper Where do I begin To write the workflow how great Perl 5 can be The project was in a state where a developer created a particular Report based on the existing snapshot. This solution did not maintain an abstraction layer for gathered data - every time when the report needs an extension, an end-to-end (snapshot to XML-Tag) enhancement had to be created. Changes shall be deployed from the same report format as installations are reported. We have to be able to say at any moment what is operated on the platform. Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 9 / 44
- 8. Beginning Baby Steps Baby Steps Improve knowledge Based on identified issues the first goal had to be to identify all entities and their relations together Surrounded Problem: The entire platform was completely unstructured Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 10 / 44
- 9. Beginning Baby Steps Baby Steps Multiple Beginnings The already known ”(Installation-)Report” Platform Snapshot (SCM Repository of selected configuration files) Puppet Classes (without Hiera) mixed with Configuration Items (within Hiera) and prepared configuration files (unsupervised) Hiera is Puppet’s built-in key/value data lookup system. By default, it uses simple YAML or JSON files, although one can extend it to work with almost any data source. Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 11 / 44
- 10. World Domination Separation Circle in the Sand :platform :snapshot :scan db:cmdb :hiera :report collect scan process-scan yaml-gen puppet run CRQ export Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 13 / 44
- 11. Concerns Identifying Technical Concerns Rough Collecting platform parameters (to query them in structured way) Identify coherences of Configuration Items (CI) Define a data model Define technical requirements Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 14 / 44
- 12. Concerns Identifying Practical Concerns Practical Validity of CI’s Limits of our CI’s Data ownership of CI’s Methods to persist CI’s Methods to access CI’s Permission management Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 15 / 44
- 13. Tuck In! MI:5 Impossible Things Impossible Things Alice laughed. ”There’s no use trying,” she said: ”one can’t believe impossible things.” ”I daresay you haven’t had much practice,” said the Queen. ”When I was your age, I always did it for half-an-hour a day. Why, sometimes I’ve believed as many as six impossible things before breakfast.” (Lewis Carroll) Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 17 / 44
- 14. Tuck In! Home Improvement The Fool with a Tool Try again So we closed our eyes, took a deep breath (multiple times) and looked around for tools to store serialized data and read in structured way . . . Tool Time MongoDB allows easy storing in any format - but lacks structured querying dedicated entities (configuration items) Data Files delegate relationship handling completely to business logic AnyData2 gotcha - allows reading most confusing stuff and could be queried in structured way Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 18 / 44
- 15. Tuck In! Home Improvement . . . is still a Fool Volatile Structure Persist structured data using SQLite Define a data model representing existing relations Develop AnyData2::Format classes representing defined ER (Entity Relationship) model Develop simple MOP inside this AnyData2 instance to manage attributes vs. columns Glue everything together using SQL The entire ER model remains a moving target Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 19 / 44
- 16. Tuck In! Clean Picture Abstraction Layer . . . of configured components Focus the goal to know what is operated Depth first search over all component configuration files Identify relationships (remember: there is no operation model at all) Clean up configuration when no reasonable relationships can be resolved or relationships are conflicting Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 20 / 44
- 17. Tuck In! Control Moo in practice Moo in practice It appears that the tools helping to do safe IoT device updating are the same tools helping to coordinate CI determining: MooX::Cmd helps separating concerns MooX::ConfigFromFile helps contribute ”divine wisdom” MooX::Options allow overriding ”divine wisdom” by ”individual wisdom” MooX::Log::Any feeds DBIx::LogAny Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 21 / 44
- 18. Tuck In! Control Moo in background Moo in background Manage database connections based on concerns Manage CI structures based on relations Manage Web-API integration Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 22 / 44
- 19. Tuck In! Reflecting Relationships Craziness Crazy I’m not crazy. My reality is just different than yours. The Cheshire Cat (Lewis Carroll) Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 23 / 44
- 20. Tuck In! Reflecting Relationships Harmonization Harmonize Craziness Practically any administrator had a different background regarding to the platform components thus a different picture of their relationships EPIC battles leads to common craziness ER model analysis sessions uncovered holes in picture Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 24 / 44
- 21. Tuck In! Meanwhile Civilized March Hare: Have some wine. (Alice looked all round the table, but there was nothing on it but tea.) Alice: I don’t see any wine. March Hare: There isn’t any. Alice: Then it wasn’t very civil of you to offer it. March Hare: It wasn’t very civil of you to sit down without being invited. (Lewis Carroll) Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 25 / 44
- 22. Tuck In! Meanwhile Adding a Goal CentOS 5 ends its maintenance Many of existing tools need to be upgraded Upgraded tools don’t support existing hacks anymore Existing hacks must be replaced by a reasonable configuration structure Same problem like the report format: ◮ neither the ER model of platform components nor issues of platform where known ◮ nor cared about Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 26 / 44
- 23. Tuck In! Meanwhile Self Protection Delegation We learned from mistakes of past: No responsibility taken for filling weird puppet templates No external data will be managed No precompiled/puzzled resources are prepared ⇒ ER model of CMDB is presented via RESTful API Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 27 / 44
- 24. Tuck In! Merging Pictures Scan completed Early implementation of above mentioned RESTful API run against ScanDB ScanDB represents just a view of the configuration snapshot There is no future, nor past Time for CMDB to enter the stage Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 28 / 44
- 25. Tuck In! Future and Past Customers . . . CREATE TABLE customer_t ( customer _id INTEGER PRIMARY KEY -- entity stuff , c ustomer_name VARCHAR (80) UNIQUE NOT NULL -- cmdb stuff , valid _from DATETIME NOT NULL , valid _to DATETIME , modified _at DATETIME NOT NULL , modified _by VARCHAR (32) NOT NULL ); primary key and global identifier for this data type the payload of this data type, automatically indexed CMDB manages history and updates using these columns Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 29 / 44
- 26. Tuck In! Future and Past VPN links to customers . . . CREATE TABLE vpn_link_t ( vpn_link_id INTEGER PRIMARY KEY -- entity stuff , customer_id INTEGER NOT NULL , v pn_link_type VARCHAR (12) , custom er_net VARCHAR (64) UNIQUE NOT NULL , servic es_net VARCHAR (64) UNIQUE NOT NULL -- cmdb stuff , valid_from DATETIME NOT NULL , valid_to DATETIME , modified_at DATETIME NOT NULL , modified_by VARCHAR (32) NOT NULL -- FK , FOREIGN KEY ( customer_id ) REFERENCES customer_t( customer_id ) ON UPDATE CASCADE ON DELETE CASCADE ); refer the customer support Cisco, Juniper, Paolo Alto, . . . networks must be unique or network admins kill you Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 30 / 44
- 27. Tuck In! Future and Past Moo Interception package Foo:: Role :: Database :: CMDB; use Moo:: Role; requires "log"; has cmdb => ( is => "ro", required => 1, handles => "Foo:: Role :: Database", isa => sub { _INSTANCE_OF($_[0], "Foo:: Helper :: CMDB") and $_[0]-> DOES("Foo:: Role :: Dat and return; die " Insufficient initialisation parameter for cmdb"; }, coerce => sub { _HASH($_ [0]) and return Foo :: Helper ::CMDB ->new (%{$_ [0]}); $_ [0]; }, ); role can be consumed by any class needing access to CMDB transform hash initializer into object Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 31 / 44
- 28. Tuck In! Future and Past Hard work package Foo:: Helper :: CMDB; use Moo; extends "Foo:: Helper :: DatabaseClass "; has config _tables => (is => "lazy", ...); has history_tabl es => (is => "lazy", ...); around deploy => sub { ... my @tables = @{$self -> config_tables }; foreach my $tbl (@tables) { my @h ist_coldefs = map { my $default = defined $_ - >[4] ? " DEFAULT $_ - >[4]" : ""; $pure_cols{$_ - >[1]} ? ("$_ - >[1] $_ - >[2] $default") : ("old_$_ - >[1] $_ - >[2] $default", "new_$_ - >[1] $_ - >[2] $default") } @table_info ; } unshift @hist_coldefs , "${base_name}_hist_id INTEGER PRIMARY KEY"; my $hist_defs = join("n , ", @hist_coldefs ); my $hist_tbl = <<EOCHT; CREATE TABLE ${base_name}_hist ( ${hist_defs} ); EOCHT that are all tables with trailing t in their names create history table for each config table memoizing old and new values on updating payload Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 32 / 44
- 29. Tuck In! Future and Past Hard work (continued) - INSERT my $new_cols = join(", ", map { $pure_cols{$_} ? $_ : "new_$_" } grep { !$skipped{$_} } @table_cols ); my $new_vals = join(", ", map {"NEW.$_"} grep { !$skipped{$_} } @table_cols ); my $new_trgr = <<EONT; CREATE TRIGGER new_${base_name}_row AFTER INSERT ON ${base_name}_t BEGIN INSERT INTO ${base_name}_hist ($new_cols) VALUES ($new_vals ); END; EONT ⇒ ON INSERT fill history rows without filling ”OLD ” columns Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 33 / 44
- 30. Tuck In! Future and Past Hard work (continued) - UPDATE my (@updt_cols , @rfrs_cond , @updt_vals ); foreach my $colnm (grep { !$skipped{$_} } @table_cols ) { my @lcd = $pure_cols{$colnm} ? $colnm : ("old_${colnm}", "new_${colnm}"); push @updt_cols , @lcd; push @rfrs_cond , $pure_cols{$colnm} ? _cmp_if_nullable ("existing.${colnm}", "NEW.${colnm}") : ( _cmp_if_nullable ("existing.old_${colnm}", "OLD.${colnm}"), _cmp_if_nullable ("existing.new_${colnm}", "NEW.${colnm}")); push @updt_vals , $pure_cols{$colnm }?("NEW.$colnm"):("OLD.$colnm","NEW.$colnm") } my $updt_cols = join(", ", @updt_cols ); my $updt_refreshed_cols = join(", ", map { "refreshed.$_" } @updt_cols ); my $updt_vals = join(", ", @updt_vals ); ⇒ Prepare for a bit complexer trigger ⇒ Distinguish between real updates and just ”touches” Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 34 / 44
- 31. Tuck In! Future and Past Hard work (continued) - UPDATE my $updt_trgr = <<EONT; CREATE TRIGGER updt_${base_name}_row AFTER UPDATE ON ${base_name}_t BEGIN INSERT OR REPLACE INTO ${base_name}_hist (${base_name}_hist_id , $updt_cols) VALUES ( (SELECT MAX(existing.${base_name}_hist_id) ${base_name}_hist_id FROM ${base_name}_hist existing WHERE $updt_refreshed_cond ), $updt_vals ); END; EONT ⇒ ON UPDATE create history (INSERT) rows with ”OLD ” and ”NEW ” columns except nothing changes (REPLACE) Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 35 / 44
- 32. Tuck In! Features ”UPSERT” MERGE INTO tablename USING table_reference ON (condition) WHEN MATCHED THEN UPDATE SET column1 = value1 [, column2 = value2 ...] WHEN NOT MATCHED THEN INSERT (column1 [, column2 ...]) VALUES (value1 [, value2 ...]); SQLite Unsupported by SQLite INSERT OR REPLACE deletes before insert → Kills UPDATE Trigger Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 36 / 44
- 33. Tuck In! Features Perl helps out $self ->cmdb ->upsert( customer_t => { customer_name => "Foo Enterprises ", }); $self ->cmdb ->upsert( vpn_link_t => { customer_name => "Foo Enterprises ", vpn_link_type => "Juniper", customer_net => " 10.116.47.8/29", services_net => " 10.126.47.8/29" } ); SQL created . . . INSERT OR IGNORE INTO vpn_link_t ( customer_id , vpn_link_type , customer_net , services_net , modified_by ) VALUES ( (SELECT customer_id FROM customer_t WHERE customer_name =?), ?, ?, ?, ?); UPDATE vpn_link_t SET vpn_link_type =?, customer_net =?, services_net =?, modified_by =?, touched_at= CURRENT_TIMESTAMP WHERE changes ()=0 AND customer_id =( SELECT customer_id FROM customer_t WHERE customer_name =?); Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 37 / 44
- 34. Tuck In! Features Known limitations Restricted to CMDB Refuse updates of identifying columns (UNIQUE constraints) WHERE clause derived from UNIQUE constraints Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 38 / 44
- 35. Tuck In! Meanwhile II CMDB to Hiera YAML Generator Development team read via RESTful API the theoretical configuration set Hiera YAML files are written Additional exports are managed via Hiera Puppet classes are rewritten to understand new ER model Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 39 / 44
- 36. Goals reached Circle closed Circle closed :platform :snapshot :scan db:cmdb :hiera :report collect scan process-scan yaml-gen puppet run CRQ export scan process-scan yaml-gen CRQ Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 41 / 44
- 37. Goals reached Circle closed Goals reached → Actual-theoretical comparison done via processing scan database → Unmaintainted installation via cronjob possible → Reaction in emergency cases by organized component moving done multiple times → Monitoring, sensors, alarming open Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 42 / 44
- 38. Conclusion Conclusion Can a programming language save a life Yes, it can - but here it saves our business Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 43 / 44
- 39. Thank you Thank you Thank You For Listening Questions? Jens Rehsack <rehsack@cpan.org> Cologne Jens Rehsack (Niederrhein.PM) CMDB Driven by Perl The Perl Conference in Amsterdam 44 / 44