SlideShare a Scribd company logo

OSDC 2016 - Automating a R&D lab with Foreman: What can be hard? by Julien Pivotto

NETWAYS, profile picture
NETWAYS

The document discusses managing a Research and Development (R&D) lab using Foreman, an open-source tool for managing physical and virtual servers across various platforms. It highlights Foreman's features for provisioning, configuration, and monitoring, including its flexibility and community-driven improvements. Additionally, it addresses installation processes and potential challenges faced in maintaining consistent system states.

Software
Related topics:
Data Center Overview
1 of 91
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
Managing a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with Foreman What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard? Julien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien Pivotto
whoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoami Julien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien Pivotto • Sysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.eu • FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004 • Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011 • DevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believer • @roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie on irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/github 1. I am a sysadmin 2. Doing Automation, Cfgmgmt, Monitoring 3. Work at Inuits 4. Contributor to Puppet, Foreman, and so on
inuits.eu 1. 60 people 2. In Belgium 3. The Netherlands 4. Ukraine 5. Cecz republic 6. Open Source Consulting 7. Development/System administration
Stephen Benjamin gave a talk about Foreman last year. This is not the same talk, for the following reasons: 1. While a contribute I am not a dev 2. I will give an intro but like 10 min 3. not affiliated/partner 4. Then I will speak about our experience
The Foreman The Foreman is an Open Source tool that can manage virtual machines and physical servers. It comes with a Web UI and an API It manages the whole lifecycle:
The Foreman Provisioning PROVISIONING 1. Provision physical/virtual 2. In Cloud services 3. In Libvirt/Baremetal 4. manages dhcp tftp dns
The Foreman Provisioning Configuration CONFIGURATION 1. Act as a Puppet ENC 2. Classifies hosts in groups 3. Add parameters to hosts
The Foreman Provisioning Configuration Monitoring MONITORING 1. Stores the reports from CM tools 2. Expose them 3. Store Puppet facts
The Foreman Provisioning Configuration Monitoring Reporting I prefer REPORTING over MONITORING
Foreman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choice • OOOOOOOOOOOOOOOOOpen-Source • LLLLLLLLLLLLLLLLLarge, active community • RRRRRRRRRRRRRRRRRest API and cli tools Foreman is a great tool 1. Open Source Minded 1.1 Accept patches (20) 1.2 Discuss publicly 1.3 Invest in community 1.4 Listen to community 1.5 REUSE libraries and other bricks 1.6 …so contributes to 3rd parties 1.7 Public testing CI infra (jenkins/travis) 2. Flexibility 3. Modularity: Plugins 4. Modularity: Disable everything 5. Abstraction: a "Middleware" 6. Mature but evolving
OSDC 2016 - Automating a R&D lab with Foreman: What can be hard? by Julien Pivotto
OSDC 2016 - Automating a R&D lab with Foreman: What can be hard? by Julien Pivotto
OSDC 2016 - Automating a R&D lab with Foreman: What can be hard? by Julien Pivotto
OSDC 2016 - Automating a R&D lab with Foreman: What can be hard? by Julien Pivotto
OSDC 2016 - Automating a R&D lab with Foreman: What can be hard? by Julien Pivotto
Behind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenes Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/reuver/10105949326
Foreman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overview Licensed under a Creative Commons Attribution-ShareAlike 3.0 License http://theforeman.org
Technology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stack • FFFFFFFFFFFFFFFFForeman-web: Ruby-on-Rails • SSSSSSSSSSSSSSSSSmart Proxies: Ruby • TTTTTTTTTTTTTTTTTLS everywhere Mainly ROR for frontend Workers are Smart Proxies They do the work TLS everywhere MySQL PGSQL tested in CI
Supported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologies • PPPPPPPPPPPPPPPPPuppet (other CM are WIP) • lllllllllllllllllibvirt (kvm), VMWare, ec2, openstack, ovirt • DDDDDDDDDDDDDDDDDHCP, DNS, TFTP, IPMI • VVVVVVVVVVVVVVVVVNC A lot of technos Some as plugins (DNS) Some less tested/used (VMWare) novnc support in the browser Of course you can mix them
PluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPlugins • KKKKKKKKKKKKKKKKKatello: Content management • RRRRRRRRRRRRRRRRRemote Execution • HHHHHHHHHHHHHHHHHubot • CCCCCCCCCCCCCCCCCockpit WIP at the foreman level to bring more visibility Foreman Katello are main part of RH satellite
UsecasesUsecasesUsecasesUsecasesUsecasesUsecasesUsecasesUsecasesUsecasesUsecasesUsecasesUsecasesUsecasesUsecasesUsecasesUsecasesUsecases Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/aydun/14108842993
Puppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporing • 22222222222222222011-2013: Puppet reporting • RRRRRRRRRRRRRRRRReplacement for Puppet Dashboard • AAAAAAAAAAAAAAAAAlready had a API, hooked into icinga You do not have to use the full stack
2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management • PPPPPPPPPPPPPPPPProvisioning: libvirt/vmware • CCCCCCCCCCCCCCCCConfig: Puppet Master+ENC • RRRRRRRRRRRRRRRRReporting DNS and DHCP DMZ almost everything very important in the lab puppet-* also puppet repo hosting
Our ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur Scale • 22222222222222222 countries • 44444444444444444 datacenter • 11111111111111111 Foreman • 55555555555555555 Foreman Proxies • 33333333333333333 Puppet env • 99999999999999999 Compute Resource (hypervisor) • 1111111111111111150 machines scale numbers what is interesting is that we have multiple physical ENV We use advanced Networking (bonds, bridges)
What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard? Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/130811041@N04/19114856463
foreman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installer Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/gabprr/8325699254
foreman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installerforeman-installer • FFFFFFFFFFFFFFFFForeman Installer is a package • UUUUUUUUUUUUUUUUUses Puppet behind the scene • IIIIIIIIIIIIIIIIInstalls and configure * The foreman installer is the recommended way (documented) to install TFM and to update it
How to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The Foreman • 11111111111111111. yum install foreman-installer • 22222222222222222. run foreman-installer • 33333333333333333. done very easy very convenient a -i option is available same procedure for proxies and web Configuration is done in a yaml file At the end you’re done
foreman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scope • FFFFFFFFFFFFFFFFForeman-web • FFFFFFFFFFFFFFFFForeman-proxy • PPPPPPPPPPPPPPPPPuppet master • DDDDDDDDDDDDDDDDDHCP, DNS, TFTP services • GGGGGGGGGGGGGGGGGit repositories • LLLLLLLLLLLLLLLLLink between those even hosting git repos publishing repos to the master etc etc
The chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problem Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/100739735@N06/15892201516
First foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman install • CCCCCCCCCCCCCCCCCreates everything needed to puppetize • gggggggggggggggggit repo, puppetmaster • bbbbbbbbbbbbbbbbbut not to puppetize itself The consequence of this is that in most of the places the installer is run then it is not run anymore and the state is not enforced it requires an effort to bring the module sin you need the good version you need t match the parameters and during upgrades you will need extra ca
State is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforced • SSSSSSSSSSSSSSSSSystem is not up to date • NNNNNNNNNNNNNNNNNo confidence that the state is still correct • SSSSSSSSSSSSSSSSSolution A: integrate within the puppet tree • SSSSSSSSSSSSSSSSSolution B: Re-run the foreman-installer but then you have other problems: consistency of answers files Update of DHCP prevents foreman-proxy t visibility of the results
Importing the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your tree • SSSSSSSSSSSSSSSSState is enforced • OOOOOOOOOOOOOOOOOnly thing to care about: updating the modules • TTTTTTTTTTTTTTTTThey are linked to the foreman if you update foreman without the modules e.g: the puppet-foreman report processor ENC script If you break it.. Foreman/Puppet will be br it is also a problem (report)
Building and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuilding Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/artbystevejohnson/6405400351
Building a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a host • CCCCCCCCCCCCCCCCCreate/cycle VM • CCCCCCCCCCCCCCCCCreate/change DNS Config • CCCCCCCCCCCCCCCCCreate/change DHCP lease • CCCCCCCCCCCCCCCCCreate/change TFTP files In a modern env, and especially a lab it matters as this operation costs, not done all the time only when you enable ‘build mode’ or edit the host
An expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operation • IIIIIIIIIIIIIIIIIf something is wrong, it rollbacks • SSSSSSSSSSSSSSSSSolution: Foreman 1.10 allows you to force config rebuild • HHHHHHHHHHHHHHHHHidden in the UI in the `All hosts' view it means: the UI can lie Foreman does not check if those settings are still valid Rollbacks DNS and it takes time before you see but that problem is solved since foreman 1. and there is a API call
DNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNS Licensed under a Creative Commons Attribution-ShareAlike 2.0 License https://www.flickr.com/photos/quinnanya/4464205726
Everything is a Freaking DNS Problem Kris Buytaert
Foreman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entries • KKKKKKKKKKKKKKKKKeeps the data consistent • CCCCCCCCCCCCCCCCCreates only hostnames that exist • AAAAAAAAAAAAAAAAAlso does the reverse entries Foreman can manage your dns entries IP Addresses/Config are correct keep tha data consistent you do not have to create the names upfro reverse DNS is also there
Adding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom records • AAAAAAAAAAAAAAAAAt some point you want CNAME records • ooooooooooooooooor change NS records The technical names you chose in FM CNAME = Canonical Name record are sometimes not enough Or juste name for the routers etc In our case we also updated the NS So the foreman-managed zones are not que directly, but we have redudant DNS in fron Foreman does not have that concept
Manipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zones • FFFFFFFFFFFFFFFFForeman uses Dynamic Zones • rrrrrrrrrrrrrrrrrndc freeze • CCCCCCCCCCCCCCCCChange the zone (incr the serial) • rrrrrrrrrrrrrrrrrndc thaw Foreman uses nsupdate to update the zone But usually it is easier for humans to work with files You can freeze the dynamic zone So you can do whatever you want
Org-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNS • YYYYYYYYYYYYYYYYYou own DNS servers should forward the unknown to your org • WWWWWWWWWWWWWWWWWe could not make it work for PTR records • NNNNNNNNNNNNNNNNNeed to set empty-zones-enable=no • PPPPPPPPPPPPPPPPPR theforeman/puppet-dns#47 Not directly related to foreman but still interesting takes some time to find that now available in foreman installer thx to our contribution Foreman is more that its core
DMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZ Licensed under a Creative Commons Attribution-ShareAlike 2.0 License https://www.flickr.com/photos/flintymcginty/9997879333
Managing hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the lab • SSSSSSSSSSSSSSSSSecurity purpose • RRRRRRRRRRRRRRRRRun Demos • HHHHHHHHHHHHHHHHHandover to other teams there are a bunch of rasons to have a DMZ but if you have to expose the foreman you loose the advantage of the DMZ of cou
Foreman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutions • DDDDDDDDDDDDDDDDDNS Proxy, DHCP proxy, TFTP… • KKKKKKKKKKKKKKKKKickstart proxying? • DDDDDDDDDDDDDDDDDigging into the documentation • FFFFFFFFFFFFFFFFFeature is there but not really visible • IIIIIIIIIIIIIIIIImprove docs: PR theforeman/theforeman.org#547 Foreman has the ability to serve the templates from proxies That way only proxies need access to the foreman feature was not in the proxies list so not sure if up to date or not now it is in the docs
DocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentation Licensed under a Creative Commons Attribution-ShareAlike 2.0 License https://www.flickr.com/photos/flyingblogspot/15361704293
DocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentation • TTTTTTTTTTTTTTTTThe Foreman documentation is huge • HHHHHHHHHHHHHHHHHosted on theforeman.org • IIIIIIIIIIIIIIIIImprove it so the next guy doesn't lose your time again We got a few trouble woth the docs Open-Source on Github so here is the story: docs are wrong ask for help Fix the doc if needed
A NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC Story • FFFFFFFFFFFFFFFFForeman embeds NoVNC • AAAAAAAAAAAAAAAAAllows you to see VNC console in browser • LLLLLLLLLLLLLLLLLots of problems with certificates So that feature is cool After an upgrade it stopped working Browser and FM config We followed then the docs Did not workkkk
Details mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails matters • IIIIIIIIIIIIIIIIIn the 1.7 docs: websockets_encrypt: true • LLLLLLLLLLLLLLLLLet's change it to false • PPPPPPPPPPPPPPPPProblem: true/false vs on/off • EEEEEEEEEEEEEEEEExtra work: Updated the docs It does not happen so often but reviewing the doc is important Fixing is as important I hope we helped there too This is an exception
Release NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease Notes • RRRRRRRRRRRRRRRRRelease notes are part of Documentation • WWWWWWWWWWWWWWWWWhen you change behaviour, think about others • LLLLLLLLLLLLLLLLLower update cost Keeping RN up to date with your breaking contrib is very important it lower the prices of upgrading because it removes uncertainty Couple your changes with a change in the R So YOU will remember; but other too MANY USECASES
ScalabilityScalabilityScalabilityScalabilityScalabilityScalabilityScalabilityScalabilityScalabilityScalabilityScalabilityScalabilityScalabilityScalabilityScalabilityScalabilityScalability Licensed under a Creative Commons Attribution-ShareAlike 2.0 License https://www.flickr.com/photos/21499502@N04/13905270178
Scaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tips • EEEEEEEEEEEEEEEEEnable organizations/locations • UUUUUUUUUUUUUUUUUse one foreman for separated entities • UUUUUUUUUUUUUUUUUse one foreman for separated countries • UUUUUUUUUUUUUUUUUse foreman proxies where needed Org and Locations are great Isolation Maintanability Visibility ACL
AssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociation • OOOOOOOOOOOOOOOOOne terrible thing in the Foreman • AAAAAAAAAAAAAAAAAssigning things to each other (but hey it gets better) (or we got used to it)
AssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociation • OOOOOOOOOOOOOOOOOS -> Location and Org • CCCCCCCCCCCCCCCCCompute Resource -> Location and Org • TTTTTTTTTTTTTTTTTemplates -> Location and Org • OOOOOOOOOOOOOOOOOS -> Installation Media • IIIIIIIIIIIIIIIIInstallation media -> Location and Org This is a subset When you START using org it is a pain When you move hosts between now it corre Makes you lose time Price for isolation
Scaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute Resources • CCCCCCCCCCCCCCCCCompute Resources = Virtualization or Cloud Service • EEEEEEEEEEEEEEEEEasy to create hosts in those CR • lllllllllllllllllibvirt, ec2, gce, vmware… We mainly use libvirt And also VMWARE but I will come to vmware later
Scaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling Libvirt • WWWWWWWWWWWWWWWWWe have 8 libvirt servers • TTTTTTTTTTTTTTTTTo create a VM, you chose CPU, RAM, disks… • AAAAAAAAAAAAAAAAAny change requires deletion and creation You can also change in libvirt you can not change after the host creation But this is fragile We generally prefer to change in the forem deletion, not rebuild Hello API deleting a host in the foreman is also rude
Creating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hosts • HHHHHHHHHHHHHHHHHostgroups • EEEEEEEEEEEEEEEEEnvironments • CCCCCCCCCCCCCCCCCompute profiles? Foreman has a lot of mechanisms to pre-populate a lot of fields For the VM specific fields, the mechanism is Compute Profiles
Libvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute Profiles • CCCCCCCCCCCCCCCCCPU • MMMMMMMMMMMMMMMMMemory • NNNNNNNNNNNNNNNNNIC • SSSSSSSSSSSSSSSSStorage • MMMMMMMMMMMMMMMMMultiple profiles per hypervisor
What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then? • LLLLLLLLLLLLLLLLLibvirt servers are not a group • TTTTTTTTTTTTTTTTThey are separated Compute Resources • AAAAAAAAAAAAAAAAA lot of work (UI and API) even of you want to sync then with API difficult from an API POV you will need to make a script this is not a 5 minutes work then you will force you to keep it uptodate WE HAVE 8 HOSTS
Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific? • YYYYYYYYYYYYYYYYYes: other providers are `centralized' • eeeeeeeeeeeeeeeeec2, gce, openstack… We were thinking about ovirt as a possible workaround but we quickly abandoned the idea even if we are probably close to the border so we keep the profiles in sync this is not good and creates complexity and inconsistency
VMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWare • SSSSSSSSSSSSSSSSSome part of the lab runs VMWare • GGGGGGGGGGGGGGGGGreat: Foreman supports VMWare • (((((((((((((((((on paper) We had the opportinity to integrate VMWa It was there before and there is support from the foreman of course we tried to integrate with mitigated success
VMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integration • MMMMMMMMMMMMMMMMMost of the settings, Foreman will take user defaults • FFFFFFFFFFFFFFFFForeman is not for VMWare Power users • SSSSSSSSSSSSSSSSSounds like a lack of interest from community No interest in community we reached basic problems
Logging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging matters • FFFFFFFFFFFFFFFFForeman requires on Fog • FFFFFFFFFFFFFFFFFog is a gem for the `clouds' • FFFFFFFFFFFFFFFFFog for vmware is not as advances as we'd like I do not BLAME fog This is just to show the lakc of interest and support One example: during first try, we got a stra ‘Wrong Credentials’ Solution: Wrong PERMISSIONS. We found that by using another client I would not recommend prople to use vmwa
Empowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the Developers Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/julesdphotographie/8421289746
Distributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the power • RRRRRRRRRRRRRRRRRebuilding a host is simple • OOOOOOOOOOOOOOOOOne clic operation • FFFFFFFFFFFFFFFFForeman as a VM shop we do not need to be in the middle devs should be autonomous devops mouvements
The DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movement • DDDDDDDDDDDDDDDDDevOps is a movement born in 2009 • CCCCCCCCCCCCCCCCCollaboration between Developers and Operations • NNNNNNNNNNNNNNNNNothing new, just common sense • DDDDDDDDDDDDDDDDDevOpsDays, a serie of conferences all around the world devops movement buzz word actually something behind getting ops and dev to work together a community that meet in different places solving challenges still what is this
#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS • CCCCCCCCCCCCCCCCCulture • AAAAAAAAAAAAAAAAAutomation • MMMMMMMMMMMMMMMMMeasurement • SSSSSSSSSSSSSSSSSharing John Willis and Damon Edwards DevOps is CAMS Definition from 2010 it is a cultural and profesionnal getting everyone to work together reduce time to business
The C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMS • DDDDDDDDDDDDDDDDDevOps is a Cultural change • EEEEEEEEEEEEEEEEEveryone is in the team • OOOOOOOOOOOOOOOOOps and Devs work together • SSSSSSSSSSSSSSSSShare the responsibilities Sharing the responsibilities also means give more power and play the same game No being in the middle
Foreman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the picture • FFFFFFFFFFFFFFFFForeman empowers the developers • WWWWWWWWWWWWWWWWWhile still providing enough security • OOOOOOOOOOOOOOOOOrganisations in Foreman The Organisations in foreman allow you to split the quickly rebuildable VM Advanced ACL from the core of your infra for security and simplicity
Developer ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACL Licensed under a Creative Commons Attribution-ShareAlike 3.0 License Screenshot
The Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer side • OOOOOOOOOOOOOOOOOne-clic rebuild (at will) • AAAAAAAAAAAAAAAAAccess to build reports • IIIIIIIIIIIIIIIIIn-browser access to VNC • FFFFFFFFFFFFFFFFFresh vm in minutes For the developers they get almost the same advantages as with vagran except that we can provide more VMs/RAM could have on the laptop also we centralize and do not give direct ac to libvirt credential. Side note: foreman is hooked to LDAP
The Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops side • KKKKKKKKKKKKKKKKKeep everything under control • AAAAAAAAAAAAAAAAAudit logs, reports • IIIIIIIIIIIIIIIIInventory • BBBBBBBBBBBBBBBBBring regular updates to VM's/developers the ops can then maintain the kickstart/pu devs will get the changes immediatly or at audit logs to know who did what when
UpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdates Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/inyucho/7866698878
Foreman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updates • AAAAAAAAAAAAAAAAAs any software: • TTTTTTTTTTTTTTTTTake a backup first • TTTTTTTTTTTTTTTTTry on your dev environment • FFFFFFFFFFFFFFFFFirst upgrade the main UI • TTTTTTTTTTTTTTTTThen update the proxies (#12506) Foreman is a regular application If upgrade hurts, do it more often e.g each minor release We DO NOT DO 0 releases
Upgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problems • 11111111111111111.5->1.6: DHCP config file not readable by foreman proxy WWWWWWWWWWWWWWWWWe did a DHCP upgrade at the same time FFFFFFFFFFFFFFFFFile ownership was changed rrrrrrrrrrrrrrrrre-run the foreman installer fixed it
Upgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problems • 11111111111111111.5->1.6: Puppet reports not coming into the foreman FFFFFFFFFFFFFFFFForeman report preprocessor has changed NNNNNNNNNNNNNNNNNeed to update foreman.rb + config nnnnnnnnnnnnnnnnnode.rb also needed an update
Upgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problems • 11111111111111111.7->1.8: Big stack trace on opening the UI yyyyyyyyyyyyyyyyyum erase ruby193-rubygem-foreman_openstack_cluster ruby193-rubygem-foreman_openstack_cluster-doc A subset of troubles we had when upgradin We get less and less of these
Upgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problems • 11111111111111111.8->1.9: Yaml transformations in Puppet parameters FFFFFFFFFFFFFFFFFixed in 1.10, backported to 1.9
Community and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-Source Licensed under a Creative Commons Attribution-2.0 License https://www.flickr.com/photos/maicosnap/4393929855
Foreman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-Source • GGGGGGGGGGGGGGGGGPLv3+ • WWWWWWWWWWWWWWWWWe've got around 20 patches integrated • TTTTTTTTTTTTTTTTThey rely mostly on open-source tools • CCCCCCCCCCCCCCCCCI, testing, bugtracker This is a HUMAN-SCALE OS Project THAT WORKS Listens to community Accept improvements, bugfixes, features Plus it is modular (plugins) you can write plugins for anything
Story: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global Status • PPPPPPPPPPPPPPPPPut in build mode, go back home • RRRRRRRRRRRRRRRRRestart on the next day • LLLLLLLLLLLLLLLLLooping reboots Enabling build mode does not force restart so I did enable the build and try to rebuild the next day but the toek was expired so the host kept reinstalling then we figured out the problem
Foreman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global status • FFFFFFFFFFFFFFFFForeman keeps several status for hosts • PPPPPPPPPPPPPPPPPlugins can register a status • BBBBBBBBBBBBBBBBBut reporting and provisioning also • LLLLLLLLLLLLLLLLLet's use that for expired tokens (bug#5883) Now the status of reporting is different tha it means that we could get a status if token
SolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutions • WWWWWWWWWWWWWWWWWrite a patch, keep it • WWWWWWWWWWWWWWWWWrite a plugin • WWWWWWWWWWWWWWWWWrite and upstream a patch Since it is core stuff we wrote a patch because this is not only for us when token expired you cant build anymore DOT. We wrote and upstreamed a patch
Advantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreaming • FFFFFFFFFFFFFFFFFeature will be available in next releases • WWWWWWWWWWWWWWWWWe won't patch in the future • CCCCCCCCCCCCCCCCCalls in the patch will evolve as well • PPPPPPPPPPPPPPPPPeer review with the Foreman team • FFFFFFFFFFFFFFFFForced to write tests, doc, etc… The fact to upstream the patch reduce our work in the future and forces un to write tests and run tests on a larger infra than ours NOT THE END OF THE STORY
2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after • PPPPPPPPPPPPPPPPPatch is accepted for 2 weeks • AAAAAAAAAAAAAAAAA new bug is opened: performance problem (bug#14050) • SSSSSSSSSSSSSSSSSomeone else fixed the bug • WWWWWWWWWWWWWWWWWe applied the second patch A perf bug was opened still far before delivery we would not have known without upstream it was patched before we could have the tim thats the beauty of open source
ConclusionConclusionConclusionConclusionConclusionConclusionConclusionConclusionConclusionConclusionConclusionConclusionConclusionConclusionConclusionConclusionConclusion Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/trevor-king/12534585293
Where can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improve • PPPPPPPPPPPPPPPPPerformance • SSSSSSSSSSSSSSSSSome features are missing from API • DDDDDDDDDDDDDDDDDecoupling from Puppet • …………………………………………… Some area I would like to see changes some of them are already reworked
Where did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improve • TTTTTTTTTTTTTTTTTests reliability • TTTTTTTTTTTTTTTTTesting Infra • CCCCCCCCCCCCCCCCCommunity • SSSSSSSSSSSSSSSSStability Foreman improved in may areas recently Community and infra changes are great CI results are meaningful
The ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe Foreman • SSSSSSSSSSSSSSSSStability, Maturity, Flexibility • MMMMMMMMMMMMMMMMMade out of bricks: complex but not complicated • IIIIIIIIIIIIIIIIIt meets our needs • IIIIIIIIIIIIIIIIIt is very active • PPPPPPPPPPPPPPPPPeople are welcoming Its professional A lot of activity Public demo/reviews on youtube
Any Question?
ContactContactContactContactContactContactContactContactContactContactContactContactContactContactContactContactContact Julien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien Pivotto julien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eu @roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie inuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuits https://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.eu info@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.eu +32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636

More Related Content

PPTX
What Mr. Spock would possibly say about modern unit testing: pragmatic and em...
Yaroslav Yermilov
 
PDF
Destruction, Decapods and Doughnuts: Continuous Delivery for Audio & Video Fa...
Rachel Evans
 
PDF
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
NETWAYS
 
PDF
OSDC 2016 - Tuning Linux for your Database by Colin Charles
NETWAYS
 
PDF
ODSC 2016 - Scalable Systems Management with Salt Stack by Sebastian Meyer
NETWAYS
 
PDF
OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...
NETWAYS
 
PDF
OSDC 2016 - Interesting things you can do with ZFS by Allan Jude&Benedict Reu...
NETWAYS
 
PDF
OSDC 2016 - Scalable Systems Management with Salt Stack by Sebastian Meyer
NETWAYS
 
What Mr. Spock would possibly say about modern unit testing: pragmatic and em...
Yaroslav Yermilov
 
Destruction, Decapods and Doughnuts: Continuous Delivery for Audio & Video Fa...
Rachel Evans
 
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
NETWAYS
 
OSDC 2016 - Tuning Linux for your Database by Colin Charles
NETWAYS
 
ODSC 2016 - Scalable Systems Management with Salt Stack by Sebastian Meyer
NETWAYS
 
OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...
NETWAYS
 
OSDC 2016 - Interesting things you can do with ZFS by Allan Jude&Benedict Reu...
NETWAYS
 
OSDC 2016 - Scalable Systems Management with Salt Stack by Sebastian Meyer
NETWAYS
 

Viewers also liked (20)

PDF
OSDC 2016 - Inspecting Security of Docker formatted Container Images to find ...
NETWAYS
 
PDF
OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes
NETWAYS
 
PDF
OSDC 2016 - Hello Redfish, goodbye IPMI by Werner Fischer
NETWAYS
 
PDF
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
NETWAYS
 
PDF
OSDC 2016 - Configuration Management for Cloud Services by Martin Schütte
NETWAYS
 
PDF
OSDC 2016 - Mesos and the Architecture of the New Datacenter by Jörg Schad
NETWAYS
 
PPTX
Apresentaço blueprinter main ppt 16;9
Augusto Nogueira
 
PDF
OSDC 2016 - DNS for Developers by Jan-Piet Mens
NETWAYS
 
PDF
OSDC 2016 - Kaiten Zushi - Chef at Goodgame Studios by Jan Ulferts
NETWAYS
 
PDF
OSDC 2016 - MySQL-Server in Teamwork - Replication and Galera Cluster by Jörg...
NETWAYS
 
PDF
Présentation Handimap - B-ware - 2012-11-12x
AFEIT
 
PDF
Présentation Yvan Galisson (Timwi)
AFEIT
 
PDF
Presentatio mazadoo opencoffee_brest
AFEIT
 
PPT
Carnet de campagne 2.0
AFEIT
 
PDF
Thd mobile brest-redx
AFEIT
 
PDF
JiVé Creation
AFEIT
 
PPTX
Présentation "mort 2.0"
AFEIT
 
PDF
Rdv Qualite AFEIT - Référentiel MASE
AFEIT
 
PPTX
EL ACCESO A INTERNET COMO DERECHO HUMANO
Irupe LB
 
PDF
OSDC 2016 - Introduction to Testing Puppet Modules by David Schmitt
NETWAYS
 
OSDC 2016 - Inspecting Security of Docker formatted Container Images to find ...
NETWAYS
 
OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes
NETWAYS
 
OSDC 2016 - Hello Redfish, goodbye IPMI by Werner Fischer
NETWAYS
 
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
NETWAYS
 
OSDC 2016 - Configuration Management for Cloud Services by Martin Schütte
NETWAYS
 
OSDC 2016 - Mesos and the Architecture of the New Datacenter by Jörg Schad
NETWAYS
 
Apresentaço blueprinter main ppt 16;9
Augusto Nogueira
 
OSDC 2016 - DNS for Developers by Jan-Piet Mens
NETWAYS
 
OSDC 2016 - Kaiten Zushi - Chef at Goodgame Studios by Jan Ulferts
NETWAYS
 
OSDC 2016 - MySQL-Server in Teamwork - Replication and Galera Cluster by Jörg...
NETWAYS
 
Présentation Handimap - B-ware - 2012-11-12x
AFEIT
 
Présentation Yvan Galisson (Timwi)
AFEIT
 
Presentatio mazadoo opencoffee_brest
AFEIT
 
Carnet de campagne 2.0
AFEIT
 
Thd mobile brest-redx
AFEIT
 
JiVé Creation
AFEIT
 
Présentation "mort 2.0"
AFEIT
 
Rdv Qualite AFEIT - Référentiel MASE
AFEIT
 
EL ACCESO A INTERNET COMO DERECHO HUMANO
Irupe LB
 
OSDC 2016 - Introduction to Testing Puppet Modules by David Schmitt
NETWAYS
 
Ad

Similar to OSDC 2016 - Automating a R&D lab with Foreman: What can be hard? by Julien Pivotto (20)

KEY
Pinto+Stratopan+Love
Jeffrey Ryan Thalhammer
 
PDF
Call Control Power Tools with Adhearsion
Mojo Lingo
 
ODP
From Config Management Sucks to #cfgmgmtlove
Kris Buytaert
 
KEY
PL/Parrot San Francisco Perl Mongers 2010/05/25
David Fetter
 
KEY
UCLA HACKU'11
Gopal Venkatesan
 
PPT
Linux Users are People, Too!
Charles Southerland
 
PDF
FIWARE Global Summit - FIROS: Helping Robots to be Context Aware
FIWARE
 
KEY
The One Way
Daniel Greenfeld
 
PDF
Frequently asked questions answered frequently - but now for the last time
Andreas Jung
 
KEY
Call Control Power Tools with Adhearsion
Adhearsion Foundation
 
KEY
A Look at Plone 4
Eric Steele
 
PDF
2021.laravelconf.tw.slides1
LiviaLiaoFontech
 
PDF
6 reasons Jubilee could be a Rubyist's new best friend
Forrest Chang
 
KEY
WebGL Camp 4 - A3 3D Engine
aerotwist
 
PDF
Using Go in DevOps
Eficode
 
PDF
Natalie Pistunovich - Using Go In Dev Ops
Fraugster
 
PDF
Test driven infrastructure development (2 - puppetconf 2013 edition)
Tomas Doran
 
PDF
200,000 Lines Later: Our Journey to Manageable Puppet Code
David Danzilio
 
PDF
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Downtown San J...
Baruch Sadogursky
 
KEY
Le PERL est mort
apeiron
 
Pinto+Stratopan+Love
Jeffrey Ryan Thalhammer
 
Call Control Power Tools with Adhearsion
Mojo Lingo
 
From Config Management Sucks to #cfgmgmtlove
Kris Buytaert
 
PL/Parrot San Francisco Perl Mongers 2010/05/25
David Fetter
 
UCLA HACKU'11
Gopal Venkatesan
 
Linux Users are People, Too!
Charles Southerland
 
FIWARE Global Summit - FIROS: Helping Robots to be Context Aware
FIWARE
 
The One Way
Daniel Greenfeld
 
Frequently asked questions answered frequently - but now for the last time
Andreas Jung
 
Call Control Power Tools with Adhearsion
Adhearsion Foundation
 
A Look at Plone 4
Eric Steele
 
2021.laravelconf.tw.slides1
LiviaLiaoFontech
 
6 reasons Jubilee could be a Rubyist's new best friend
Forrest Chang
 
WebGL Camp 4 - A3 3D Engine
aerotwist
 
Using Go in DevOps
Eficode
 
Natalie Pistunovich - Using Go In Dev Ops
Fraugster
 
Test driven infrastructure development (2 - puppetconf 2013 edition)
Tomas Doran
 
200,000 Lines Later: Our Journey to Manageable Puppet Code
David Danzilio
 
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Downtown San J...
Baruch Sadogursky
 
Le PERL est mort
apeiron
 
Ad

Recently uploaded (20)

PDF
Complete Guide to Website Development in Malaysia for SMEs
Asian Business Services & Technologies
 
PDF
Ableton Live Suite for MacOS Crack Full Download (Latest 2025)
hashmi66g66
 
PPTX
"Secure File Sharing Solutions on AWS".pptx
sudharani74442
 
PDF
wealthsignaloriginal-com-DS-text-... (1).pdf
skmiani786
 
PDF
DNT Brochure 2025 – ISV Solutions @ D365
sai vignesh
 
PPTX
AMADEUS TRAVEL AGENT SOFTWARE | AMADEUS TICKETING SYSTEM
philipnathen82
 
PDF
STL Containers in C++ : Sequence Container : Vector
Mohammed Sikander
 
PDF
Cost to Outsource Software Development in 2025
Omega Incorporations- Best Software Development Company
 
PDF
How to Make Money in the Metaverse_ Top Strategies for Beginners.pdf
Herond Labs
 
PPTX
WiFi Honeypot Detecscfddssdffsedfseztor.pptx
singlesrihari
 
PPTX
Patient Appointment Booking in Odoo with online payment
AxisTechnolabs
 
PPTX
Log360_SIEM_Solutions Overview PPT_Feb 2020.pptx
nonameist3434
 
PDF
Types of Token_ From Utility to Security.pdf
Herond Labs
 
DOCX
How to Use SharePoint as an ISO-Compliant Document Management System
Sharepoint Designs
 
PPTX
Introduction to Windows Operating System
ssuser1028f8
 
PPTX
Computer Software and OS of computer science of grade 11.pptx
GauravDahal9
 
PPTX
Monitoring Stack: Grafana, Loki & Promtail
GhanshyamSwami3
 
PDF
Salesforce Agentforce AI Implementation.pdf
Robert Mark
 
PDF
How AI/LLM recommend to you ? GDG meetup 16 Aug by Fariman Guliev
HusseinMalikMammadli
 
DOCX
Greta — No-Code AI for Building Full-Stack Web & Mobile Apps
niloyislam1187
 
Complete Guide to Website Development in Malaysia for SMEs
Asian Business Services & Technologies
 
Ableton Live Suite for MacOS Crack Full Download (Latest 2025)
hashmi66g66
 
"Secure File Sharing Solutions on AWS".pptx
sudharani74442
 
wealthsignaloriginal-com-DS-text-... (1).pdf
skmiani786
 
DNT Brochure 2025 – ISV Solutions @ D365
sai vignesh
 
AMADEUS TRAVEL AGENT SOFTWARE | AMADEUS TICKETING SYSTEM
philipnathen82
 
STL Containers in C++ : Sequence Container : Vector
Mohammed Sikander
 
Cost to Outsource Software Development in 2025
Omega Incorporations- Best Software Development Company
 
How to Make Money in the Metaverse_ Top Strategies for Beginners.pdf
Herond Labs
 
WiFi Honeypot Detecscfddssdffsedfseztor.pptx
singlesrihari
 
Patient Appointment Booking in Odoo with online payment
AxisTechnolabs
 
Log360_SIEM_Solutions Overview PPT_Feb 2020.pptx
nonameist3434
 
Types of Token_ From Utility to Security.pdf
Herond Labs
 
How to Use SharePoint as an ISO-Compliant Document Management System
Sharepoint Designs
 
Introduction to Windows Operating System
ssuser1028f8
 
Computer Software and OS of computer science of grade 11.pptx
GauravDahal9
 
Monitoring Stack: Grafana, Loki & Promtail
GhanshyamSwami3
 
Salesforce Agentforce AI Implementation.pdf
Robert Mark
 
How AI/LLM recommend to you ? GDG meetup 16 Aug by Fariman Guliev
HusseinMalikMammadli
 
Greta — No-Code AI for Building Full-Stack Web & Mobile Apps
niloyislam1187
 

OSDC 2016 - Automating a R&D lab with Foreman: What can be hard? by Julien Pivotto

  • 1. Managing a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with Foreman What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard? Julien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien Pivotto
  • 2. whoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoami Julien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien Pivotto • Sysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.eu • FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004 • Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011 • DevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believer • @roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie on irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/github 1. I am a sysadmin 2. Doing Automation, Cfgmgmt, Monitoring 3. Work at Inuits 4. Contributor to Puppet, Foreman, and so on
  • 3. inuits.eu 1. 60 people 2. In Belgium 3. The Netherlands 4. Ukraine 5. Cecz republic 6. Open Source Consulting 7. Development/System administration
  • 4. Stephen Benjamin gave a talk about Foreman last year. This is not the same talk, for the following reasons: 1. While a contribute I am not a dev 2. I will give an intro but like 10 min 3. not affiliated/partner 4. Then I will speak about our experience
  • 5. The Foreman The Foreman is an Open Source tool that can manage virtual machines and physical servers. It comes with a Web UI and an API It manages the whole lifecycle:
  • 6. The Foreman Provisioning PROVISIONING 1. Provision physical/virtual 2. In Cloud services 3. In Libvirt/Baremetal 4. manages dhcp tftp dns
  • 7. The Foreman Provisioning Configuration CONFIGURATION 1. Act as a Puppet ENC 2. Classifies hosts in groups 3. Add parameters to hosts
  • 8. The Foreman Provisioning Configuration Monitoring MONITORING 1. Stores the reports from CM tools 2. Expose them 3. Store Puppet facts
  • 10. Foreman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choice • OOOOOOOOOOOOOOOOOpen-Source • LLLLLLLLLLLLLLLLLarge, active community • RRRRRRRRRRRRRRRRRest API and cli tools Foreman is a great tool 1. Open Source Minded 1.1 Accept patches (20) 1.2 Discuss publicly 1.3 Invest in community 1.4 Listen to community 1.5 REUSE libraries and other bricks 1.6 …so contributes to 3rd parties 1.7 Public testing CI infra (jenkins/travis) 2. Flexibility 3. Modularity: Plugins 4. Modularity: Disable everything 5. Abstraction: a "Middleware" 6. Mature but evolving
  • 16. Behind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenes Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/reuver/10105949326
  • 17. Foreman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overviewForeman overview Licensed under a Creative Commons Attribution-ShareAlike 3.0 License http://theforeman.org
  • 18. Technology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stackTechnology stack • FFFFFFFFFFFFFFFFForeman-web: Ruby-on-Rails • SSSSSSSSSSSSSSSSSmart Proxies: Ruby • TTTTTTTTTTTTTTTTTLS everywhere Mainly ROR for frontend Workers are Smart Proxies They do the work TLS everywhere MySQL PGSQL tested in CI
  • 19. Supported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologiesSupported technologies • PPPPPPPPPPPPPPPPPuppet (other CM are WIP) • lllllllllllllllllibvirt (kvm), VMWare, ec2, openstack, ovirt • DDDDDDDDDDDDDDDDDHCP, DNS, TFTP, IPMI • VVVVVVVVVVVVVVVVVNC A lot of technos Some as plugins (DNS) Some less tested/used (VMWare) novnc support in the browser Of course you can mix them
  • 20. PluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPluginsPlugins • KKKKKKKKKKKKKKKKKatello: Content management • RRRRRRRRRRRRRRRRRemote Execution • HHHHHHHHHHHHHHHHHubot • CCCCCCCCCCCCCCCCCockpit WIP at the foreman level to bring more visibility Foreman Katello are main part of RH satellite
  • 22. Puppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporingPuppet reporing • 22222222222222222011-2013: Puppet reporting • RRRRRRRRRRRRRRRRReplacement for Puppet Dashboard • AAAAAAAAAAAAAAAAAlready had a API, hooked into icinga You do not have to use the full stack
  • 23. 2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management2013-now: Lab management • PPPPPPPPPPPPPPPPProvisioning: libvirt/vmware • CCCCCCCCCCCCCCCCConfig: Puppet Master+ENC • RRRRRRRRRRRRRRRRReporting DNS and DHCP DMZ almost everything very important in the lab puppet-* also puppet repo hosting
  • 24. Our ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur ScaleOur Scale • 22222222222222222 countries • 44444444444444444 datacenter • 11111111111111111 Foreman • 55555555555555555 Foreman Proxies • 33333333333333333 Puppet env • 99999999999999999 Compute Resource (hypervisor) • 1111111111111111150 machines scale numbers what is interesting is that we have multiple physical ENV We use advanced Networking (bonds, bridges)
  • 25. What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard? Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/130811041@N04/19114856463
  • 28. How to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The Foreman • 11111111111111111. yum install foreman-installer • 22222222222222222. run foreman-installer • 33333333333333333. done very easy very convenient a -i option is available same procedure for proxies and web Configuration is done in a yaml file At the end you’re done
  • 29. foreman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scopeforeman-installer scope • FFFFFFFFFFFFFFFFForeman-web • FFFFFFFFFFFFFFFFForeman-proxy • PPPPPPPPPPPPPPPPPuppet master • DDDDDDDDDDDDDDDDDHCP, DNS, TFTP services • GGGGGGGGGGGGGGGGGit repositories • LLLLLLLLLLLLLLLLLink between those even hosting git repos publishing repos to the master etc etc
  • 30. The chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problemThe chicken-and-egg problem Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/100739735@N06/15892201516
  • 31. First foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman installFirst foreman install • CCCCCCCCCCCCCCCCCreates everything needed to puppetize • gggggggggggggggggit repo, puppetmaster • bbbbbbbbbbbbbbbbbut not to puppetize itself The consequence of this is that in most of the places the installer is run then it is not run anymore and the state is not enforced it requires an effort to bring the module sin you need the good version you need t match the parameters and during upgrades you will need extra ca
  • 32. State is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforced • SSSSSSSSSSSSSSSSSystem is not up to date • NNNNNNNNNNNNNNNNNo confidence that the state is still correct • SSSSSSSSSSSSSSSSSolution A: integrate within the puppet tree • SSSSSSSSSSSSSSSSSolution B: Re-run the foreman-installer but then you have other problems: consistency of answers files Update of DHCP prevents foreman-proxy t visibility of the results
  • 33. Importing the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your tree • SSSSSSSSSSSSSSSSState is enforced • OOOOOOOOOOOOOOOOOnly thing to care about: updating the modules • TTTTTTTTTTTTTTTTThey are linked to the foreman if you update foreman without the modules e.g: the puppet-foreman report processor ENC script If you break it.. Foreman/Puppet will be br it is also a problem (report)
  • 34. Building and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuilding Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/artbystevejohnson/6405400351
  • 35. Building a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a host • CCCCCCCCCCCCCCCCCreate/cycle VM • CCCCCCCCCCCCCCCCCreate/change DNS Config • CCCCCCCCCCCCCCCCCreate/change DHCP lease • CCCCCCCCCCCCCCCCCreate/change TFTP files In a modern env, and especially a lab it matters as this operation costs, not done all the time only when you enable ‘build mode’ or edit the host
  • 36. An expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operation • IIIIIIIIIIIIIIIIIf something is wrong, it rollbacks • SSSSSSSSSSSSSSSSSolution: Foreman 1.10 allows you to force config rebuild • HHHHHHHHHHHHHHHHHidden in the UI in the `All hosts' view it means: the UI can lie Foreman does not check if those settings are still valid Rollbacks DNS and it takes time before you see but that problem is solved since foreman 1. and there is a API call
  • 37. DNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNSDNS Licensed under a Creative Commons Attribution-ShareAlike 2.0 License https://www.flickr.com/photos/quinnanya/4464205726
  • 38. Everything is a Freaking DNS Problem Kris Buytaert
  • 39. Foreman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entries • KKKKKKKKKKKKKKKKKeeps the data consistent • CCCCCCCCCCCCCCCCCreates only hostnames that exist • AAAAAAAAAAAAAAAAAlso does the reverse entries Foreman can manage your dns entries IP Addresses/Config are correct keep tha data consistent you do not have to create the names upfro reverse DNS is also there
  • 40. Adding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom records • AAAAAAAAAAAAAAAAAt some point you want CNAME records • ooooooooooooooooor change NS records The technical names you chose in FM CNAME = Canonical Name record are sometimes not enough Or juste name for the routers etc In our case we also updated the NS So the foreman-managed zones are not que directly, but we have redudant DNS in fron Foreman does not have that concept
  • 41. Manipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zones • FFFFFFFFFFFFFFFFForeman uses Dynamic Zones • rrrrrrrrrrrrrrrrrndc freeze • CCCCCCCCCCCCCCCCChange the zone (incr the serial) • rrrrrrrrrrrrrrrrrndc thaw Foreman uses nsupdate to update the zone But usually it is easier for humans to work with files You can freeze the dynamic zone So you can do whatever you want
  • 42. Org-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNS • YYYYYYYYYYYYYYYYYou own DNS servers should forward the unknown to your org • WWWWWWWWWWWWWWWWWe could not make it work for PTR records • NNNNNNNNNNNNNNNNNeed to set empty-zones-enable=no • PPPPPPPPPPPPPPPPPR theforeman/puppet-dns#47 Not directly related to foreman but still interesting takes some time to find that now available in foreman installer thx to our contribution Foreman is more that its core
  • 43. DMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZDMZ Licensed under a Creative Commons Attribution-ShareAlike 2.0 License https://www.flickr.com/photos/flintymcginty/9997879333
  • 44. Managing hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the lab • SSSSSSSSSSSSSSSSSecurity purpose • RRRRRRRRRRRRRRRRRun Demos • HHHHHHHHHHHHHHHHHandover to other teams there are a bunch of rasons to have a DMZ but if you have to expose the foreman you loose the advantage of the DMZ of cou
  • 45. Foreman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutions • DDDDDDDDDDDDDDDDDNS Proxy, DHCP proxy, TFTP… • KKKKKKKKKKKKKKKKKickstart proxying? • DDDDDDDDDDDDDDDDDigging into the documentation • FFFFFFFFFFFFFFFFFeature is there but not really visible • IIIIIIIIIIIIIIIIImprove docs: PR theforeman/theforeman.org#547 Foreman has the ability to serve the templates from proxies That way only proxies need access to the foreman feature was not in the proxies list so not sure if up to date or not now it is in the docs
  • 47. DocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentationDocumentation • TTTTTTTTTTTTTTTTThe Foreman documentation is huge • HHHHHHHHHHHHHHHHHosted on theforeman.org • IIIIIIIIIIIIIIIIImprove it so the next guy doesn't lose your time again We got a few trouble woth the docs Open-Source on Github so here is the story: docs are wrong ask for help Fix the doc if needed
  • 48. A NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC StoryA NoVNC Story • FFFFFFFFFFFFFFFFForeman embeds NoVNC • AAAAAAAAAAAAAAAAAllows you to see VNC console in browser • LLLLLLLLLLLLLLLLLots of problems with certificates So that feature is cool After an upgrade it stopped working Browser and FM config We followed then the docs Did not workkkk
  • 49. Details mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails mattersDetails matters • IIIIIIIIIIIIIIIIIn the 1.7 docs: websockets_encrypt: true • LLLLLLLLLLLLLLLLLet's change it to false • PPPPPPPPPPPPPPPPProblem: true/false vs on/off • EEEEEEEEEEEEEEEEExtra work: Updated the docs It does not happen so often but reviewing the doc is important Fixing is as important I hope we helped there too This is an exception
  • 50. Release NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease NotesRelease Notes • RRRRRRRRRRRRRRRRRelease notes are part of Documentation • WWWWWWWWWWWWWWWWWhen you change behaviour, think about others • LLLLLLLLLLLLLLLLLower update cost Keeping RN up to date with your breaking contrib is very important it lower the prices of upgrading because it removes uncertainty Couple your changes with a change in the R So YOU will remember; but other too MANY USECASES
  • 52. Scaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tipsScaling tips • EEEEEEEEEEEEEEEEEnable organizations/locations • UUUUUUUUUUUUUUUUUse one foreman for separated entities • UUUUUUUUUUUUUUUUUse one foreman for separated countries • UUUUUUUUUUUUUUUUUse foreman proxies where needed Org and Locations are great Isolation Maintanability Visibility ACL
  • 54. AssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociationAssociation • OOOOOOOOOOOOOOOOOS -> Location and Org • CCCCCCCCCCCCCCCCCompute Resource -> Location and Org • TTTTTTTTTTTTTTTTTemplates -> Location and Org • OOOOOOOOOOOOOOOOOS -> Installation Media • IIIIIIIIIIIIIIIIInstallation media -> Location and Org This is a subset When you START using org it is a pain When you move hosts between now it corre Makes you lose time Price for isolation
  • 55. Scaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute ResourcesScaling Compute Resources • CCCCCCCCCCCCCCCCCompute Resources = Virtualization or Cloud Service • EEEEEEEEEEEEEEEEEasy to create hosts in those CR • lllllllllllllllllibvirt, ec2, gce, vmware… We mainly use libvirt And also VMWARE but I will come to vmware later
  • 56. Scaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling LibvirtScaling Libvirt • WWWWWWWWWWWWWWWWWe have 8 libvirt servers • TTTTTTTTTTTTTTTTTo create a VM, you chose CPU, RAM, disks… • AAAAAAAAAAAAAAAAAny change requires deletion and creation You can also change in libvirt you can not change after the host creation But this is fragile We generally prefer to change in the forem deletion, not rebuild Hello API deleting a host in the foreman is also rude
  • 57. Creating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hostsCreating consistent hosts • HHHHHHHHHHHHHHHHHostgroups • EEEEEEEEEEEEEEEEEnvironments • CCCCCCCCCCCCCCCCCompute profiles? Foreman has a lot of mechanisms to pre-populate a lot of fields For the VM specific fields, the mechanism is Compute Profiles
  • 58. Libvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute ProfilesLibvirt Compute Profiles • CCCCCCCCCCCCCCCCCPU • MMMMMMMMMMMMMMMMMemory • NNNNNNNNNNNNNNNNNIC • SSSSSSSSSSSSSSSSStorage • MMMMMMMMMMMMMMMMMultiple profiles per hypervisor
  • 59. What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then? • LLLLLLLLLLLLLLLLLibvirt servers are not a group • TTTTTTTTTTTTTTTTThey are separated Compute Resources • AAAAAAAAAAAAAAAAA lot of work (UI and API) even of you want to sync then with API difficult from an API POV you will need to make a script this is not a 5 minutes work then you will force you to keep it uptodate WE HAVE 8 HOSTS
  • 60. Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific? • YYYYYYYYYYYYYYYYYes: other providers are `centralized' • eeeeeeeeeeeeeeeeec2, gce, openstack… We were thinking about ovirt as a possible workaround but we quickly abandoned the idea even if we are probably close to the border so we keep the profiles in sync this is not good and creates complexity and inconsistency
  • 61. VMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWareVMWare • SSSSSSSSSSSSSSSSSome part of the lab runs VMWare • GGGGGGGGGGGGGGGGGreat: Foreman supports VMWare • (((((((((((((((((on paper) We had the opportinity to integrate VMWa It was there before and there is support from the foreman of course we tried to integrate with mitigated success
  • 62. VMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integration • MMMMMMMMMMMMMMMMMost of the settings, Foreman will take user defaults • FFFFFFFFFFFFFFFFForeman is not for VMWare Power users • SSSSSSSSSSSSSSSSSounds like a lack of interest from community No interest in community we reached basic problems
  • 63. Logging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging matters • FFFFFFFFFFFFFFFFForeman requires on Fog • FFFFFFFFFFFFFFFFFog is a gem for the `clouds' • FFFFFFFFFFFFFFFFFog for vmware is not as advances as we'd like I do not BLAME fog This is just to show the lakc of interest and support One example: during first try, we got a stra ‘Wrong Credentials’ Solution: Wrong PERMISSIONS. We found that by using another client I would not recommend prople to use vmwa
  • 64. Empowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the Developers Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/julesdphotographie/8421289746
  • 65. Distributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the power • RRRRRRRRRRRRRRRRRebuilding a host is simple • OOOOOOOOOOOOOOOOOne clic operation • FFFFFFFFFFFFFFFFForeman as a VM shop we do not need to be in the middle devs should be autonomous devops mouvements
  • 66. The DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movement • DDDDDDDDDDDDDDDDDevOps is a movement born in 2009 • CCCCCCCCCCCCCCCCCollaboration between Developers and Operations • NNNNNNNNNNNNNNNNNothing new, just common sense • DDDDDDDDDDDDDDDDDevOpsDays, a serie of conferences all around the world devops movement buzz word actually something behind getting ops and dev to work together a community that meet in different places solving challenges still what is this
  • 67. #DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS#DevOps CAMS • CCCCCCCCCCCCCCCCCulture • AAAAAAAAAAAAAAAAAutomation • MMMMMMMMMMMMMMMMMeasurement • SSSSSSSSSSSSSSSSSharing John Willis and Damon Edwards DevOps is CAMS Definition from 2010 it is a cultural and profesionnal getting everyone to work together reduce time to business
  • 68. The C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMS • DDDDDDDDDDDDDDDDDevOps is a Cultural change • EEEEEEEEEEEEEEEEEveryone is in the team • OOOOOOOOOOOOOOOOOps and Devs work together • SSSSSSSSSSSSSSSSShare the responsibilities Sharing the responsibilities also means give more power and play the same game No being in the middle
  • 69. Foreman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the picture • FFFFFFFFFFFFFFFFForeman empowers the developers • WWWWWWWWWWWWWWWWWhile still providing enough security • OOOOOOOOOOOOOOOOOrganisations in Foreman The Organisations in foreman allow you to split the quickly rebuildable VM Advanced ACL from the core of your infra for security and simplicity
  • 70. Developer ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACLDeveloper ACL Licensed under a Creative Commons Attribution-ShareAlike 3.0 License Screenshot
  • 71. The Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer sideThe Developer side • OOOOOOOOOOOOOOOOOne-clic rebuild (at will) • AAAAAAAAAAAAAAAAAccess to build reports • IIIIIIIIIIIIIIIIIn-browser access to VNC • FFFFFFFFFFFFFFFFFresh vm in minutes For the developers they get almost the same advantages as with vagran except that we can provide more VMs/RAM could have on the laptop also we centralize and do not give direct ac to libvirt credential. Side note: foreman is hooked to LDAP
  • 72. The Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops sideThe Ops side • KKKKKKKKKKKKKKKKKeep everything under control • AAAAAAAAAAAAAAAAAudit logs, reports • IIIIIIIIIIIIIIIIInventory • BBBBBBBBBBBBBBBBBring regular updates to VM's/developers the ops can then maintain the kickstart/pu devs will get the changes immediatly or at audit logs to know who did what when
  • 73. UpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdatesUpdates Licensed under a Creative Commons Attribution 2.0 License https://www.flickr.com/photos/inyucho/7866698878
  • 74. Foreman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updates • AAAAAAAAAAAAAAAAAs any software: • TTTTTTTTTTTTTTTTTake a backup first • TTTTTTTTTTTTTTTTTry on your dev environment • FFFFFFFFFFFFFFFFFirst upgrade the main UI • TTTTTTTTTTTTTTTTThen update the proxies (#12506) Foreman is a regular application If upgrade hurts, do it more often e.g each minor release We DO NOT DO 0 releases
  • 75. Upgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problems • 11111111111111111.5->1.6: DHCP config file not readable by foreman proxy WWWWWWWWWWWWWWWWWe did a DHCP upgrade at the same time FFFFFFFFFFFFFFFFFile ownership was changed rrrrrrrrrrrrrrrrre-run the foreman installer fixed it
  • 76. Upgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problems • 11111111111111111.5->1.6: Puppet reports not coming into the foreman FFFFFFFFFFFFFFFFForeman report preprocessor has changed NNNNNNNNNNNNNNNNNeed to update foreman.rb + config nnnnnnnnnnnnnnnnnode.rb also needed an update
  • 77. Upgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problems • 11111111111111111.7->1.8: Big stack trace on opening the UI yyyyyyyyyyyyyyyyyum erase ruby193-rubygem-foreman_openstack_cluster ruby193-rubygem-foreman_openstack_cluster-doc A subset of troubles we had when upgradin We get less and less of these
  • 78. Upgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problems • 11111111111111111.8->1.9: Yaml transformations in Puppet parameters FFFFFFFFFFFFFFFFFixed in 1.10, backported to 1.9
  • 79. Community and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-Source Licensed under a Creative Commons Attribution-2.0 License https://www.flickr.com/photos/maicosnap/4393929855
  • 80. Foreman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-Source • GGGGGGGGGGGGGGGGGPLv3+ • WWWWWWWWWWWWWWWWWe've got around 20 patches integrated • TTTTTTTTTTTTTTTTThey rely mostly on open-source tools • CCCCCCCCCCCCCCCCCI, testing, bugtracker This is a HUMAN-SCALE OS Project THAT WORKS Listens to community Accept improvements, bugfixes, features Plus it is modular (plugins) you can write plugins for anything
  • 81. Story: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global Status • PPPPPPPPPPPPPPPPPut in build mode, go back home • RRRRRRRRRRRRRRRRRestart on the next day • LLLLLLLLLLLLLLLLLooping reboots Enabling build mode does not force restart so I did enable the build and try to rebuild the next day but the toek was expired so the host kept reinstalling then we figured out the problem
  • 82. Foreman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global status • FFFFFFFFFFFFFFFFForeman keeps several status for hosts • PPPPPPPPPPPPPPPPPlugins can register a status • BBBBBBBBBBBBBBBBBut reporting and provisioning also • LLLLLLLLLLLLLLLLLet's use that for expired tokens (bug#5883) Now the status of reporting is different tha it means that we could get a status if token
  • 83. SolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutionsSolutions • WWWWWWWWWWWWWWWWWrite a patch, keep it • WWWWWWWWWWWWWWWWWrite a plugin • WWWWWWWWWWWWWWWWWrite and upstream a patch Since it is core stuff we wrote a patch because this is not only for us when token expired you cant build anymore DOT. We wrote and upstreamed a patch
  • 84. Advantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreaming • FFFFFFFFFFFFFFFFFeature will be available in next releases • WWWWWWWWWWWWWWWWWe won't patch in the future • CCCCCCCCCCCCCCCCCalls in the patch will evolve as well • PPPPPPPPPPPPPPPPPeer review with the Foreman team • FFFFFFFFFFFFFFFFForced to write tests, doc, etc… The fact to upstream the patch reduce our work in the future and forces un to write tests and run tests on a larger infra than ours NOT THE END OF THE STORY
  • 85. 2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after • PPPPPPPPPPPPPPPPPatch is accepted for 2 weeks • AAAAAAAAAAAAAAAAA new bug is opened: performance problem (bug#14050) • SSSSSSSSSSSSSSSSSomeone else fixed the bug • WWWWWWWWWWWWWWWWWe applied the second patch A perf bug was opened still far before delivery we would not have known without upstream it was patched before we could have the tim thats the beauty of open source
  • 87. Where can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improve • PPPPPPPPPPPPPPPPPerformance • SSSSSSSSSSSSSSSSSome features are missing from API • DDDDDDDDDDDDDDDDDecoupling from Puppet • …………………………………………… Some area I would like to see changes some of them are already reworked
  • 88. Where did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improve • TTTTTTTTTTTTTTTTTests reliability • TTTTTTTTTTTTTTTTTesting Infra • CCCCCCCCCCCCCCCCCommunity • SSSSSSSSSSSSSSSSStability Foreman improved in may areas recently Community and infra changes are great CI results are meaningful
  • 89. The ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe Foreman • SSSSSSSSSSSSSSSSStability, Maturity, Flexibility • MMMMMMMMMMMMMMMMMade out of bricks: complex but not complicated • IIIIIIIIIIIIIIIIIt meets our needs • IIIIIIIIIIIIIIIIIt is very active • PPPPPPPPPPPPPPPPPeople are welcoming Its professional A lot of activity Public demo/reviews on youtube
  • 91. ContactContactContactContactContactContactContactContactContactContactContactContactContactContactContactContactContact Julien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien Pivotto julien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eujulien@inuits.eu @roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie inuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuitsinuits https://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.euhttps://inuits.eu info@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.euinfo@inuits.eu +32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636+32 473 441 636