HashiTalks2020: Making Automatically Compliant Design Documents With Infrastructure as Code

© 2020 NTT DATA Corporation 0
© 2020 NTT DATA Corporation
February 21, 2020
Yutaro Sasaki, NTT DATA Corporation
Making Automatically Compliant Design Documents
With Infrastructure as Code
HashiTalks 2020

Today's Goals
• Backgrounds
• SI business in Japan
• Importance of parameter sheets
(the most detailed design documents)
• 3 key factors
• Technical topics
• History of our IaC approaches
• Our new IaC approach
• 3 key points
Design Documents × IaC Software

Agenda of this talk
1. Section 1: NTT DATA & SI Business in Japan
1. Role of System Integrators in the IT Services Industry
2. Cloud Consulting ～ Lift & Shift to the Cloud
2. Section 2: Deep Dive Into Parameter Sheets, the Most Detailed Design Document
1. Design Documents in Deliverables
2. Appearance of Parameter Sheet
3. Appearance of Parameter Sheet: Example
4. Expectations for Parameter Sheets and Issues
5. 3 Key Factors of Parameter Sheets
6. Reality of the Parameter Sheets
7. Summery of This Section
3. Section 3: IaC Approaches With Parameter Sheets
1. The History of Our IaC Approaches
2. Our Initial Approach: Extraction Data From Unified Format
3. The Overview of Our New IaC Approach
4. Point 1: Separation of View and Data
5. Point 2: Integration With Version Control and Reuse as Best Practices
6. Point 3: Separation of logic and data
7. Best Case Scenario: Example of Parameter Change

Presenter
Yutaro Sasaki（佐々木優太朗）
• Affiliation
Technology and Innovation Headquarters, NTT DATA Corporation
• Specialty
Server virtualization, Infrastructure automation
(Puppet, Chef, Ansible, Serverspec, Terraform)
• Recent Activity
Published book "Puppet [Configure & Manage] Practical Guide"
Talked on Japan Java User Group (JJUG) CCC 2017 Fall
Talked on NTT DATA Technology Conference 2019

[Section 1]
NTT DATA & SI Business in Japan

Role of System Integrators in the IT Services Industry
System Integrators (SIers) are total coordinators that undertake design,
development, operations, etc., in information system development.
Orders Providing Optimal
Systems
Customers
Computers, etc.
Software
Programmers
Network, communication
lines, etc.
System Engineers
Hardware Vendors
Packaged
Software
Vendors
Software Houses Telecommunications
Companies
Internet
Network
(Control and management of overall system development)
System Integrators

On-premises
Current System・
Operational Survey1
NTT DATA's Policy
Any Cloud
●AWS ●Azure ●GCP
●Oracle ●ECL ●VMware
●OpenStack ･･･
Cloud Consulting ～ Lift & Shift to the Cloud
Analyze the current system, clarify Fit & Gap of cloud migration, provide optimal
solutions including IT costs according to the characteristics of various Cloud forms
Provide the appropriate Cloud
for customers.
Digital System Assessment
Existing IT utilization assessment Digital system design Roadmap formulation
Digital Road map DesignⅣ ⅥDigital System DesignⅤ
Fit & Gap analysis,
overview design2 Transition Policy・
Roadmap Formulation3
'Any Cloud' policy
perfectly fits

[Section 2] Deep Dive Into Parameter Sheets,
the Most Detailed Design Document

Design Documents in Deliverables
Of course, it is required that design documents
and the system are surely consistent.
Orders Providing Optimal
Systems
Customers
(Control and management of overall system development)
System Integrators
As part of deliverables, engineers in develop team need to arrange
• Architecture design documents
• Detailed design documents (including parameter sheets)

Appearance of Parameter Sheet
What does delivered
parameter sheet look like?

Appearance of Parameter Sheet: Example

Appearance of Parameter Sheet: Example
Parameter
names
Actual values in
environment
Default values Design policy
Parameter
description

Expectations for Parameter Sheets and Issues
• Expectations for parameter sheets:
• Making implicit design knowledge explicit and avoiding "secret
sauce"
• Reuse and distribute as best practices
• Inconsistency issues
• In reality, inconsistencies occur between documents and actual
machines mostly because of human errors.
One-stop solution:
Automation from design documents
with IaC software https://www.slideshare.net/nttdata-
tech/infrastructure-as-code-2019-nttdata-sasaki-takai
NTT DATA Technology Conference 2019

3 Key Factors of Parameter Sheets
Parameter sheets are meaningful
in typical SI projects
3 key factors:
1) The only documents showing the status of the actual machines
2) Explicit documents that describe most detailed design policy
3) Communication tool between develop team and operation team

3 Key Factors of Parameter Sheets: The 1st Factor
1) The only documents showing the status
of the actual machines
I think it's the most reliable way
to directly investigate the actual machines.

For security reasons,
the actual machines are not easily
accessible in most cases.
An application for authorization must be
submitted days in advance.
Only the planned and permitted operations
can be performed.
(Unplanned or ad-hoc operations
are strictly prohibited.)

Even if they are easily accessible,
it's hard to compare between many
machines.
SSH, SSH, SSH, ... or Parallel SSH ...
diff, diff, diff, ... or
how to compare more than 3 files ...

3 Key Factors of Parameter Sheets: The 2nd Factor
2) Explicit documents that describe most
detailed design policy
I comprehend the design policy. I couldn't
understand why I have to spend much time
on documentation.

We should assume the engineers leave the
project. Explicit knowledge is important in
operation phase.
3 Key Factors of Parameter Sheets: The 2nd Factor
A few years after the
launch of the service, a
parameter change was
suddenly required. ・・・
Can I change this parameter which
has been changed many times?
I have no idea how and why it
happened at all ...
(It is so called "secret sauce"...)

3 Key Factors of Parameter Sheets: The 3rd Factor
3) Communication tool between develop team
and operation team
Developer: "Here is
the list of monitoring
items."
Operator: "Very
specific and clear.
(There's no room for
interpretation!)"

Reality of the Parameter Sheets
Parameter sheets are
one of the most frequently
modified documents
in actual projects.

Reality of the Parameter Sheets
Because parameter sheets are the most
detailed and specific design documents,
they are often affected by operational changes.
Architecture
design
documents
Detailed design
documents
Parameter Sheets
[Examples of operational changes]
• Add new users
• Security updates

One More Thing ...
Documents in binary format are difficult
to integrate with version control and IaC
Traditionally, design documents are in MS office format
(Excel or Word) and therefore, we have needed to extract
values somehow for integration with Git and Terraform.
Parameter
sheet
XLSX

Summery of This Section
Parameter sheets are necessary, but have some difficulties
• What parameter sheets in SI projects
• Part of deliverables
• One of the most frequently modified documents
• 3 Key factors of parameter sheets
• The only documents showing the status of the actual machines
• Explicit documents that describe most detailed design policy
• Communication tool between develop team and operation team
• Difficulties in integration
• Version control and IaC software, due to binary format

[Section 3]
IaC Approaches With Parameter Sheets

The History of Our IaC Approaches
Traditional IaC approach (Automation codes are transcribed manually)
Ideal IaC approach (Automation codes (data) are generated automatically)
Automated data generation
Prehistory (Not automated at all)
+ Manual construction and manual testing×
Automated task
+
[Legend] Manual task
Manual data transcription×
Automated construction
and testing
Automated construction
and testing

• Difficulties remain in reality
• Users start customizing "unified format"
• Data extraction logic is relatively complex
• Parameter sheets cannot be completely under version control
Our Initial Approach: Extraction Data From Unified Format
Extraction data from Excel files have some difficulties
Repository
Our initial approach in the pastPredefined unified format (Excel)
data elements View Elements

The Overview of Our New IaC Approach
Automation
server
Code
Code
Code +
+
+
convert + execute
Generates parameter
sheet (Excel) as a view
AWS vSphere
Point 3: Separation of logic and data
Point 1: Separation of view and data
Point 2: Integration with version control
and reuse as best practices
Repository
Repository also contains
automation codes, such
as Terraform TF files
...
execute test
apply settings
apply settings
deploy

7. Appendix (TERASOLUNA IA Editor)
Separation is achieved by the Editor that can view
and edit design information (in plain text format)
Point 1: Separation of View and Data
The Editor
(Excel file with VBA Macros)
Export (Optional)
Save file
(Save button / Ctrl + S)
Open file
(Open button / double click file)
• View part
• The Editor offers view (rendering) feature
(Exporting static parameter sheet (xlsx, not xlsm))
• Data part
• Data files are always in plain text format
(Primary information and single source of truth)

7. Appendix (TERASOLUNA IA Editor)
Version control ecosystem encourages sharing
best practices across various projects
Point 2: Integration With Version Control and Reuse as Best Practices
Repository
Design information file opened by text editor (notepad++)
• File format (Extention of CSV)
• Semantic Design Format (SDF)
(Line (Row) oriented format)

Point 3: Separation of logic and data
Same as other IaC software, Terraform supports referencing
external variables, which means we can use shared logic code
node1 = {
template-name = "template_rhel7"
vm-name = "node1"
vm-folder = "/folder"
vm-disk-size = 20
vm-append_disk = [
{
label = "append_disk0"
size = 5
eagerly_scrub = false
thin_provisioned = true
unit_number = 1
},
{
label = "append_disk_1"
size = 3
・・・
resource "vsphere_virtual_machine" "node1" {
name = "${var.node1.vm-name}"
folder = "${var.node1.vm-folder}"
datastore_id = "${data.vsphere_datastore.datastore.id}"
resource_pool_id = "${data.vsphere_resource_pool.pool.id}"
disk {
label = "disk0"
size = "${var.node1.vm-disk-size}"
eagerly_scrub = "${data...disks.0.eagerly_scrub}"
thin_provisioned = "${data...disks.0.thin_provisioned}"
dynamic "disk" {
for_each = "${var.node1.vm-append_disk}"
content {
label = disk.value.label
size = disk.value.size
+

Best Case Scenario: Example of Parameter Change
With shared automation code (logic), developer only have to edit
"design information" in order to add/change/delete parameters
Automation server
Code +
convert
Repository
apply settingsdeploy
Developer: Oh, I forgot adding 'disk3' and need
to deal with it ASAP. All I have to do is to edit
"design information" file. It's easy.
Edit (manual)

References and Conclusion

References
• Talks on conferences
• NTT DATA Technology Conference 2019 (Sep. 2019)(Japanese)
https://www.slideshare.net/nttdata-tech/infrastructure-as-code-2019-nttdata-sasaki-takai
• Red Hat Forum Tokyo 2019 (Nov. 2019)(Japanese)
• Solution: Our new IaC approach
• Part of our methodology, TERASOLUNATM, and not OSS

Conclusion
We will continue to broaden the coverage of automation
with IaC technology. Thank you for your attention.
Automation
server
Code
Code
Code +
+
+
convert + execute
Generates parameter
sheet (Excel) as a view
Repository
execute test
apply settings
apply settings
deploy

© 2020 NTT DATA Corporation
The names of products and services referred to in this slide are trademarks or
registered trademarks of NTT DATA or other companies in Japan and other countries.

HashiTalks2020: Making Automatically Compliant Design Documents With Infrastructure as Code

More Related Content

What's hot (20)

Similar to HashiTalks2020: Making Automatically Compliant Design Documents With Infrastructure as Code (20)

More from NTT DATA Technology & Innovation (20)

Recently uploaded (20)

HashiTalks2020: Making Automatically Compliant Design Documents With Infrastructure as Code