Level Up Your Biml: Best Practices and Coding Techniques (TUGA IT 2016)

THANK YOU TO OUR TEAM
ANDRÉ BATISTA ANDRÉ MELANCIA ANDRÉ VALA ANTÓNIO LOURENÇO BRUNO LOPES CLÁUDIO SILVA
NIKO NEUGEBAUER
RUI REISRICARDO CABRAL
NUNO CANCELO PAULO MATOS PEDRO SIMÕES
SANDRA MORGADO SANDRO PEREIRARUI BASTOS
NUNO ÁRIAS SILVA

Level Up Your Biml:
Best Practices and Coding Techniques
Cathrine Wilhelmsen · TUGA IT · May 21st 2016

Session Description
Is your Biml solution starting to remind you of a bowl of tangled spaghetti code? Good! That
means you are solving real problems while saving a lot of time. The next step is to make sure
that your solution does not grow too complex and confusing – you do not want to waste all
that saved time on future maintenance!
Attend this session for an overview of Biml best practices and coding techniques. Step by step,
we will simplify and enhance our solution. Learn how to centralize and reuse code with
Include files and the CallBimlScript method. Make your code easier to read and write by
utilizing LINQ (Language-Integrated Queries). Share code between files by using Annotations
and ObjectTags. And finally, if standard Biml is not enough to solve your problems, you can
create your own C# helper classes and extension methods to implement custom logic.
Start improving your code today and level up your Biml in no time!

You…
…?
Know basic Biml and BimlScript
Completed BimlScript.com lessons
Have created a staging environment

Next 70 minutes…
…?
Code Management
Practical Biml Coding
C# Classes and Methods

Cathrine Wilhelmsen
@cathrinew
cathrinewilhelmsen.net
Data Warehouse Architect
Business Intelligence Developer

BIDS Helper
Free open-source add-in for Visual Studio
60+ features for SSIS, SSAS and SSRS
Includes basic Biml package generator
bidshelper.codeplex.com

…or you can use the new Biml tools

BimlExpress
Free add-in for Visual Studio
Code editor with syntax highlighting and Biml Intellisense
More frequent updates than BIDS Helper
varigence.com/bimlexpress

BimlOnline
Free browser-based Biml editor
Code editor with Biml and C# Intellisense
Reverse-engineer from SSIS to Biml
bimlonline.com

BimlStudio
Licensed full-featured development environment for Biml
Visual designer and metadata modeling
Full-stack automation and transformers
varigence.com/bimlstudio

Comparing Biml Tools
Free
Biml Intellisense
"Black Box"
Free
Full Intellisense
Logical View
Licensed
Full IDE
Preview BimlScript

Biml
Business Intelligence Markup Language
XML Language – just text
Easy to read and write – but can be verbose
You can use any tool to generate the Biml for you…
(Text editor macros, Excel, PowerShell, T-SQL ++)

BimlScript
…but use BimlScript :)
Extend Biml with integrated C# (or VB) code blocks
Allows you to generate, control and manipulate Biml
Import metadata and quickly script entire solutions

BimlScript Code Nuggets
<# … #> Control Nuggets (Control logic)
<#= … #> Text Nuggets (Returns string)
<#@ … #> Directives (Compiler instructions)
<#+ … #> Class Nuggets (Create C# classes)

Yes, but how does it actually work?
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<# foreach (var table in RootNode.Tables) { #>
<Package Name="Load_<#=table.Name#>"></Package>
<# } #>
</Packages>
</Biml>
<Packages>
<Package Name="Load_Customer"/>
<Package Name="Load_Product"/>
<Package Name="Load_Sales"/>
</Packages>
</Biml>

Biml and BimlScript
Flat XML
"Just text"
Generate, control and
manipulate Biml with C#

Don't Repeat Yourself
Move common code to separate files
Centralize and reuse in many projects
Update code once for all projects
1. Include files
2. CallBimlScript with Parameters
3. Tiered Biml files

Include Files
Include common code in multiple files and projects
Can include many file types: .biml .txt .sql .cs
Use the include directive
<#@ include file="CommonCode.biml" #>
The directive will be replaced by the included file
Works like an automated Copy & Paste

CallBimlScript with Parameters
Works like a parameterized include (or stored procedure)
File to be called (callee) specifies input parameters it accepts
<#@ property name="Parameter" type="String" #>
File that calls (caller) passes input parameters
<#=CallBimlScript("CommonCode.biml", Parameter)#>

CallBimlScript with Parameters

Tiered Biml Files
Split Biml code in multiple files and use the template directive:
<#@ template tier="1" #>
Compile Biml from lowest to highest tier to:
• Solve logical dependencies
• Simulate manual workflows
For each tier, objects are added to the RootNode
Higher tiers can use objects from lower tiers

What is this RootNode?
When working with flat Biml,
the <Biml> root element contains
collections of root objects:
<Biml>
<Connections>...</Connections>
<Databases>...</Databases>
<Schemas>...</Schemas>
<Tables>...</Tables>
<Projects>...</Projects>
<Packages>...</Packages>
</Biml>
When working with BimlScript,
the RootNode contains collections of
root objects:
=

How do I query the RootNode?
Query the RootNode to loop over collections:
Query the RootNode to get specific objects:
<#=RootNode.Tables["Product"].Schema#>

Inside the Black Box: Tiered Biml Files
<Connections>...</Connections>
<Packages>...</Packages>
<Package>...</Package>

Inside the Black Box: Tiered Biml Files

How do you use Tiered Biml files?
1. Create Biml files with specified tiers
2. Select all the tiered Biml files
3. Right-click and click Generate SSIS Packages
1
2
3

Debugging Biml
BimlExpress is a "black box":
• You can only see the generated SSIS packages
• It is not possible to see the compiled Biml first
Add a high-tier helper file to save compiled Biml to file
• Check Biml For Errors saves compiled Biml
without generating packages

SaveFlatBimlToFile.biml
Add the helper file to your project…
<# System.IO.File.WriteAllText(
@"C:BimlFlatBiml.xml",
RootNode.GetBiml()
); #>

…with a high tier so it is executed as the last step
RootNode.GetBiml()
); #>

It creates a file…
RootNode.GetBiml()
); #>

…at the specified path…
RootNode.GetBiml()
); #>

…with all the Biml for all the objects in RootNode :)
RootNode.GetBiml()
); #>

How do you use this helper file?
1. Create the helper file
2. Select all the Biml files and the helper file
3. Right-click and click Check Biml For Errors
1
2
3

Annotations and ObjectTags
Biml Annotations != SSIS Annotations
Annotations are string/string Key/Value pairs
ObjectTags are string/object Key/Value pairs
Use Annotations and ObjectTags to pass code
between Biml files

Annotations
Create annotations:
<OleDbConnection Name="Destination" ConnectionString="…">
<Annotations>
<Annotation Tag="Schema">AW2014</Annotation>
</Annotations>
</OleDbConnection>
Use annotations:
<# var destinationSchema =
RootNode.OleDbConnections["Destination"].GetTag("Schema"); #>

ObjectTags
Create ObjectTags:
<#
RootNode.OleDbConnections["Destination"].ObjectTag["TableFilter"] =
new List<string> {"Product","ProductSubcategory","ProductCategory"};
#>
Use ObjectTags:
<#
var TableFilter = (List<string>)
RootNode.OleDbConnections["Destination"].ObjectTag["TableFilter"];
#>

LINQ (Language-Integrated Query)
One language to query:
SQL Server Databases
XML Documents
Datasets
Collections
Two ways to write queries:
SQL-like Syntax
Extension Methods

LINQ Extension Methods
..and many, many more!
Sort
OrderBy, ThenBy
Filter
Where, OfType
Group
GroupBy
Aggregate
Count, Sum
Check Collections
All, Any, Contains
Get Elements
First, Last, ElementAt
Project Collections
Select, SelectMany

LINQ Extension Methods
var numConnections = RootNode.Connections.Count()
foreach (var table in RootNode.Tables.Where(…))
if (RootNode.Packages.Any(…))

LINQ and Lambda expressions
Use lambda expressions to filter or specify values:
.Where(table => table.Schema.Name == "Production")
.OrderBy(table => table.Name)

For each element in the collection…

…evaluate a criteria or get a value:

LINQ: Filter collections
Where()
Returns the filtered collection with all elements that meet the criteria
RootNode.Tables.Where(t => t.Schema.Name == "Production")
OfType()
Returns the filtered collection with all elements of the specified type
RootNode.Connections.OfType<AstExcelOleDbConnectionNode>()

LINQ: Sort collections
OrderBy()
Returns the collection sorted by key…
RootNode.Tables.OrderBy(t => t.Name)
ThenBy()
…then sorted by secondary key
RootNode.Tables.OrderBy(t => t.Schema.Name)
.ThenBy(t => t.Name)

OrderByDescending()
Returns the collection sorted by key…
RootNode.Tables.OrderByDescending(t => t.Name)
ThenByDescending()
…then sorted by secondary key
RootNode.Tables.OrderBy(t => t.Schema.Name)
.ThenByDescending(t => t.Name)

Reverse()
Returns the collection sorted in reverse order
RootNode.Tables.Reverse()

LINQ: Group collections
GroupBy()
Returns a collection of key-value pairs where each value is a new collection
RootNode.Tables.GroupBy(t => t.Schema.Name)

LINQ: Aggregate collections
Count()
Returns the number of elements in the collection
RootNode.Tables.Count()
RootNode.Tables.Count(t => t.Schema.Name == "Production")

Sum()
Returns the sum of the (numeric) values in the collection
RootNode.Tables.Sum(t => t.Columns.Count)
Average()
Returns the average value of the (numeric) values in the collection
RootNode.Tables.Average(t => t.Columns.Count)

Min()
Returns the minimum value of the (numeric) values in the collection
RootNode.Tables.Min(t => t.Columns.Count)
Max()
Returns the maximum value of the (numeric) values in the collection
RootNode.Tables.Max(t => t.Columns.Count)

LINQ: Check collections
All()
Returns true if all elements in the collection meet the criteria
RootNode.Databases.All(d => d.Name.StartsWith("A"))
Any()
Returns true if any element in the collection meets the criteria
RootNode.Databases.Any(d => d.Name.Contains("DW"))

LINQ: Check collections
Contains()
Returns true if collection contains element
RootNode.Databases.Contains(AdventureWorks2014)

LINQ: Get elements
First()
Returns the first element in the collection (that meets the criteria)
RootNode.Tables.First()
RootNode.Tables.First(t => t.Schema.Name == "Production")
FirstOrDefault()
Returns the first element in the collection or default value (that meets the criteria)
RootNode.Tables.FirstOrDefault()
RootNode.Tables.FirstOrDefault(t => t.Schema.Name == "Production")

LINQ: Get elements
Last()
Returns the last element in the collection (that meets the criteria)
RootNode.Tables.Last()
RootNode.Tables.Last(t => t.Schema.Name == "Production")
LastOrDefault()
Returns the last element in the collection or default value (that meets the criteria)
RootNode.Tables.LastOrDefault()
RootNode.Tables.LastOrDefault(t => t.Schema.Name == "Production")

LINQ: Get elements
ElementAt()
Returns the element in the collection at the specified index
RootNode.Tables.ElementAt(42)
ElementAtOrDefault()
Returns the element in the collection or default value at the specified index
RootNode.Tables.ElementAtOrDefault(42)

LINQ: Project collections
Select()
Creates a new collection from one collection
A list of table names:
RootNode.Tables.Select(t => t.Name)
A list of table and schema names:
RootNode.Tables.Select(t => new {t.Name, t.Schema.Name})

LINQ: Project collections
SelectMany()
Creates a new collection from many collections and merges the collections
A list of all columns from all tables:
RootNode.Tables.SelectMany(t => t.Columns)

How is LINQ used
in Biml projects?

C# Classes and Methods
BimlScript and LINQ not enough?
Need to reuse C# code?
Create your own classes and methods!

C# Classes and Methods: From this…
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
if (node.GetTag(tag) != "") {
return true;
} else {
return false;
}
}
}

C# Classes and Methods: …to this
if (node.GetTag(tag) != "") {
return true;
}
return false;
}
}

return (node.GetTag(tag) != "") ? true : false;
}
}

return (node.GetTag(tag) != "");
}
}

Where do you put your C# code?
Inline code nuggets
Included Biml files with code nuggets
Reference code files

C# Classes and Methods: Inline
<# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
<#+
}
}
#>

C# Classes and Methods: Included Files
<#@ include file="HelperClass.biml" #>
...
<# } #>
<# } #>
</Biml>
<#+
}
}
#>

C# Classes and Methods: Code Files
<#@ code file="HelperClass.cs" #>
...
<# } #>
<# } #>
</Biml>
}
}

Extension Methods
"Make it look like the method belongs to
an object instead of a helper class"

Extension Methods: From this…
...
<# } #>
<# } #>
</Biml>
}
}

Extension Methods: …to this
...
<# } #>
<# } #>
</Biml>
public static bool AnnotationTagExists(this AstNode node, string tag) {
}
}

Extension Methods: …to this
<# if (table.AnnotationTagExists("SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
}
}

Extension Methods: …to this :)
<# foreach (var table in RootNode.Tables.Where(t =>
t.AnnotationTagExists("SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
}
}

Get things done
Start small
Start simple
Start with ugly code
Keep going
Expand
Improve
Deliver often

Level Up Your Biml: Best Practices and Coding Techniques (TUGA IT 2016)

@cathrinew
cathrinewilhelmsen.net
linkedin.com/in/cathrinewilhelmsen
contact@cathrinewilhelmsen.net
slideshare.net/cathrinewilhelmsen
Biml resources and references:
cathrinewilhelmsen.net/biml

Level Up Your Biml: Best Practices and Coding Techniques (TUGA IT 2016)

More Related Content

What's hot (20)

Similar to Level Up Your Biml: Best Practices and Coding Techniques (TUGA IT 2016) (15)

More from Cathrine Wilhelmsen (20)

Recently uploaded (20)

Level Up Your Biml: Best Practices and Coding Techniques (TUGA IT 2016)