Tame cloud complexity with F# powered DSLs (build stuff)
0 likes652 views
The document covers an in-depth case study of using AWS services like DynamoDB, Simple Workflow, and CloudWatch with F# domain-specific languages (DSLs). It discusses query capabilities, error handling, and activity worker implementation while emphasizing the importance of automating decision workflows. Additionally, it highlights the benefits of abstraction and code simplicity in functional programming.
![SELECT * FROM GameScore
type Attributes =
| Asterisk
| Attributes of string[]](https://image.slidesharecdn.com/tamecomplexapibuildstuff-170430105047/85/Tame-cloud-complexity-with-F-powered-DSLs-build-stuff-44-320.jpg)
![let (|Float|) input =
match Double.TryParse input with
| true, n -> n
| _ ->
failwithf “not a float [%s]” input](https://image.slidesharecdn.com/tamecomplexapibuildstuff-170430105047/85/Tame-cloud-complexity-with-F-powered-DSLs-build-stuff-163-320.jpg)
![let (|Float|) input =
match Double.TryParse input with
| true, n -> n
| _ ->
failwithf “not a float [%s]” input](https://image.slidesharecdn.com/tamecomplexapibuildstuff-170430105047/85/Tame-cloud-complexity-with-F-powered-DSLs-build-stuff-164-320.jpg)
![let (|Float|) input =
match Double.TryParse input with
| true, n -> n
| _ ->
failwithf “not a float [%s]” input
Float : string -> float](https://image.slidesharecdn.com/tamecomplexapibuildstuff-170430105047/85/Tame-cloud-complexity-with-F-powered-DSLs-build-stuff-165-320.jpg)
![[
“namespaceIs”; “‘JustEat’”; “and”;
“nameLike”; “‘cpu’”; “and”;
…
]](https://image.slidesharecdn.com/tamecomplexapibuildstuff-170430105047/85/Tame-cloud-complexity-with-F-powered-DSLs-build-stuff-189-320.jpg)
![let (|EmptyString|_|) input =
if String.IsNullOrWhiteSpace input
then Some ()
else None
let (|StartsWith|_|) char (input : string) =
if input.[0] = char then Some () else None
let (|EndsWith|_|) char (input : string) =
if input.[input.Length-1] = char
then Some ()
else None](https://image.slidesharecdn.com/tamecomplexapibuildstuff-170430105047/85/Tame-cloud-complexity-with-F-powered-DSLs-build-stuff-194-320.jpg)
Tame cloud complexity with F# powered DSLs (build stuff)
- 1. tame |> (cloud <| complexity) |> with |> (fsharp >> powered >> DSLs)
- 2. hi,I’mYanCui
- 6. AWS user since 2009
- 8. image by nerovivo license : https://creativecommons.org/licenses/by-sa/2.0/
- 21. can be changed on-the-fly
- 23. infinitely scalable (but you still have to pay for it)
- 26. Query: given a hash key filter on range key, or local secondary index
- 28. Hash Key Range Key Local Secondary Index Global Secondary Index
- 29. Scan: FULL TABLE search (performance + cost concern)
- 31. Hash Key Range Key Local Secondary Index Global Secondary Index
- 32. Hash Key Range Key Local Secondary Index Who are the TOP 3 players in “Starship X” with a score of at least 1000? Global Secondary Index
- 34. select GameTitle, UserId, TopScore from GameScores where GameTitle = “Starship X” and TopScore >= 1000 order desc limit 3 with (NoConsistentRead, Index(GameTitleIndex, true))
- 38. GOAL SELECT UserId, TopScore FROM GameScore WHERE GameTitle CONTAINS “Zelda” ORDER DESC LIMIT 3 WITH (NoConsistentRead)
- 40. F# & FParsec* *www.quanttec.com/fparsec External DSL via
- 42. SELECT * FROM GameScore Abstract Syntax Tree (AST) FParsec
- 43. SELECT * FROM GameScore keyword keyword * | attribute, attribute, … table name
- 44. SELECT * FROM GameScore type Attributes = | Asterisk | Attributes of string[]
- 45. SELECT * FROM GameScore type Query = { Attributes : Attributes Table : string }
- 46. SELECT * FROM GameScore Parser for “SELECT” keyword pSelect
- 47. SELECT * FROM GameScore pSelect let pSelect = skipStringCI "select"
- 48. SELECT * FROM GameScore pSelect let pSelect = skipStringCI "select" matches the string “select” (Case Insensitive) and ignores it
- 49. SELECT * FROM GameScore pFrom let pFrom = skipStringCI "from"
- 50. SELECT * FROM GameScore Parser for a string that represents the table name pTableName
- 51. SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName
- 52. SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName
- 53. SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName
- 54. SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName
- 55. SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName parses a sequence of one or more chars that satisfies the predicate function
- 56. SELECT * FROM GameScore pAsterisk * pAttributeName UserId, GameTitle, TopScore, …
- 57. SELECT * FROM GameScore pAsterisk * pAttributeName UserId, GameTitle, TopScore, … let pAsterisk = stringCIReturn "*" Asterisk
- 58. SELECT * FROM GameScore pAsterisk * pAttributeName UserId, GameTitle, TopScore, … let pAsterisk = stringCIReturn "*" Asterisk matches the specified string and return the given value
- 59. SELECT * FROM GameScore pAsterisk * pAttributeName UserId, GameTitle, TopScore, … let isAttributeName = isLetter <||> isDigit let pAttributeName = many1Satisfy isAttributeName
- 60. SELECT * FROM GameScore UserId, GameTitle, TopScore, … pAttributeName pCommapAsterisk * let pComma = skipStringCI ","
- 61. SELECT * FROM GameScore UserId, GameTitle, TopScore, … pAttributeName pCommapAsterisk * let pAttributeNames = sepBy1 pAttributeName pComma
- 62. SELECT * FROM GameScore UserId, GameTitle, TopScore, … pAttributeName pCommapAsterisk * let pAttributeNames = sepBy1 pAttributeName pComma parses one or more occurrences of pAttributeName separated by pComma
- 63. SELECT * FROM GameScore UserId, GameTitle, TopScore, … pAttributeName pComma sepBy1 pAttributeNames pAsterisk *
- 64. SELECT * FROM GameScore pAsterisk * pAttributeNames UserId, GameTitle, TopScore, …
- 65. SELECT * FROM GameScore pAsterisk * pAttributeNames UserId, GameTitle, TopScore, … let pAttribute = pAsterisk <|> pAttributeNames
- 66. SELECT * FROM GameScore pAsterisk * pAttributeNames UserId, GameTitle, TopScore, … let pAttribute = pAsterisk <|> pAttributeNames
- 67. SELECT * FROM GameScore pAsterisk * pAttributeNames UserId, GameTitle, TopScore, … choice pAttribute
- 68. SELECT * FROM GameScore pAttribute
- 69. SELECT * FROM GameScore pAttribute pTableNamepFrompSelect
- 70. SELECT * FROM GameScore pAttribute pTableNamepFrompSelect let pQuery = tuple4 pSelect pAttribute pFrom pTableName |>> (fun (_, attributes, _, table) -> { Attributes = attributes Table = table })
- 71. SELECT * FROM GameScore pAttribute pTableNamepFrompSelect let pQuery = tuple4 pSelect pAttribute pFrom pTableName |>> (fun (_, attributes, _, table) -> { Attributes = attributes Table = table })
- 72. SELECT * FROM GameScore pAttribute pTableNamepFrompSelect let pQuery = tuple4 pSelect pAttribute pFrom pTableName |>> (fun (_, attributes, _, table) -> { Attributes = attributes Table = table })
- 73. SELECT * FROM GameScore pAttribute pTableNamepFrompSelect let pQuery = tuple4 pSelect pAttribute pFrom pTableName |>> (fun (_, attributes, _, table) -> { Attributes = attributes Table = table }) Query
- 74. SELECT * FROM GameScore pAttribute pTableNamepFrompSelect tuple4 pQuery
- 76. < 50 lines of code
- 78. Recap
- 80. select GameTitle, UserId, TopScore from GameScores where GameTitle = “Starship X” and TopScore >= 1000 order desc limit 3 with (NoConsistentRead, Index(GameTitleIndex, true))
- 83. Decision Worker
- 91. Workers can run from anywhere
- 93. input = “Yan” result = “Hello Yan!” Start Finish Activity
- 95. image by Ryan Hageman license : https://creativecommons.org/licenses/by-sa/2.0/
- 97. API
- 101. – Kris Jordan “Good simplicity is less with leverage, not less with less. Good simplicity is complexity disguised, not complexity denied.”
- 105. the workflow is implied by decision worker logic…
- 106. instead..
- 107. the workflow should drive decision worker logic
- 108. the workflow should driveautomate decision worker logic
- 111. GOAL Code that matches the way you think
- 120. input result
- 122. Recap
- 129. wanna find correlations? you can DIY it! ;-)
- 130. “what latencies spiked at the same time as payment service?”
- 132. Find metrics whose 5 min average exceeded 1 second during last 12 hours
- 133. cloudWatch.Select( unitIs “milliseconds” + average (>) 1000.0 @ last 12 hours |> intervalOf 5 minutes)
- 134. cloudWatch.Select(“ unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes”)
- 135. “did any cache nodes’ CPU spike yesterday?”
- 136. cloudWatch.Select( namespaceLike “elasticache” + nameLike “cpu” + max (>) 80.0 @ last 24 hours |> intervalOf 15 minutes)
- 137. cloudWatch.Select( namespaceLike “elasticache” + nameLike “cpu” + max (>) 80.0 @ last 24 hours |> intervalOf 15 minutes) Regex
- 141. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
- 142. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes Filters
- 143. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes TimeFrame
- 144. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes Period
- 145. type Query = { Filter : Filter TimeFrame : TimeFrame Period : Period option }
- 147. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
- 148. type MetricTerm = Namespace | Name type Filter = | MetricFilter of MetricTerm * (string -> bool)
- 149. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
- 150. type MetricTerm = Namespace | Name type Unit = | Unit type Filter = | MetricFilter of MetricTerm * (string -> bool) | UnitFilter of Unit * (string -> bool)
- 151. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
- 152. type MetricTerm = Namespace | Name type Unit = | Unit type StatsTerm = | Average | Min | Max | Sum | SampleCount type Filter = | MetricFilter of MetricTerm * (string -> bool) | UnitFilter of Unit * (string -> bool) | StatsFilter of StatsTerm * (float -> bool)
- 153. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
- 154. type MetricTerm = Namespace | Name type Unit = | Unit type StatsTerm = | Average | Min | Max | Sum | SampleCount type Filter = | MetricFilter of MetricTerm * (string -> bool) | UnitFilter of Unit * (string -> bool) | StatsFilter of StatsTerm * (float -> bool) | CompositeFilter of Filter * Filter
- 155. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
- 156. type TimeFrame = | Last of TimeSpan | Since of DateTime | Between of DateTime * DateTime
- 157. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
- 158. type Period = | Period of TimeSpan
- 159. type Query = { Filter : Filter TimeFrame : TimeFrame Period : Period option }
- 160. Active Patterns a primer on
- 161. allow patterns to be abstracted away into named functions
- 162. Single-Case Patterns
- 163. let (|Float|) input = match Double.TryParse input with | true, n -> n | _ -> failwithf “not a float [%s]” input
- 164. let (|Float|) input = match Double.TryParse input with | true, n -> n | _ -> failwithf “not a float [%s]” input
- 165. let (|Float|) input = match Double.TryParse input with | true, n -> n | _ -> failwithf “not a float [%s]” input Float : string -> float
- 166. match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x
- 167. let (|Float|) input = match Double.TryParse input with | true, n -> n match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x
- 168. let (|Float|) input = match Double.TryParse input with | true, n -> n match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x
- 169. match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x let (|Float|) input = match Double.TryParse input with | true, n -> n
- 170. match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x let (|Float|) input = match Double.TryParse input with | true, n -> n
- 171. match “42” with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x
- 172. match “boo” with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x Error!!!
- 173. Partial Patterns
- 174. let (|Float|_|) input = match Double.TryParse input with | true, n -> Some n | _ -> None
- 175. let (|Float|_|) input = match Double.TryParse input with | true, n -> Some n | _ -> None Float : string -> float option
- 176. match “boo” with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x | _ -> “not a float”
- 177. match “boo” with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x | _ -> “not a float”
- 178. Multi-Case Patterns
- 179. let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
- 180. let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
- 181. let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
- 182. let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
- 183. let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
- 184. match someString with | Prime n -> … | NotPrime n -> … | NaN -> …
- 185. match someString with | Prime n & Float 11.0 -> … | Prime n -> … | Float 42.0 | Float 8.0 -> … | NotPrime n -> … | NaN -> …
- 186. match someString with | Prime n & Float 11.0 -> … | Prime n -> … | Float 42.0 | Float 8.0 -> … | NotPrime n -> … | NaN -> …
- 187. match someString with | Prime (IsPalindrome n) -> … | Prime (IsEven n) -> … | _ -> …
- 188. Tokenise (string -> string list)
- 189. [ “namespaceIs”; “‘JustEat’”; “and”; “nameLike”; “‘cpu’”; “and”; … ]
- 190. “namespaceIs” “and” … F# List = LinkedList “‘JustEat’”
- 191. “namespaceIs” “and” … F# List = LinkedList “namespaceIs” “‘JustEat’” “and”:::: :: tail cons (::) operator to prepend “‘JustEat’”
- 192. “namespaceIs” “and” … F# List = LinkedList “namespaceIs” “‘JustEat’” “and”:::: :: tail match aList with | “‘JustEat’” works as a pattern too!
- 193. let (|StringCI|_|) (str : string) (input : string) = if str.ToLower() = input.ToLower() then Some () else None let (|Float|_|) input = match System.Double.TryParse input with | true, n -> Some n | _ -> None
- 194. let (|EmptyString|_|) input = if String.IsNullOrWhiteSpace input then Some () else None let (|StartsWith|_|) char (input : string) = if input.[0] = char then Some () else None let (|EndsWith|_|) char (input : string) = if input.[input.Length-1] = char then Some () else None
- 195. let (|QuotedString|_|) (input : string) = match input with | EmptyString -> None | str when str.Length < 2 -> None | StartsWith ''' & EndsWith ''' -> Some <| input.Substring(1, input.Length - 2) | _ -> None
- 196. let (|QuotedString|_|) (input : string) = match input with | EmptyString -> None | str when str.Length < 2 -> None | StartsWith ''' & EndsWith ''' -> Some <| input.Substring(1, input.Length - 2) | _ -> None
- 197. let (|NamespaceIs|_|) = function | StringCI “NamespaceIs" :: QuotedString ns :: tl -> Some (eqFilter MetricFilter Namespace ns, tl) | _ -> None
- 198. let (|NamespaceIs|_|) = function | StringCI “NamespaceIs" :: QuotedString ns :: tl -> Some (eqFilter MetricFilter Namespace ns, tl) | _ -> None namespaceIs ‘JustEat’ and nameLike ‘cpu’ and …
- 199. let (|NamespaceIs|_|) = function | StringCI “NamespaceIs" :: QuotedString ns :: tl -> Some (eqFilter MetricFilter Namespace ns, tl) | _ -> None “namespaceIs” “‘JustEat’” :::: …
- 200. let (|NamespaceIs|_|) = function | StringCI “NamespaceIs" :: QuotedString ns :: tl -> Some (eqFilter MetricFilter Namespace ns, tl) | _ -> None type MetricTerm = Namespace | Name type Filter = | MetricFilter of MetricTerm * (string -> bool)
- 201. let (|NamespaceIs|_|) = function | StringCI “NamespaceIs" :: QuotedString ns :: tl -> Some (eqFilter MetricFilter Namespace ns, tl) | _ -> None “and” :: “nameLike” :: “‘cpu’” :: “and” :: …
- 202. let rec loop acc = function | NamespaceIs (filter, tl) | NamespaceLike (filter, tl) | NameIs (filter, tl) | NameLike (filter, tl) | UnitIs (filter, tl) | Average (filter, tl) | Sum (filter, tl) | Min (filter, tl) | Max (filter, tl) | SampleCount (filter, tl) -> match tl with | And tl -> loop (filter::acc) tl | _ -> flatten (filter::acc), tl | _ -> failwith “No filters?!?!?”
- 203. let rec loop acc = function | NamespaceIs (filter, tl) | NamespaceLike (filter, tl) | NameIs (filter, tl) | NameLike (filter, tl) | UnitIs (filter, tl) | Average (filter, tl) | Sum (filter, tl) | Min (filter, tl) | Max (filter, tl) | SampleCount (filter, tl) -> match tl with | And tl -> loop (filter::acc) tl | _ -> flatten (filter::acc), tl | _ -> failwith “No filters?!?!?”
- 204. let rec loop acc = function | NamespaceIs (filter, tl) | NamespaceLike (filter, tl) | NameIs (filter, tl) | NameLike (filter, tl) | UnitIs (filter, tl) | Average (filter, tl) | Sum (filter, tl) | Min (filter, tl) | Max (filter, tl) | SampleCount (filter, tl) -> match tl with | And tl -> loop (filter::acc) tl | _ -> flatten (filter::acc), tl | _ -> failwith “No filters?!?!?”
- 205. let rec loop acc = function | NamespaceIs (filter, tl) | NamespaceLike (filter, tl) | NameIs (filter, tl) | NameLike (filter, tl) | UnitIs (filter, tl) | Average (filter, tl) | Sum (filter, tl) | Min (filter, tl) | Max (filter, tl) | SampleCount (filter, tl) -> match tl with | And tl -> loop (filter::acc) tl | _ -> flatten (filter::acc), tl | _ -> failwith “No filters?!?!?”
- 206. let rec loop acc = function | NamespaceIs (filter, tl) | NamespaceLike (filter, tl) | NameIs (filter, tl) | NameLike (filter, tl) | UnitIs (filter, tl) | Average (filter, tl) | Sum (filter, tl) | Min (filter, tl) | Max (filter, tl) | SampleCount (filter, tl) -> match tl with | And tl -> loop (filter::acc) tl | _ -> flatten (filter::acc), tl | _ -> failwith “No filters?!?!?”
- 207. let parse (input : string) = input |> tokenize |> parseFilter |> parseTimeFrame |> parsePeriod
- 208. Amazing F# =
- 209. usable from anywhere you can run F# code e.g. F# REPL, executable, .. Internal DSL
- 210. useful for building tools e.g. CLI, … External DSL
- 214. Recap