DerbyCon 7.0 Legacy: Regular Expressions (Regex) Overview
2 likes465 views
This document provides an overview of regular expressions (regex), detailing their functionality and applications in various fields, particularly in information technology and security. It discusses basic regex syntax, common use cases, and variations across different programming languages, alongside several practical examples for capturing specific data from text. Resources for further learning and tools for testing regex are also included.
![Regular Expressions (Regex) Overview
September 24, 2017
Matt Scheurer
@c3rkah
Slides:
https://www.slideshare.net/cerkah
((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9]).(25[0-5]|2[0-4]
[0-9]|1[0-9][0-9]|[1-9]?[0-9]).(25[0-5]|2[0-4][0-9]|1[0-9][0-
9]|[1-9]?[0-9]).(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9]))](https://image.slidesharecdn.com/derbycon09-2017regexpresentation-170924064121/85/DerbyCon-7-0-Legacy-Regular-Expressions-Regex-Overview-1-320.jpg)
![Regex Basics – Quantifiers
● “.” matches any single character
● “+” suffix matches one or more repetitions
● “*” suffix matches zero or more repetitions
● “?” suffix means the character is optional
● “|” is an ‘or’ separator between characters
● “^” is a ‘not’ specifier to exclude a character
– Enclosed in square brackets prefixing the pattern
– [^<pattern>]](https://image.slidesharecdn.com/derbycon09-2017regexpresentation-170924064121/85/DerbyCon-7-0-Legacy-Regular-Expressions-Regex-Overview-17-320.jpg)
![Regex Basics – Ranges
● In addition to quantifiers (wild cards), ranges may be
specified with pattern matching
– Characters are enclosed inside of square brackets
“[“ “]” and separated by a hyphen “-”
● Examples:
– [a-z], [A-Z], and [0-9]](https://image.slidesharecdn.com/derbycon09-2017regexpresentation-170924064121/85/DerbyCon-7-0-Legacy-Regular-Expressions-Regex-Overview-19-320.jpg)
![Example 5 – Expression
"d{4}/d{2}/d{2}_d{2}:d{2}","(w[.|-|w]+)
Capture Group
down.mykings.pw
ssl-6582datamanager.de
privatkunden.datapipe9271.com
alegroup.info
fourthgate.org
dieutribenhkhop.com
dieutribenhkhop.com
amazon-sicherheit.kunden-ueberpruefung.xyz
sarahdaniella.com](https://image.slidesharecdn.com/derbycon09-2017regexpresentation-170924064121/85/DerbyCon-7-0-Legacy-Regular-Expressions-Regex-Overview-40-320.jpg)
DerbyCon 7.0 Legacy: Regular Expressions (Regex) Overview
- 1. Regular Expressions (Regex) Overview September 24, 2017 Matt Scheurer @c3rkah Slides: https://www.slideshare.net/cerkah ((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9]).(25[0-5]|2[0-4] [0-9]|1[0-9][0-9]|[1-9]?[0-9]).(25[0-5]|2[0-4][0-9]|1[0-9][0- 9]|[1-9]?[0-9]).(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9]))
- 2. About Me Matt Scheurer Systems Security Engineer Working in the Financial Services Industry Meeting Organizer for the CiNPA Security SIG DerbyCon 5.0 “Unity” Speaker Certifications: CompTIA Security+, MCP, MCPS, MCTS, MCSA, and MCITP
- 3. What Regular Expressions are Not! ● The term “Regular Expressions” or often simply called “Regex” for short should not be confused with “Old Sayings” – Adages, Allegories, Aphorisms, Axioms, Clichés, Epigrams, Idioms, Hyperboles, Maxims, Platitudes, Proverbs, Truisms, etc.
- 4. When it comes to “Old Sayings”... You would be hard pressed to beat the recollection and retelling of old sayings than my own mother...
- 5. What is Regex? Regex is a common syntax used to match patterns when parsing text data or output. Regex capture groups are used to extract strings of specific data into reference points for retrieval or processing.
- 6. Why learn Regex? ● Regex is a great skill set to have in the back pocket of nearly any interdisciplinary role across the Information Technology landscape ● Uses include: – Application and Software Development – Database queries – Linux Administration and power user commands such as grep, awk, sed, find, etc. – Searching through any type of text data or system logs
- 7. Regex uses in InfoSec ● Content filtering ● Input validation ● NGFW / UTM Layer 7 definitions ● Parsing large volumes of data or system logs to pick out specific data points of interest ● SIEM systems – Building or refining entire searches, or performing advanced parsing to narrow down extraneous information – Finding specific log events or log event items and sub-data ● Understand the underpinnings of many security products and utilities
- 8. Regex Variations and Variances
- 9. Different flavors of Regex ● While all versions of Regex share common conventions there are proprietary differences across the various Regex engines ● Popular Regex Engines include: – Perl, PCRE, PHP, .NET, Java, JavaScript, XRegExp, VBScript, Python, Ruby, Delphi, R, Tcl, POSIX, and others
- 10. Regex Resources ● Online Learning Site - https://regexone.com/ ● Regex Test Site - http://regexr.com/ ● Tutorial Site - http://www.rexegg.com/ ● Countless Additional Resources - https://www.google.com/search?q=regex ● Further Reading - https://en.wikipedia.org/wiki/Regular_expression
- 11. Let’s Begin...
- 12. Regex Basics – Simple Matching ● Simply type in exactly what you are trying to match ● Text string pattern matching is case-sensitive! – NOTE: certain non-alpha-numeric characters may require an escape prefix to match ●
- 13. Regex Basics – Text Matching ● In addition to typing in an exact text string for an exact match “w” will match a single alphanumeric character – Matches any word character (alphanumeric & underscore) – Only matches low-ascii characters (no accented or non-roman characters)
- 14. Regex Basics – Number Matching ● In addition to typing in an exact numeric string for an exact match “d” will match a single digit. – Matches any digit character (0-9)
- 15. Regex Basics – Matching a Space ● In addition to typing in an exact string with a space included for an exact match “s” will match a space in text – Matches any whitespace character (spaces, tabs, line breaks)
- 16. Regex Basics – Matching Opposites ● We just looked at a few character classes – All character classes are case-sensitive – Specifying those character classes in upper-case changes the pattern match to match the opposite ● “W”, “D”, and “S” respectively translate to – Not a word character – Not a digit – Not whitespace
- 17. Regex Basics – Quantifiers ● “.” matches any single character ● “+” suffix matches one or more repetitions ● “*” suffix matches zero or more repetitions ● “?” suffix means the character is optional ● “|” is an ‘or’ separator between characters ● “^” is a ‘not’ specifier to exclude a character – Enclosed in square brackets prefixing the pattern – [^<pattern>]
- 18. Regex Basics – Escaped Characters ● What if I want to match escaped characters such as a “., +, *, ?, |, ^, etc.” in my pattern against the data? – Prefix reserved escape characters with a “” ● What if I want to match a “” in my pattern against the data? –
- 19. Regex Basics – Ranges ● In addition to quantifiers (wild cards), ranges may be specified with pattern matching – Characters are enclosed inside of square brackets “[“ “]” and separated by a hyphen “-” ● Examples: – [a-z], [A-Z], and [0-9]
- 20. Regex Basics – Repetitions ● In addition to a range quantifier, repetitions may be specified with pattern matching – The number of character occurrences are specified inside of curly brackets/braces “{“ “}”, or separated by a comma “,” for a range of occurrences ● A{4} matches exactly “AAAA” ● A{1,4} matches “A”, “AA”, “AAA”, or “AAAA” ● A{4,} matches four or more consecutive “A’s”
- 21. Regex Basics – Line Matching ● The beginning of a line and/or end of a line may be specified in Regex pattern matching – “^”, matches the beginning (starts with) of a line – “$”, matches the end of a line – “^<pattern>$”, matches when the line begins with and ends with the specified pattern
- 22. Regex Capture Groups ● The true power of Regex is fully realized with defined capture groups ● These essentially define array like variables to pattern matched data – This is how we return the precise data we want, while ignoring the content we do not care about ● Capture groups are defined by patterns enclosed inside of parenthesis “(“ “)”
- 23. Regex Sub-Capture Groups ● Regex sub-capture groups can be defined by using nested parenthesis “(“ “)” – Example: ● “(Pattern (match))” – First Capture Group = Pattern match – Second Capture Group = match
- 24. Regex Pattern Matching Problems?
- 25. Really Stuck? Just Remember...
- 26. Regex Example 1 ● Threat Feed: malware-domains – Latest Blackhole-DNS File list – "BOOT" format – http://malware-domains.com/files/BOOT.zip ● Objective: Capture a list of FQDN’s
- 27. Example 1 – Data Format
- 28. Example 1 – Expression PRIMARYs(S+) Capture Group amazon.co.uk.security-check.ga autosegurancabrasil.com christianmensfellowshipsoftball.org dadossolicitado-antendimento.sad879.mobi hitnrun.com.my houmani-lb.com maruthorvattomsrianjaneyatemple.org paypalsecure-2016.sucurecode524154241.arita.ac.tz tei.portal.crockerandwestridge.com tonyyeo.com update-apple.com.betawihosting.net
- 29. Regex Example 2 ● Threat Feed: malware-domains – Complete Zone File (bind) – Spyware Domains – http://malware-domains.com/files/spywaredomains.zones.zip ● Objective: Capture a list of FQDN’s
- 30. Example 2 – Data Format
- 31. Example 2 – Expression zones"(S+)" Capture Group amazon.co.uk.security-check.ga autosegurancabrasil.com christianmensfellowshipsoftball.org dadossolicitado-antendimento.sad879.mobi hitnrun.com.my houmani-lb.com maruthorvattomsrianjaneyatemple.org
- 32. Regex Example 3 ● Threat Feed: DNS BlackHole – IP Blacklist – http://malc0de.com/bl/IP_Blacklist.txt ● Objective: Capture a list of IP addresses
- 33. Example 3 – Data Format
- 34. Example 3 – Expression (d{1,3}.d{1,3}.d{1,3}.d{1,3}) Capture Group 185.165.29.49 185.91.116.237 76.74.167.171 193.227.248.241 149.210.167.172 216.114.192.21 89.255.9.102 86.109.162.144 85.25.203.171 209.90.88.139
- 35. Regex Example 4 ● Threat Feed: SpamCop – Spam in progress – Source of Mail – wget https://www.spamcop.net/w3m?action=inprogress ● Objective: Capture a list of IP addresses
- 36. Example 4 – Data Format
- 37. Example 4 – Expression >(d{1,3}.d{1,3}.d{1,3}.d{1,3})< Capture Group 182.139.29.84 201.37.197.39 182.151.104.105 119.5.175.57 119.5.175.57
- 38. Regex Example 5 ● Threat Feed: Malware Domain List – Complete database in CSV format – http://www.malwaredomainlist.com/mdlcsv.php ● export.csv ● Objective: Capture a list of FQDN’s
- 39. Example 5 – Data Format
- 40. Example 5 – Expression "d{4}/d{2}/d{2}_d{2}:d{2}","(w[.|-|w]+) Capture Group down.mykings.pw ssl-6582datamanager.de privatkunden.datapipe9271.com alegroup.info fourthgate.org dieutribenhkhop.com dieutribenhkhop.com amazon-sicherheit.kunden-ueberpruefung.xyz sarahdaniella.com
- 41. Keeping the Regex Saw Sharpened
- 43. Questions?
- 44. The End Big Thank You and shout out to my dear sweet mother! She’s a very special person in my life, and a fantastic Grandmother! ...Plus she endured the unenviable task of raising me as a child and teenager. :) Pictured above: My mom with my son Love you mom!