Looking for Patterns
1 like1,182 views
This document provides an overview of regular expressions (regex or regexps), including what they are, common uses, and examples in different programming languages. Regular expressions are strings used to search for patterns in text. They are more powerful than wildcards and are available in many languages and programs. The document explains basic regex syntax like characters, anchors, quantifiers, character classes and grouping and provides examples of regex patterns for validating postal codes and URLs.
![POSIX PROGRAMS SUPPORT RE…
Basic Regular Expressions (BRE)
Character classes [ ]
Named Character classes
[[:digit:]]
Asterisk *
Dot .
Carat ^
Dollar $
Backslashed Braces { }
Backslashed Parens ( )
Extended Regular Expressions (ERE)
Question mark ?
Plus sign +
Pipe symbol |
Braces { }
Parentheses ( )
All other BRE](https://image.slidesharecdn.com/lookingforpatternsocs-150222103239-conversion-gate02/85/Looking-for-Patterns-7-320.jpg)
![grep [options] 'pattern' [file…]
grep is command line tool for
printing lines that match a pattern
Useful for demonstrating how
regular expressions work
By default, grep interprets regular
expressions as BRE
Using egrep, or grep -E interprets
regular expressions as ERE
• --color=auto highlights the part of the
line that matched the pattern
• -i is used to make grep case-
insensitive
• -c is used to have grep report a count
of the lines that matched
• -v is used to print the lines that don't
match the pattern](https://image.slidesharecdn.com/lookingforpatternsocs-150222103239-conversion-gate02/85/Looking-for-Patterns-8-320.jpg)
![RE CHARACTER CLASSES
Character classes match a single
character in the list or range enclosed
by brackets [ ]
If the first character enclosed is the
carat ^, then the list or range is
negated
To match the right square bracket ] it
must be the first character enclosed.
To not match it, it must be the second
character after a carat
To match a hyphen, it can be the first
or last character enclosed. To not
match it, it must be the second
character after a carat](https://image.slidesharecdn.com/lookingforpatternsocs-150222103239-conversion-gate02/85/Looking-for-Patterns-11-320.jpg)
![RE NAMED CHARACTER CLASSES
Named character classes must
be enclosed in brackets like
[[:xdigit:]]
Many are available: [:alnum:],
[:alpha:], [:cntrl:], [:digit:],
[:graph:], [:lower:], [:print:],
[:punct:], [:space:], [:upper:],
and [:xdigit:]](https://image.slidesharecdn.com/lookingforpatternsocs-150222103239-conversion-gate02/85/Looking-for-Patterns-12-320.jpg)
Looking for Patterns
- 1. Looking for Patterns - Finding them with Regular Expressions Presented by Keith Wright One Course Source keith@OneCourseSource.com
- 2. From http://xkcd.com/1171/ If this is how you think of regular expression now… Regular expressions…
- 3. REGULAR EXPRESSIONS ARE… ➢Strings used to search for patterns in text ➢More powerful than wildcards ➢Available in many programming languages and programs ➢Also known as "regexp", "RegEx", and "RE"
- 4. RE DOS AND DON'TS… ✔ Input Validation ✔ Data Extraction ✔ Data Elimination ✔ Search/Replace Do this… Don't do this… ✗Parsing ✗Allow publicly available searches ✗Use where better tools exists ✗Where using a procedure would be better
- 5. RE ARE AVAILABLE IN…AND MORE! .NET C# Delphi Java JavaScript Perl PCRE PHP Python Ruby Tcl PowerShell
- 6. POSIX PROGRAMS USING RE awk pattern scanning and processing language find utility to search for files grep utility to print lines matching a pattern sed stream editor for filtering and transforming text
- 7. POSIX PROGRAMS SUPPORT RE… Basic Regular Expressions (BRE) Character classes [ ] Named Character classes [[:digit:]] Asterisk * Dot . Carat ^ Dollar $ Backslashed Braces { } Backslashed Parens ( ) Extended Regular Expressions (ERE) Question mark ? Plus sign + Pipe symbol | Braces { } Parentheses ( ) All other BRE
- 8. grep [options] 'pattern' [file…] grep is command line tool for printing lines that match a pattern Useful for demonstrating how regular expressions work By default, grep interprets regular expressions as BRE Using egrep, or grep -E interprets regular expressions as ERE • --color=auto highlights the part of the line that matched the pattern • -i is used to make grep case- insensitive • -c is used to have grep report a count of the lines that matched • -v is used to print the lines that don't match the pattern
- 9. BASIC RE LITERALS Alphanumeric characters and non-regular expression characters match themselves Regular expression characters will match themselves if preceded by the backslash character
- 10. RE DOT (PERIOD) The dot . will match any single character To match the dot itself, it must be preceded by a backslash The RE .* is used to match an entire string
- 11. RE CHARACTER CLASSES Character classes match a single character in the list or range enclosed by brackets [ ] If the first character enclosed is the carat ^, then the list or range is negated To match the right square bracket ] it must be the first character enclosed. To not match it, it must be the second character after a carat To match a hyphen, it can be the first or last character enclosed. To not match it, it must be the second character after a carat
- 12. RE NAMED CHARACTER CLASSES Named character classes must be enclosed in brackets like [[:xdigit:]] Many are available: [:alnum:], [:alpha:], [:cntrl:], [:digit:], [:graph:], [:lower:], [:print:], [:punct:], [:space:], [:upper:], and [:xdigit:]
- 13. RE CARAT ANCHOR The character after the carat character ^ must appear at the beginning of the text If used as the first character in square brackets, it negates the list or range of characters If preceded by the backslash, the carat character loses it's special meaning
- 14. RE DOLLAR SIGN ANCHOR The character before the dollar sign character $ must appear at the end of the text If not at the end of the regular expression, then the dollar sign loses it's special meaning When combined with the carat character ^, the dollar sign character $ must match the entire text
- 15. RE REPETITION Basic Regular Expressions * preceding item repeated zero or more times or {0,} + preceding item repeated one or more times or {1,} ? preceding item is optional or {0,1} {n} preceding item repeated exactly n times {n,} preceding item repeated n or more times {,m} preceding item matched at most m times {n,m} preceding item matched at least n times, but not more than m times Extended Regular Expressions * preceding item repeated zero or more times or {0,} + preceding item repeated one or more times or {1,} ? preceding item is optional or {0,1} {n} preceding item repeated exactly n times {n,} preceding item repeated n or more times {,m} preceding item matched at most m times {n,m} preceding item matched at least n times, but not more than m times
- 16. RE ASTERISK The asterisk * will match zero or more of the item that precedes it The asterisk is equivalent to the BRE {0,} and the ERE {0,} expressions for zero or more A single item followed by an asterisk will always match To match an asterisk, it can be preceded by a backslash
- 17. RE PLUS SIGN In BRE, the backslashed plus sign + will match one or more of the item that precedes it In ERE, the plus sign + will match one or more of the item that precedes it The plus sign is equivalent to the BRE {1,} and the ERE {1,} expressions for one or more In BRE, the plus sign matches itself. In ERE to match a plus sign, it can be preceded by a backslash
- 18. RE QUESTION MARK In BRE, the backslashed question mark ? optionally matches the item that precedes it In ERE, the question mark will optionally match the item that precedes it The question mark equivalent to the BRE {0,1} and the ERE {0,1} expressions for zero to one In BRE, the question mark matches itself. In ERE to match a question mark, it can be preceded by a backslash
- 19. RE GROUPING In BRE, the backslashed parentheses ( and ) are used to create groups of characters that may repeat as specified by repetition expressions In ERE, the parentheses ( and ) are used to create groups of characters that may repeat as specified by repetition expressions In BRE, the parentheses will match themselves, and in ERE they can be matched if backslashed
- 20. RE ALTERNATION In ERE, the pipe symbol | can be used to perform alternation Alternation allows for two or more alternatives to match as separated by the pipe symbol | In BRE, the pipe symbol | will match itself, and in ERE it will match if backslashed
- 21. PERL US POSTAL CODE EXAMPLE ^d{5}((-|s)?d{4})?$ ^ - Starts with d{5} - exactly five digits ()? - optional group (two) -|s - hyphen or whitespace d{4} - exactly four digits $ - Ends with To use the perl debugger type: perl -d -e1
- 22. PERL CHARACTER SEQUENCES w Alphanumeric and _ (word characters) W Not word characters d Digit characters D Not digit characters s Whitespace characters S Not whitespace characters b Word boundaries • grep supports the perl character sequences in ERE except d and D
- 23. PYTHON PROTOCOL EXAMPLE (mailto:|(news|(ht|f)tp(s?))://){1} (){1} - group repeats only once mailto: - mailto followed by a colon | - separates alternatives news|(ht|f)tp - news, http or ftp (ht|f)tp(s?) - optional s added :// - added to news, http, https, ftp, or ftps • To start the python shell type: python
- 24. USE THE LIBRARY RegExLib.com The Regular Expression Library Comes with a cheat sheet A Regular Expression tester Search thousands of rated expressions You don't have to reinvent the wheel!
- 26. About One Course Source ➢Online public classes (Linux, Programming & Security) ➢Custom corporate classes ➢Develop custom training programs www.OneCourseSource.com
Editor's Notes
- #9: In ed or vi, g/re/p was to do a global search for the regular expression and print
- #14: Backslash example: echo 'xyz^abzzz' | grep '\^ab'
- #22: # Source: http://neilk.net/blog/2000/06/01/abigails-regex-to-test-for-prime-numbers/ # Source: Abigail -- perl -wle 'print "Prime" if (1 x shift) !~ /^1?$|^(11+?)\1+$/' sub is_prime { if ((1 x shift) !~ /^1?$|^(11+?)\1+$/) { return 1; } else { return 0; } } <number>
- #26: sub is_what { if ((1 x shift) !~ /^1?$|^(11+?)\1+$/) { return 1; } else { return 0; } }