Regexp secrets
2 likes1,506 views
The document discusses the complexities of regular expressions (regex), emphasizing the potential pitfalls of overreliance on them in problem-solving. It outlines formal language theory, including the characteristics of regular languages and how regex can represent these languages, while cautioning about the use of regex for non-regular languages. Additionally, it provides examples and tips for effectively utilizing regex in programming, particularly focusing on performance and security considerations.
![Use /x
UP_TO_256 = /b(?:25[0-5] # 250-255
|2[0-4][0-9] # 200-249
|1[0-9][0-9] # 100-199
|[1-9][0-9] # 2-digit numbers
|[0-9]) # single-digit numbers
b/x
IPV4_ADDRESS = /#{UP_TO_256}(?:.#{UP_TO_256}){3}/](https://image.slidesharecdn.com/regexpsecrets-130228042149-phpapp02/85/Regexp-secrets-43-320.jpg)
![Security Implications
class File < ActiveRecord::Base
validates :name, :format => /^[w.-+]+$/
end
http://guides.rubyonrails.org/security.html#regular-expressions](https://image.slidesharecdn.com/regexpsecrets-130228042149-phpapp02/85/Regexp-secrets-49-320.jpg)
![file.txtn<script>alert(‘hello’)</script>
/^[w.-+]+$/](https://image.slidesharecdn.com/regexpsecrets-130228042149-phpapp02/85/Regexp-secrets-53-320.jpg)
![file.txtn<script>alert(‘hello’)</script>
/^[w.-+]+$/
Match succeeds
ActiveRecord validation succeeds](https://image.slidesharecdn.com/regexpsecrets-130228042149-phpapp02/85/Regexp-secrets-54-320.jpg)
![file.txtn<script>alert(‘hello’)</script>
/A[w.-+]+z/](https://image.slidesharecdn.com/regexpsecrets-130228042149-phpapp02/85/Regexp-secrets-55-320.jpg)
![file.txtn<script>alert(‘hello’)</script>
/A[w.-+]+z/
Match fails
ActiveRecord validation fails](https://image.slidesharecdn.com/regexpsecrets-130228042149-phpapp02/85/Regexp-secrets-56-320.jpg)
![Prefer Character Class
to Alterations
require 'benchmark'
# simple benchmark for alternations and character class
n = 5_000
str = 'cafebabedeadbeef'*5_000
Benchmark.bmbm do |x|
x.report('alternation') do
str =~ /^(a|b|c|d|e|f)+$/
end
x.report('character class') do
str =~ /^[a-f]+$/
end
end](https://image.slidesharecdn.com/regexpsecrets-130228042149-phpapp02/85/Regexp-secrets-57-320.jpg)
![Beware of Character
Classes
# case-insensitively match any non-word character…
# one is unlike the others
'r' =~ /(?i:[W])/
's' =~ /(?i:[W])/ matches, even if 's' is a word character
't' =~ /(?i:[W])/
https://bugs.ruby-lang.org/issues/4044](https://image.slidesharecdn.com/regexpsecrets-130228042149-phpapp02/85/Regexp-secrets-59-320.jpg)
Regexp secrets
- 1. Secrets of Regexp Hiro Asari Red Hat, Inc.
- 2. Let's Talk About Regular Expressions
- 3. Let's Talk About Regular Expressions • There is no regular expression
- 4. Let's Talk About Regular Expressions • A good approximation as a name
- 5. Let's Talk About Regexp
- 6. Some people, when confronted with a problem, think, "I know, I'll use regular expressions." Now they have two problems. Jaime Zawinski 12 Aug, 1997 http://regex.info/blog/2006-09-15/247 http://www.codinghorror.com/blog/2008/06/regular-expressions-now-you-have-two- problems.html The point is not so much the evils of regular expressions, but the evils of overuse of it.
- 7. Formal Language Theory • The Language L • Over Alphabet Σ
- 8. Formal Language Theory • Alphabet Σ={a, b, c, d, e, …, z, λ} (example)
- 9. Formal Language Theory • Alphabet Σ={a, b, c, d, e, …, z, λ} (example) • Words over Σ: "a", "b", "ab", "aequafdhfad"
- 10. Formal Language Theory • Alphabet Σ={a, b, c, d, e, …, z, λ} (example) • Words over Σ: "a", "b", "ab", "aequafdhfad" • Σ*: The set of all words over Σ
- 11. Formal Language over Σ • A subset L of Σ* (with various properties) • L can be finite, and enumerate well-formed words, but often infinite
- 12. Example • Language L over Σ = {a,b} • 'a' is a word • a word may be obtained by appending 'ab' to an existing word • only words thus formed are legal
- 16. Expression • Textual representation of the formal language against which an input is tested whether it is a well-formed word in that language
- 17. Regular Languages • ∅ (empty language) is regular
- 18. Regular Languages • ∅ (empty language) is regular • For each a ∈ Σ (a belongs to Σ), the singleton language {a} is a regular language.
- 19. Regular Languages • ∅ (empty language) is regular • For each a ∈ Σ (a belongs to Σ), the singleton language {a} is a regular language. • If A and B are regular languages, then A ∪ B (union), A•B (concatenation), and A* (Kleene star) are regular languages
- 20. Regular Languages • ∅ (empty language) is regular • For each a ∈ Σ (a belongs to Σ), the singleton language {a} is a regular language. • If A and B are regular languages, then A ∪ B (union), A•B (concatenation), and A* (Kleene star) are regular languages • No other languages over Σ are regular.
- 21. Regular Expressions • Expressions of regular languages
- 22. Regular Expressions ot • Expressions of regular languages N
- 23. Regular? Expressions • It turns out that some expressions are more powerful and expresses non-regular languages • Language of 'squares': (.*)1 • a, aa, aaaa, WikiWiki
- 24. How does Regexp work? • Build a finite state automaton representing a given regular expression • Feed the String to the regular expression and see if the match succeeds
- 25. a a
- 26. ab* a b
- 27. .* .
- 28. a$ a $
- 29. a? a ε
- 30. a|b a b
- 31. (ab|c) a b c
- 32. (ab+|c) b a b c
- 33. Match is attempted at every character, left to right
- 34. /a$/ zyxwvutsrqponmlkjihgfedcba ^ Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward to the end of the line
- 35. /a$/ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward to the end of the line
- 36. /a$/ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward to the end of the line
- 37. /a$/ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward to the end of the line
- 38. /a$/ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ ⋮ zyxwvutsrqponmlkjihgfedcba ^ Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward to the end of the line
- 39. ^s*(.*)s*$ abc d a dfadg ^ abc d a dfadg ^ abc d a dfadg ^ abc d a dfadg ^ # matches 'abc d a dfadg '
- 40. a?a?a?…a?aaa…a def pathological(n=5) Regexp.new('a?' * n + 'a' * n) end 1.upto(40) do |n| print n, ": " print Time.now, "n" if 'a'*n =~ pathological(n) end
- 41. a?a?a?aaa aaa ^
- 42. Regexp tips
- 43. Use /x UP_TO_256 = /b(?:25[0-5] # 250-255 |2[0-4][0-9] # 200-249 |1[0-9][0-9] # 100-199 |[1-9][0-9] # 2-digit numbers |[0-9]) # single-digit numbers b/x IPV4_ADDRESS = /#{UP_TO_256}(?:.#{UP_TO_256}){3}/
- 44. A, z for strings ^, $ for lines • A: the beginning of the string • z: the end of the string • ^: after n • $: before n
- 45. A, z for strings ^, $ for lines • A: the beginning of the string • z: the end of the string • ^: after n • $: before n always in Ruby
- 46. What's the problem? also note the difference in what /m means
- 47. What's the problem? #! /usr/bin/env perl $a = "abcndef"; if ($a =~ /^d/) { print "yesn"; } if ($a =~ /^d/m) { print "yes nown"; } # prints 'yes now' also note the difference in what /m means
- 48. What's the problem? #! /usr/bin/env ruby a = "abcndef"; if (a =~ /^d/) p "yes" end http://guides.rubyonrails.org/security.html#regular-expressions
- 49. Security Implications class File < ActiveRecord::Base validates :name, :format => /^[w.-+]+$/ end http://guides.rubyonrails.org/security.html#regular-expressions
- 53. file.txtn<script>alert(‘hello’)</script> /^[w.-+]+$/
- 54. file.txtn<script>alert(‘hello’)</script> /^[w.-+]+$/ Match succeeds ActiveRecord validation succeeds
- 55. file.txtn<script>alert(‘hello’)</script> /A[w.-+]+z/
- 56. file.txtn<script>alert(‘hello’)</script> /A[w.-+]+z/ Match fails ActiveRecord validation fails
- 57. Prefer Character Class to Alterations require 'benchmark' # simple benchmark for alternations and character class n = 5_000 str = 'cafebabedeadbeef'*5_000 Benchmark.bmbm do |x| x.report('alternation') do str =~ /^(a|b|c|d|e|f)+$/ end x.report('character class') do str =~ /^[a-f]+$/ end end
- 58. Benchmarks Ruby 1.8.7 user system total real alternation 0.030000 0.010000 0.040000 ( 0.036702) character class 0.000000 0.000000 0.000000 ( 0.004704) Ruby 2.0.0 user system total real alternation 0.020000 0.010000 0.030000 ( 0.023139) character class 0.000000 0.000000 0.000000 ( 0.009641) JRuby 1.7.4.dev user system total real alternation 0.030000 0.000000 0.030000 ( 0.021000) character class 0.010000 0.000000 0.010000 ( 0.007000)
- 59. Beware of Character Classes # case-insensitively match any non-word character… # one is unlike the others 'r' =~ /(?i:[W])/ 's' =~ /(?i:[W])/ matches, even if 's' is a word character 't' =~ /(?i:[W])/ https://bugs.ruby-lang.org/issues/4044
- 61. /^1?$|^(11+?)1+$/ Matches '1' or ''
- 62. /^1?$|^(11+?)1+$/ Non-greedily match 2 or more 1's
- 63. /^1?$|^(11+?)1+$/ 1 or more additional times
- 64. /^1?$|^(11+?)1+$/ matches a composite number
- 65. /^1?$|^(11+?)1+$/ Matches a string of 1's if and only if there are a non-prime # of 1's
- 66. Integer#prime? class Integer def prime? "1" * self !~ /^1?$|^(11+?)1+$/ end end No performance guarantee Attributed a Perl hacker Abigail
- 67. • @hiro_asari • Github: BanzaiMan