Design Summit - Migrating to Ruby 2 - Joe Rafaniello

Upgrading to Ruby 2.x
Joe Rafaniello
@jrafanie

Agenda
1. History
2. Why upgrade?
3. Ruby 2.1
4. Ruby 2.0
5. "Fall cleanup" of old code
6. Slow tests
7. Building 2.0 appliances
8. Developer setup
9. Links
10. Questions?

History
Tue Nov 1 2011:
First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ

History
Tue Nov 1 2011:
Tue Apr 23 2013:
Ruby 1.9.3 finally...

History
Tue Nov 1 2011:
Tue Apr 23 2013:
540 days???

History
Tue Nov 1 2011:
Tue Apr 23 2013:
540 days???
Lesson learned: Don't wait to upgrade!

Why upgrade?
We're behind!!!
Ruby 1.9.3 is ending
In maintenance until February 23, 2014
Security only mode until February 23, 2015
Ruby 2.0.0 is nearly 20 months old
Ruby 2.1.0 is nearly 10 months old
Ruby 2.2.0 is scheduled for a Christmas release

Why upgrade? ... Because ruby 2.1!
Generational mark and sweep garbage collector
http://tmm1.net/ruby21-rgengc/
String#freeze - reuse String objects
Less objects == less memory == less GC time
Object allocation tracing
http://tmm1.net/ruby21-objspace/
https://github.com/srawlins/allocation_stats
Required keyword arguments
def returns method name
Exception#cause - ActiveRecord::StatementInvalid#cause -> Real error
More...

Useless benchmark
bundle exec rspec spec/models/ems_refresh/refreshers
1.9.3-p545
70.85s user 2.23s system 96% cpu 1:15.98 total

Useless benchmark
1.9.3-p545
2.0.0-p576
54.02s user 2.03s system 95% cpu 58.980 total

Useless benchmark
1.9.3-p545
2.0.0-p576
2.1.3

Example allocation information:
Line number
Number of allocations by object type
Such as:
Running: ./spec/controllers/application_controller/buttons_spec.rb:91
223730 Arrays @ .../activerecord/lib/active_record/result.rb:35
203280 Strings @ .../activerecord/lib/active_record/relation.rb:27

Example allocation information:
Line number
Number of allocations by object type
Such as:
Running: ./spec/controllers/application_controller/buttons_spec.rb:91
223730 Arrays @ .../activerecord/lib/active_record/result.rb:35
203280 Strings @ .../activerecord/lib/active_record/relation.rb:27
See Issue 241 and 762

But that's 2.1, let's get to 2.0 first...

Faster Rails startup
Optimizations were made to speed up 'require'

(master) (1.9.3-p545) + time bundle exec rake environment
bundle exec rake environment 3.51s user 0.63s system 99% cpu 4.148 total

25% faster loading of rails environment!

25% faster loading of rails environment!
Most obvious when:
Running tests
Loading Rails console

Keyword arguments
Simplifies conventions:
Accessing option hash values
Default hash values
Optional / can't handle all cases

Keyword arguments
Example: EmsVmware#vm_connect_all
def vm_connect_all(vm, options={})
defaults = { :onStartup => false }
options = defaults.merge(options)
vm_connect_disconnect_all_connectable_devices(
vm,
true,
options[:onStartup],
options[:user_event]
)
end

Keyword arguments
Example: EmsVmware#vm_connect_all
def vm_connect_all(vm, options={})
defaults = { :onStartup => false }
options = defaults.merge(options)
vm_connect_disconnect_all_connectable_devices(
vm,
true,
options[:onStartup],
options[:user_event]
)
end
With keyword arguments:
def vm_connect_all(vm, user_event: nil, onStartup: false)
vm_connect_disconnect_all_connectable_devices(vm, true, onStartup, user_event)
end

Keyword arguments
Invoking methods not changed
Valid on 1.9.3 / 2.0.0:
vm_connect_all(:vm_object1, :onStartup => true, :user_event => "event1")
vm_connect_all(:vm_object2)
vm_connect_all(:vm_object3, :user_event => "event3")
vm_connect_all(:vm_object3, user_event: "event3")

Keyword arguments
Shortcomings and gotchas:
if - valid hash key / invalid keyword argument
Required keyword arguments added in 2.1
http://magazine.rubyist.net/?Ruby200SpecialEn-kwarg
http://robots.thoughtbot.com/ruby-2-keyword-arguments
http://chriszetter.com/blog/2012/11/02/keyword-arguments-in-ruby-2-
dot-0/

Module#prepend
Problem: We want to debug a slow method.

Module#prepend
Problem: We want to debug a slow method.
Wrap the method so can time it!

Module#prepend
Using alias_method:
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
def run
puts "Sub"
super
end
def run_with
puts "DebugIt"
run_without
end
alias_method :run_without, :run
alias_method :run, :run_with
end

Module#prepend
Using alias_method:
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
def run
puts "Sub"
super
end
def run_with
puts "DebugIt"
run_without
end
Sub is a class with the slow run method...
end

class Parent
def run
puts "Parent"
end
end
class Sub < Parent
def run
puts "Sub"
super
end
def run_with
puts "DebugIt"
run_without
end
end
irb(main):01:0> Sub.new.run
DebugIt
Sub
Parent
Module#prepend
Using alias_method:

class Parent
def run
puts "Parent"
end
end
class Sub < Parent
def run
puts "Sub"
super
end
def run_with
puts "DebugIt"
run_without
end
end
DebugIt
Sub
Parent
YAY! But that's really dirty!
(alias_method_chain)
Module#prepend
Using alias_method:

Module#prepend
Using include:
module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
include DebugIt
def run
puts "Sub"
super
end
end

module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
include DebugIt
def run
puts "Sub"
super
end
end
Module#prepend
Using include:

module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
include DebugIt
def run
puts "Sub"
super
end
end
irb(main):002:0> Sub.ancestors
=> [Sub, DebugIt, Parent, Object, Kernel, BasicObject]
Module#prepend
Using include:

module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
include DebugIt
def run
puts "Sub"
super
end
end
Sub
DebugIt
Parent
Module#prepend
Using include:

module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
include DebugIt
def run
puts "Sub"
super
end
end
Sub
DebugIt
Parent
UGH, Sub's method comes first!
Module#prepend
Using include:

Module#prepend
Using prepend:
module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
prepend DebugIt
def run
puts "Sub"
super
end
end

module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
prepend DebugIt
def run
puts "Sub"
super
end
end
=> [DebugIt, Sub, Parent, Object, Kernel, BasicObject]
Module#prepend
Using prepend:

module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
prepend DebugIt
def run
puts "Sub"
super
end
end
DebugIt
Sub
Parent
Module#prepend
Using prepend:

module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
prepend DebugIt
def run
puts "Sub"
super
end
end
DebugIt
Sub
Parent
Ship it!
... but don't forget to call super!
Module#prepend
Using prepend:

Array of symbols: %i and %I
irb(main):001:0> %i{vmware redhat microsoft}
=> [:vmware, :redhat, :microsoft]

%I allows interpolation:

%I allows interpolation:
irb(main):002:0> prefix = "vm_"
=> "vm_"
irb(main):003:0> %I{#{prefix}vmware #{prefix}redhat #{prefix}microsoft}
=> [:vm_vmware, :vm_redhat, :vm_microsoft]

Refinements
Goal: localize monkey patches
I'm not going to explain them because:
Ruby's open classes
Many gotchas...
See Charles Nutter (@headius/jruby guy) explanation:
http://blog.headius.com/2012/11/refining-ruby.html

Enumerable#lazy
Enumerable methods evaluate left to right
With lazy, chains of enumerations are evaluated right to left

Enumerable#lazy
Enumerable methods evaluate left to right
With lazy, chains of enumerations are evaluated right to left
Ruby may "cheat":
May skip creating intermediate objects
Large collection operations may be optimized

Enumerable#lazy
Example benchmark
(0...1000).select(&:odd?).take(5).to_a
(0...1000).lazy.select(&:odd?).take(5).to_a
What is this doing?

Enumerable#lazy
Example benchmark
(0...1000).select(&:odd?).take(5).to_a
(0...1000).lazy.select(&:odd?).take(5).to_a
What is this doing?
First 5 odd numbers
=> [1, 3, 5, 7, 9]

Enumerable#lazy
require 'benchmark/ips'
Benchmark.ips do |x|
x.report("normal") { (0...1000).select(&:odd?).take(5).to_a }
x.report("lazy") { (0...1000).lazy.select(&:odd?).take(5).to_a }
end

Enumerable#lazy
require 'benchmark/ips'
Benchmark.ips do |x|
x.report("normal") { (0...1000).select(&:odd?).take(5).to_a }
x.report("lazy") { (0...1000).lazy.select(&:odd?).take(5).to_a }
end
Calculating -------------------------------------
normal 1539 i/100ms
lazy 5778 i/100ms
-------------------------------------------------
normal 15123.2 (±2.5%) i/s - 76950 in 5.091373s
lazy 59797.4 (±4.0%) i/s - 300456 in 5.033123s
See http://patshaughnessy.net/2013/4/3/ruby-2-0-works-hard-so-you-can-be-
lazy

__dir__
__dir__ path of the script without the filename
# cat test.rb
puts __dir__
# ruby test.rb
/Users/joerafaniello/Code/test

__dir__
__dir__ path of the script without the filename
# cat test.rb
puts __dir__
# ruby test.rb
/Users/joerafaniello/Code/test
We have 984 instances of File.dirname(__FILE__)!
WAT...Why?

Ruby 2.0 breaking changes and deprecations
Note: We're green on travis, so we're getting close...

Objects don't respond_to? to protected methods
Ruby 1.9.3:
respond_to?(symbol) => public and protected methods
respond_to?(symbol, true) => all methods

Ruby 1.9.3:
respond_to?(symbol) => public and protected methods
Ruby 2.0.0
respond_to?(symbol) => public methods only

class Worker
protected
def run
end
end

class Worker
protected
def run
end
end
Worker.new.respond_to?(:run)
1.9.3 => true
2.0.0 => false

class Worker
protected
def run
end
end
1.9.3 => true
2.0.0 => false
Pass true as second argument...
Worker.new.respond_to?(:run, true)
1.9.3 => true
2.0.0 => true

class Worker
protected
def run
end
end
1.9.3 => true
2.0.0 => false
Pass true as second argument...
Worker.new.respond_to?(:run, true)
1.9.3 => true
2.0.0 => true
See Pull #685 - default_value_for gem

UTF-8 is the default character encoding of ruby
scripts
# cat test.rb
FOO = "222dL256"

scripts
# cat test.rb
FOO = "222dL256"
Ruby 1.9.3
US-ASCII is the default encoding of ruby scripts

scripts
# cat test.rb
FOO = "222dL256"
Ruby 1.9.3
Binary string literals become ASCII-8BIT:

scripts
# cat test.rb
FOO = "222dL256"
Ruby 1.9.3
Binary string literals become ASCII-8BIT:
irb(main):001:0> require './test'
=> true
irb(main):002:0> FOO.encoding
=> #<Encoding:ASCII-8BIT>

scripts
# cat test.rb
FOO = "222dL256"
Ruby 2.0.0
UTF-8 is the default encoding

scripts
# cat test.rb
FOO = "222dL256"
Ruby 2.0.0
UTF-8 is the default encoding
Binary string literals become UTF-8 even if invalid:
=> true
=> #<Encoding:UTF-8>
irb(main):003:0> FOO.valid_encoding?
=> false

scripts
So, what's the problem?
Binary strings are used in many places for vm "fleecing"
Invalid UTF-8 encoded strings != raw binary:

scripts
So, what's the problem?
Binary strings are used in many places for vm "fleecing"
Invalid UTF-8 encoded strings != raw binary:
=> false
irb(main):004:0> FOO == "222dL256".force_encoding("ASCII-8BIT")
=> false

scripts
Solutions:
Force binary on individual Strings:
1.9.3/2.0.0 compatible
Painful on files with many binary string literals

scripts
Solutions:
# cat test.rb
FOO = "222dL256".force_encoding("ASCII-8BIT")

scripts
Solutions:
# cat test.rb
FOO = "222dL256".force_encoding("ASCII-8BIT")
=> true
=> true
irb(main):004:0> FOO == "222dL256".force_encoding("ASCII-8BIT")
=> true

scripts
Solutions:
Use String#b on individual strings
Not compatible with ruby 1.9.3
Copies the String in ASCII-8BIT encoding
Note: String#force_encoding("ASCII-8BIT") modifies the receiver

scripts
Solutions:
Add #encoding magic comment at top
Good option when binary strings are expected

scripts
Solutions:
# cat test.rb
# encoding: US-ASCII
FOO = "222dL256"

scripts
Solutions:
# cat test.rb
# encoding: US-ASCII
FOO = "222dL256"
=> true

Descriptors except 0, 1, 2 are closed in child
processes
Prevents file descriptor leakage
See Pull #682, Issue #459, and https://bugs.ruby-lang.org/issues/5041

processes
Example: shared pipe to communicate data between two processes

processes
Example: shared pipe to communicate data between two processes
Use IO#close_on_exec = false
reader, writer = IO.pipe
writerfd = writer.fileno
my_env["WRITER_FD"] = writerfd.to_s
+
+ writer.close_on_exec = false
+
pid = Kernel.spawn(my_env, "ruby #{SERVER_PATH}VixDiskLibServer.rb",
[:out, :err] => [LOG_FILE, "a"],
:unsetenv_others => true,

String#lines returns an array
Also String#chars, #bytes and #codepoints
Previously, returned enumerators
Above are deprecated for StringIO, IO and friends

StringIO#lines deprecated, so use #each_line instead:
@log_stream.rewind
- lines = @log_stream.lines.to_a
+ lines = @log_stream.each_line.to_a
lines.length.should == 1
line = lines.first.chomp

StringIO#lines deprecated, so use #each_line instead:
@log_stream.rewind
- lines = @log_stream.lines.to_a
+ lines = @log_stream.each_line.to_a
lines.length.should == 1
line = lines.first.chomp
See Pull #714

Ok, great, but are we there yet???

"Fall cleanup" of old code
"...Now I've only been an OpenBSD developer for 11 years, one year less than
this header has existed, but in that brief time, I've learned a thing or two
about deleting obsolete code. It doesn't delete itself. And worse, people
will continue using it until you force them onto a better path."
http://freshbsd.org/commit/openbsd/68dc781944a2c5b90f8b6e1069a4201750c67f94

"...Now I've only been an OpenBSD developer for 11 years, one year less than
this header has existed, but in that brief time, I've learned a thing or two
about deleting obsolete code. It doesn't delete itself. And worse, people
will continue using it until you force them onto a better path."
http://freshbsd.org/commit/openbsd/68dc781944a2c5b90f8b6e1069a4201750c67f94
Git is our friend if we really want it back

Path to ruby 2.0...

Path to ruby 2.0...
83 commits
565 lines added
5,553 lines deleted

Path to ruby 2.0...
83 commits
565 lines added
5,553 lines deleted
A good start?

More "opportunities"
host directory
soap4r/actionwebservice (fork)
handsoap (fork) - note, useful but still forked :-(
ruport (fork)
ziya (fork) - patches rails!
prototype
old rails plugins
old gems
old monkey patches

Slow tests
Tests take 30+ minutes on CI servers
Separate tests
Minimizing setup (database inserts)
Remove invalid/not useful/duplicate tests
Allocation tracing with ruby 2.1
Cut support for 1.9.3 when 2.0 is stable

Building 2.0 appliances
Verified Ruby 2.0 on CentOS appliance:
Appliance startup
SmartState Analysis "fleecing" using vddk
vCenter inventory
Basic reporting

Goal: automate building ruby 2.0 appliances
We currently use ruby 1.9.3 through SCL rpms
Not multi-platform
Restricts updating of some gems
Ruby 2.1 is not yet packaged as SCL rpms

Solution:
rpms for base CentOS OS
rpms needed to build ruby and compiled gems
libxml2-devel, libxslt-devel, etc.

Solution:
Use ruby-install or ruby-build for building ruby
https://github.com/postmodern/ruby-install
https://github.com/sstephenson/ruby-build

Solution:
Use ruby-install or ruby-build for building ruby
https://github.com/postmodern/ruby-install
https://github.com/sstephenson/ruby-build
Let bundler handle what it does well...

Developer setup
2.0.0 is not much different from 1.9.3:
Install using rvm, ruby-install, or ruby-build
Manage with rvm, rbenv, or chruby
Need guinea pigs to try it and document any issues
Others tools, such as rubymine, may require some configuration

Links
2.0 open issues: https://github.com/ManageIQ/manageiq/labels/ruby%202
2.0 closed issues: https://github.com/ManageIQ/manageiq/issues?
q=label%3A%22ruby+2%22+is%3Aclosed
2.0 in depth: http://globaldev.co.uk/2013/03/ruby-2-0-0-in-detail/
2.1 in depth: http://globaldev.co.uk/2014/05/ruby-2-1-in-detail/
Slides available here: https://github.com/jrafanie/manageiq_summit_ruby20
Slides written in markdown using remarkjs: http://remarkjs.com/#1

Design Summit - Migrating to Ruby 2 - Joe Rafaniello

More Related Content

What's hot (20)

Similar to Design Summit - Migrating to Ruby 2 - Joe Rafaniello (20)

More from ManageIQ (20)

Recently uploaded (20)

Design Summit - Migrating to Ruby 2 - Joe Rafaniello