Gemification for Ruby 2.5/3.0
5 likes9,021 views
Future of Ruby standard libraries will focus on gemification. Standard libraries will be extracted out of the Ruby core repository and maintained as default gems or bundled gems in GitHub repositories. This allows libraries to be updated independently of Ruby releases and more easily accept contributions. While this approach has benefits, it also has challenges around maintaining compatibility and complex dependencies. The process of gemification will be gradual to reduce the size of changes.
![self.introduce
=>
{
name: “SHIBATA Hiroshi”,
nickname: “hsbt”,
organizations: [“pepabo”, “ruby_core_team”, “asakusarb”],
commit_bits: [“ruby”, “rake”, “rubygems”, “rdoc”, “psych”, “syck”, “ruby-
build”, “railsgirls”, “railsgirls-jp”, …],
sites: [“hsbt.org”, “ruby-lang.org”, “rubyci.org”, “railsgirls.com”,
“railsgirls.jp”],
}](https://image.slidesharecdn.com/20170918-rubykaigi2017-170918080403/85/Gemification-for-Ruby-2-5-3-0-4-320.jpg)
![Inside default gems
• `tool/rbinstall.rb` put gemspec files for default gems on Ruby
core repository.
• We can release default gems to the rubygems.org. It’s a
Standard library that seems to be installed as a gem.
• Rubygems have a detection method for default gems.
>> Gem.loaded_specs["did_you_mean"].default_gem?
=> false
>> require 'openssl'
=> true
>> Gem.loaded_specs["openssl"].default_gem?
=> true](https://image.slidesharecdn.com/20170918-rubykaigi2017-170918080403/85/Gemification-for-Ruby-2-5-3-0-17-320.jpg)
![A problem of test-unit/minitest
• It has difference behaviors and namespace from released
gems that are test-unit and minitest.
• [Bug #9660] test/unit, minitest & bundler
• We gave up to maintain compatibility with released gems.
• But @zenspider said “I AM against shipping ruby without a
testing library.”](https://image.slidesharecdn.com/20170918-rubykaigi2017-170918080403/85/Gemification-for-Ruby-2-5-3-0-31-320.jpg)
![Make in Ruby
Rake is a Make-like program implemented in Ruby. Tasks and
dependencies are specified in standard Ruby syntax.
task :awesome do
puts :bar
end
task beat: [:awesome] do
puts :buzz
end
task default: :beat](https://image.slidesharecdn.com/20170918-rubykaigi2017-170918080403/85/Gemification-for-Ruby-2-5-3-0-33-320.jpg)
Gemification for Ruby 2.5/3.0
- 1. Future of Ruby standard libraries SHIBATA Hiroshi / GMO Pepabo, inc. 2017.09.18 RubyKaigi 2017 Gemification for Ruby 2.5/3.0
- 3. Executive Officer CPO(Chief Productivity Officer) Director of Business Process Re-engineering Office Hiroshi SHIBATA @hsbt https://www.hsbt.org
- 4. self.introduce => { name: “SHIBATA Hiroshi”, nickname: “hsbt”, organizations: [“pepabo”, “ruby_core_team”, “asakusarb”], commit_bits: [“ruby”, “rake”, “rubygems”, “rdoc”, “psych”, “syck”, “ruby- build”, “railsgirls”, “railsgirls-jp”, …], sites: [“hsbt.org”, “ruby-lang.org”, “rubyci.org”, “railsgirls.com”, “railsgirls.jp”], }
- 7. Library ecosystem of Ruby 1.
- 8. Packaging and Disribution(1) • RAA(Ruby Application Archive) • 2013/08: raa.ruby-lang.org 終了のご報告 https://www.ruby-lang.org/ja/news/ 2013/08/08/rip-raa/ • RubyForge • 2009/10: RubyForge To Be Phased Out, RubyGems.org Takes Over Gem Hosting https://www.infoq.com/news/2009/10/rubyforge-phased-out-rubygemsorg • gems.github.com • 2009/10: Gem Building is Defunct https://www.infoq.com/news/2009/10/ github-stops-gem-building
- 9. Packaging and Disribution(2) • gemcutter.org: • https://github.com/rubygems/gemcutter • You can use `gem yank` command after you did invoke `gem i gemcutter`. • rubygems.org: • gemcutter.org was renamed to rubygems.org. • bundler: • 2010: Released to 1.0.0 version.
- 10. A History of Ruby(Short Ver.) • Ruby 1.8.7 bundled a lot of library named standard library a.k.a stdlib. • Ruby 1.9.x bundled rubygems that is ruby library packaging and distributing. • Many of libraries was born after that. Example for rspec, nokogiri, rake and thor. It’s start time for Ruby language and library ecosystems. • First Authors of famous library is burnout. • Some of useful gems are changed maintainers of second generation. • Ruby 2.4, 2.5 2013 2017 2009 2008
- 11. What’s a standard library?
- 12. What’s Standard library? • We called its “標準添付ライブラリ” in Japanese. • It needs to `require` difference from embedded libraries like String, Thread, etc. • https://tech.pepabo.com/2017/09/11/rubykaigi-2017/ • It can be used without Bundler or RubyGems
- 13. Classification of standard libraries Standard Libraries Default Gems Bundled Gems Pure Ruby 69 1 7 C extensions 23 5 0 This matrix shows number of standard libraries and their classifications in Ruby 2.4.
- 14. What differences these libraries? • Standard Libraries • Upstream: Only Ruby core repository(svn.ruby-lang.org) • Release cycle: 1 year • Default Gems • Upstream: Ruby core repository and GitHub • Release cycle: 1year or maintainers matter • Bundled Gems • Upstream: Only GitHub • Release cycle: Maintainers matter
- 15. What number of these libraries - 2.4.0 to 2.5.0 In Ruby 2.4 Standard Libraries • Pure ruby: 69 • Extensions: 23 Default gems • Pure ruby: 1 • Extensions: 5 Bundled Gems • Pure ruby: 7 • Extensions: 0 In Ruby 2.5 Standard Libraries • Pure ruby: 64 (-5) • Extensions: 14 (-9) Default gems • Pure ruby: 6 (+5) • Extensions: 14 (+9) Bundled Gems • Pure ruby: 7 • Extensions: 0
- 16. Default gems 2.
- 17. Inside default gems • `tool/rbinstall.rb` put gemspec files for default gems on Ruby core repository. • We can release default gems to the rubygems.org. It’s a Standard library that seems to be installed as a gem. • Rubygems have a detection method for default gems. >> Gem.loaded_specs["did_you_mean"].default_gem? => false >> require 'openssl' => true >> Gem.loaded_specs["openssl"].default_gem? => true
- 18. ruby/openssl
- 19. Status of OpenSSL binding • OpenSSL is already extracted default gems. You can update it separated ruby core releases same as rubygems, rdoc, bigdecimal. • https://github.com/ruby/openssl • It’s maintained by @rhenium • Upstream was changed github repository from svn.ruby-lang.org • He aggressively maintains new feature of openssl
- 20. Security release We hard to fix and release these security issue. so all of release maintainer are volunteer work. But If your Ruby supports Default gems/Bundled gems, You can upgrade these gems without upgrading ruby interpreter.
- 21. ruby/psych
- 22. Whats psych? A libyaml wrapper for Ruby It supports bundled libyaml-* sources. You can build psych without libyaml-devel(dev) packages. # Load YAML in to a Ruby object Psych.load('--- foo') # => 'foo' # Emit YAML from a Ruby object Psych.dump("foo") # => "--- foon...n"
- 23. JRuby integration • We need to support Java implementation now. • I guarantee compatibility for other implementation used by Travis. In current status, It compatibility was breaking. Because It’s beta status.
- 24. ruby/rdoc
- 25. Whats rdoc? RDoc produces HTML and online documentation for Ruby projects. # puts listings.product + "n" + listings.price # # _Generates_ # # Chicken Fried Steak # A well messages pattie, breaded and fried. # # Chicken Fried Steak -- 9.95 # A well messages pattie, breaded and fried. # def initialize(str, safe_level=nil, trim_mode=nil, eoutvar='_erbout') @safe_level = safe_level compiler = make_compiler(trim_mode) set_eoutvar(compiler, eoutvar) @src, @encoding, @frozen_string = *compiler.compile(str) @filename = nil @lineno = 0 end
- 26. Current status of rdoc • I moved canonical repository to ruby/rdoc. • I handle a project of rdoc and management releases(not product management) • Our official tracker named “bugs.ruby-lang.org” is high threshold. But GitHub resolve this by pull requests. VS
- 27. Bundled gems 3.
- 28. Inside Bundled gems • It was installed by `tool/rbinstall.rb` on Ruby core repository. • You can see list of Bundled gems via `gems/bundled_gems` • We bundled *.gem and unpacked files to tarball package for Bundled gems. • `make install` installed Bundled gem your box.
- 30. Why separated the test framework? The following libraries uses minitest directly in Ruby 2.3: • rubygems • rdoc Other libraries uses test-unit. rubygems and rdoc are developed at github.com/rubygems/rubygems and github.com/ruby/rdoc. We need to support these libraries and their tests.
- 31. A problem of test-unit/minitest • It has difference behaviors and namespace from released gems that are test-unit and minitest. • [Bug #9660] test/unit, minitest & bundler • We gave up to maintain compatibility with released gems. • But @zenspider said “I AM against shipping ruby without a testing library.”
- 32. ruby/rake
- 33. Make in Ruby Rake is a Make-like program implemented in Ruby. Tasks and dependencies are specified in standard Ruby syntax. task :awesome do puts :bar end task beat: [:awesome] do puts :buzz end task default: :beat
- 34. ruby/rake •Rake was originally created by Jim Weirich, who unfortunately passed away in February 2014. •This repository was originally hosted at github.com/ jimweirich/rake, It has been moved to github.com/ruby/rake by @drbrain •Rake adopted SemVer policy. (v0.9.x to v10.x.y) •@drbrain and @hsbt maintain ruby/rake now.
- 35. Concerns of bundled gems • We couldn’t support to compile C extension on Bundled gems. • Bundled gems couldn’t support cross compilation. Ex. We need to make Date gem to bundled gem before extracted from ruby core.
- 36. Concerns of bundled gems(2) We need test suite for bundled gem with ruby trunk. • https://gist.github.com/unak/a80b03d9a33de59bedb52e2711410e0d • https://github.com/ko1/rubyhackchallenge/issues/25 No one guarantee work bundled gems and develop version of ruby.
- 38. Gemification for standard library https://bugs.ruby-lang.org/issues/5481 • We extracted stdlibs like net-telnet, xmlrpc, rake to bundled gems. • These are extracted under the https://github.com/ruby/ . And shipped on rubygems.org • Other gems are also extracted at the future.
- 39. What’s problem of Gamification? • It issue size is too big for human. • I resolved it used default gems and bundled gems. • We can promote standard libraries to default gems, after that we promote it to bundled gems. • I am moving toward the final goal for Gemification.
- 40. Pros of Gemification • Maintainers can release gem for bugfix, new feature independent with Ruby core. • Easily backport stable version from develop version. Ruby users can use new feature on stable version. • If upstream is available on GitHub, Ruby users easily send patch via Pull request. • Ruby interpreter developper can concentrate development Ruby internal.
- 41. Cons of Gemification • Abandoned and complex dependency on rubygems and bundler. • Maintainers need to maintain ruby core and GitHub repositories both. • It’s hard to maintain compatibility.
- 42. How to merge upstream from others I merged upstream into ruby/ruby periodically using following instructions. ruby and rubygems guarantee to work to test and code each other. it’s the same situation for ruby and rdoc $ git clone https://github.com/ruby/ruby $ git clone https://github.com/rubygems/rubygems $ cd ruby $ rm -rf lib/rubygems test/rubygtems lib/rubygems.rb $ cp -rf ../../rubygems/rubygems/lib/rubygems ./lib $ cp -rf ../../rubygems/rubygems/lib/rubygems.rb ./lib $ cp -rf ../../rubygems/rubygems/test/rubygems ./test $ git checkout lib/rubygems/LICENSE.txt
- 43. backport is hard rubygems still supports Ruby 1.8. % g show a34fb569e41cd87866e644d92a9df4be89b3cad2 test/rubygems/test_gem_package.rb commit a34fb569e41cd87866e644d92a9df4be89b3cad2 Author: Eric Hodel <drbrain@segment7.net> (snip) --- test/rubygems/test_gem_package.rb +++ test/rubygems/test_gem_package.rb @@ -638,7 +638,7 @@ class TestGemPackage < Gem::Package::TarTestCase e.message io end - tf.close! + tf.close! if tf.respond_to? :close! end def test_verify_empty
- 44. RubyGems and Gemification for Ruby 2.5 5.
- 45. What’s rubygems RubyGems is a package management framework for Ruby. • rubygems/rubygems.org: • The Ruby community's gem host. • rubygems.org is maintain by infrastructure team of rubygems. It is different team from rubygems cli team. • rubygems/rubygems: • Command line tool of rubygems • Rubygems are created by Seattle.rb
- 46. Current problem of rubygems Bundler team funded RubyTogether take over rubygems/ rubygems In the past, We easily discuss maintenance plan and policy about ruby and rubygems to @tenderlove and @drbrain. I hard to backport a patch to rubygems from ruby core. Because these patch only need ruby trunk. It is needless stable versions.
- 47. Reserved words on rubygems.org Rubygems block reserved name same as standard libraries. • But you can see: https://rubygems.org/gems/fileutils/versions/ 0.7 • If you install fileutils-0.7, fileutils-0.7 break your ruby environment. Why???
- 48. fiddle fiddle is standard library for wrapper of libffi. But fiddle was already reserved another implementation(https:// github.com/bsm/fiddle) on rubygems.org I did coordinate to transfer above namespace and override CRuby implementation(https://github.com/ruby/ fiddle) now.
- 50. What we will do? I must offer to transfer ownership of these gems to rubygems.org and overwrite it used by ruby stdlib. My plan for reserved gems: ・Transfer request to owners of reserved gems. ・Removed gemification gems from blacklist on rubygems.org. ・Override reserved gems by standard libraries.
- 51. Default gems on Ruby 2.5 bigdecimal (default: 1.3.2) bundler (default: 1.15.4) cmath (default: 0.0.1) csv (default: 0.1.0) date (default: 0.0.1) dbm (default: 1.0.0.beta1) digest (default: 0.1.0) etc (default: 0.2.1) fcntl (default: 0.0.1) fiddle (default: 1.0.0.beta2) fileutils (default: 0.7.2) gdbm (default: 2.0.0.beta1) io-console (default: 0.4.6) ipaddr (default: 1.0.0) json (default: 2.1.0) openssl (default: 2.1.0.beta1) psych (default: 3.0.0.beta3) rdoc (default: 6.0.0.beta2, 5.1.0) scanf (default: 0.0.1) sdbm (default: 0.0.1) stringio (default: 0.0.1) strscan (default: 0.0.1) webrick (default: 1.4.0.beta1) zlib (default: 0.1.0) Current status of Default gems. I’m going to promote following libraries to default gem at Ruby 2.5.0 • matrix • digest • ostruct • stringio
- 52. Gamification for Ruby 3.0 6.
- 53. Reducing Ruby package size •In Ruby 2.5, We added “bundler” to default gems. •Bundler will be integrated RubyGems until Ruby 3.0 releasing. •I will promote all of standard libraries to default gems. •I will promote all of default gems without Rubygems dependencies to bundled gems.
- 54. Bundle Bundler to Ruby core • Bundler uses rspec. • I made `test-bundler` task that is invoking rspec examples of Bundler. ~/D/r/trunk > mk test-bundler ./miniruby -I../../github.com/ruby/ruby/lib -I. -I.ext/common ../../github.com/ruby/ruby/tool/runruby.rb --extout=.ext -- --disable-gems -C "../../github.com/ruby/ruby" bin/gem install --no-ri --no-rdoc --install-dir spec/rspec --conservative 'rspec:~> 3.5' Run options: exclude {:ruby_repo=>true, :rubygems_master=>true, :git=>"=< 2.14.1", :rubygems=>"=< 2.6.13", :ruby=>"=< 2.5.0", :realworld=>true, :sudo=>true} .......................................................................................................... .......................................................................................................... ....................
- 55. Migration status of Rubygems and Bundler • rubygems-2.7.0 uses bundler on only test code. • After bundler-2.0 released, We will target rubygems-3.0 • Ruby 2.5.0 have rubygems(-2.6.13) and bundler(-1.15.4), It helps migration plan of rubygems/bundler.
- 56. Activated problem for Rubygems • Rubygems have a activated problem with default gems. • https://bugs.ruby-lang.org/issues/13847 $ ruby -ropenssl -e 'p OpenSSL::VERSION' "2.0.5" $ cat Gemfile # frozen_string_literal: true source "https://rubygems.org" gem 'openssl', '2.0.4' $ bundle exec ruby -ropenssl -e 'p OpenSSL::VERSION' /path/to/2.4.1/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/runtime.rb:317:in `check_for_activated_spec!': You have already activated openssl 2.0.5, but your Gemfile requires openssl 2.0.4. Prepending `bundle exec` to your command may solve this. (Gem::LoadError)
- 57. require :into https://bugs.ruby-lang.org/issues/10320 Matz has some concerns about ruby internal. • Conflicts versions of a shared library like libyaml-., libssl-, libffi, etc... • Order of LOADED_FEATURE So, This feature is difficult to implement on current ruby specification now. require 'libfile', into: :Lib
- 58. What are requirements of Rubygems? base64 benchmark cgi digest English erb fileutils find io/console monitor net/http net/https openssl optparse pathname pp rbconfig resolv set shellwords socket stringio strscan tempfile thread time timeout tmpdir tsort uri webrick Win32API zlib
- 59. Does Gemification make happy Rubyists? • It’s heavy to maintain like github triage, gem release, resolve dependency. • I should provide a benefit of gemification for Ruby core team.
- 60. Conclusion
- 61. Summary of Today’s talk • I described about a standard libraries of the Ruby language. • I introduced specification of default gems/bundled gems of Ruby. • I shared current status of Rubygems and Gemification projects. • I shared my plan towards Ruby 3.0.
- 62. Towards Ruby 3.0