If you have used Ruby for any length of time you have probably, knowingly or not, used a gem. It could’ve been already installed on your system as part of your Ruby release. You may also have downloaded it with the RubyGems package manager. Irrespective of how you got hold of a gem, you required it as a dependency in your file and used it. But, have you ever wondered what a gem is? What’s inside a Ruby gem? Having written a Ruby gem manifest file in the Writing a Ruby Gem Specification article, it’s only natural to use it to learn how a Ruby package is built.
In this article, I’d like to take you to the RubyGems factory to see where it all begins. We will first learn how to create a Ruby gem and then unpack it and look inside the internals. Whether you intend to share your own gems or continue using third-party ones, knowing the inner workings of a gem will be useful. If anything it will make you appreciate the elegance with which the whole Ruby ecosystem works.
Now, on to the fun.
What is a Gem?
Any popular programming language has an ecosystem of packages that solve various problems. These packages capture reusable functionality and save tons of time for everyone else. To be accessible to everyone, libraries need a central storage place. A service for automated uploading and downloading of named packages referred to as - a registry.
Examples of popular registries are NPM (www.npmjs.com) for JavaScript, Hex (hex.pm) for Elixir and Erlang code, and Crates (crates.io) for Rust. Ruby also has an official place to store and distribute its packages: the rubygems.org. This is a service that provides you with the ability to browse and download hundreds of thousands of Ruby packages called a gem.
Each gem in a registry has a unique name and a release number. The Ruby community has a taste for quirky and whimsical names. Don’t believe me, look at these:
- Cucumber - tool for running automated tests written in plain language.
- Jekyll - a static site generator.
- Nokogiri - HTML and XML parser.
- Sinatra - a minimal web framework.
- Unicorn - a web server.
The vibrant and large ecosystem of Ruby gems plays a considerable role in making the Ruby language as popular as it is now. To be part of this success and to create our own gems, we need to learn about the tool that makes it all possible, the RubyGems package manager.
The RubyGems Package Manager
When a Ruby language is installed on your system it includes a handful of gems. One of these gems is the RubyGems gem (this is a bit meta, a gem to manage gems). The RubyGems is a powerful package manager that will do things for you like:
- Installation of third-party gems.
- Dependencies resolution.
- Searching local and remote packages.
- Inspection of gems including their files and metadata.
- Building and serving
RDoc
documentation. - Creation and publishing of gems.
This is not an exhaustive list but it should give you a taste of how many tasks a full-blown package manager handles. As of this writing, Ruby 2.7 installs RubyGems 3.1.2 on my system. The RubyGems gem provides you with a command-line tool that will support all the before listed tasks called the gem. I agree this is a bit confusing. Both a Ruby package and the tool to manage these packages are named the same.
You may also have used a tool called Bundler to install gems. Both Bundler and RubyGems are partners in the same crime. They help you above all manage gem dependencies. Bundler’s focus is on dealing with the installation of gems for a local application. In contrast, RubyGems manages all the gems installed on your system under a given Ruby version. The plot thickens even more starting from Ruby version 2.6. In this version, the Bundler has been merged into the RubyGems as a default gem alongside packages like irb, logger or fileutils.
For us, the important feature of RubyGems is that it can convert a gem specification into a package. That’s what we’re going to take a closer look at next.
Building a Gem
Most likely you used the gem command to install or uninstall a dependency. There are more than 30 commands available at your fingertips. In particular, the gem build command is what we’re going to look at. Its job is to take a manifest file and turn a bunch of files into a Ruby gem. The typical layout of a gem follows a well-established pattern for organising files. For example, the layout of the emoticon gem that we used to write a manifest file for looks like this:
emoticon/
├── exe
│ └── emote
├── lib
│ ├── emoticon
│ │ └── version.rb
│ └── emoticon.rb
├── CHANGELOG.md
├── LICENSE.txt
├── README.md
└── emoticon.gemspec
For our demonstration, we’re going to use the already created emoticon.gemspec that contains all the information necessary to bake a new gem.
To create a gem from our gemspec, run the gem build
command:
$ gem build emoticon.gemspec
This will display confirmation information in the console:
Successfully built RubyGem
Name: emoticon
Version: 0.1.0
File: emoticon-0.1.0.gem
At this point, you will have the emoticon-0.1.0.gem file in the same directory. Nice!
Under the covers the build command delegates its work to the Gem::Package
class and the class level build
method. This method accepts as an argument Gem::Specification
instance and few options. That’s exactly the instance we used to describe metadata about the emoticon gem in our manifest file. As a reminder here is the emoticon.gemspec snippet, for the sake of clarity:
# frozen_string_literal: true
require_relative "lib/emoticon/version"
gemspec = Gem::Specification.new do |spec|
spec.name = "emoticon"
spec.version = Emoticon::VERSION
...
end
Instead of relying on the gem build
command, we will create a package manually to learn what’s involved. To begin, we create a new file build.rb and load the "rubygems/package"
dependency. As it happens, all we need to do to build a gem is add a couple lines of code. First, we load the gem specification. Next, we invoke the Gem::Package
’s build
method with a gemspec as an argument:
require "rubygems/package"
gemspec = Gem::Specification.load("emoticon.gemspec")
Gem::Package.build(gemspec)
With these two lines in place, we can now execute our build.rb file:
$ ruby build.rb
This produces exactly the same result as before - an emoticon-0.1.0.gem file. Now, we have a gem for further exploration.
What’s Inside a Gem?
The emoticon-0.1.0.gem
has an unusual .gem extension name. But, this file is nothing more than an archived directory packaged using the tar Unix utility. We can take a look inside of the archive and list its content to the console with the tar utility. All we need to do is pass the -t flag for printing to the terminal and -f option for reading from a file:
$ tar -tf emoticon-0.1.0.gem
Inside the tar archive file, we find a set of three gzipped files:
checksums.yaml.gz
data.tar.gz
metadata.gz
If you build your gem and sign it with a cryptographic key then the archive will have a total of six files. Each gzipped file will have a matching signature file:
checksums.yaml.gz
checksums.yaml.gz.sig
data.tar.gz
data.tar.gz.sig
metadata.gz
metadata.gz.sig
What are these signature files?
To create a signature file, the RubyGems selects a digest algorithm from the standard library:
# lib/rubygems/security.rb
module Gem::Security
DIGEST_ALGORITHM =
if defined?(OpenSSL::Digest::SHA256)
OpenSSL::Digest::SHA256
elsif defined?(OpenSSL::Digest::SHA1)
OpenSSL::Digest::SHA1
else
require 'digest'
Digest::SHA512
end
end
In the first step, the digest algorithm is applied to every file in the archive to create a summary of its content. The summary is a unique value of a fixed-length often referred to as a file checksum or digest. Then, a signing key and a certificate are combined into a signing instance of Gem::Security::Signer
. The signer signs the file’s digest and creates a digital signature that is then saved to a file with .sig extension. To see the content of the signature, a base64 encoded binary file, we can use the base64 command:
$ base64 data.tar.gz.sig
In this case, we see some random characters in the terminal output, truncated to save the space:
bAclqtMmrTKFSMFn74w8kwjRrmv/8Wlc1prNXxHfYtqHypaFdnoIOKKPhYAp3D5MKiqmqIsY8VTb8gQvbT/rEcMm40C57fh5JEQ3I7Tkvs0D76nL+cIaqnnqJ1H5Irknhkj+fu...
The signature can be thought of as a software equivalent of what we do in everyday life when we sign a paper document with a pen. By signing a document we verify its accuracy and authenticity. In our case, the signature files provide an extra layer of protection when installing gems in two ways. First, they allow us to verify that the gem source files haven’t changed since they were originally signed. Second, they ensure that the signature belongs to the gem’s author who owns the private signing key.
By default gems are installed as unsigned without checking their file signatures. This leaves the possibility open that a gem may have been tampered with by an attacker. To take advantage of the signatures during the gem’s installation process, you need to use the - -trust-policy option or short flag -P with the highest and most secure policy:
$ gem install emoticon -P HighSecurity
With the signatures files out of the way, we can come back to our checksums, data and metadata compressed files. Our good friend the Gem::Package
has more to teach us about how it performs its packaging process. Diving into the build
method, we discover that all these three archived files are generated using the Gem::Package::TarWriter
class. The tar writer does its work by exposing a few helper methods via an internal DSL (Domain Specific Language) that add content to the main gem archive file:
# lib/rubygems/package.rb
@gem.with_write_io do |gem_io|
Gem::Package::TarWriter.new gem_io do |gem|
add_metadata gem
add_contents gem
add_checksums gem
end
end
Let’s go ahead and extract all the files from the emoticon gem archive:
$ tar -xf emoticon-0.1.0.gem
One by one, we’re going to examine files from the emoticon gem archive so we can learn their purpose.
Inspecting the Files
The first file that we’re going to take under a microscope is a compressed YAML
file called checksums.yaml.gz. The file compression is done with a lesser-known zlib gem from the Ruby standard library. This gem provides an interface to general purpose and portable zlib utility for compressing files. Among the formats it supports is the gzip (.gz) format.
In more detail, the Gem::Package
offloads compression of all the archived files to the gzip_to
method. This method accepts as an argument a file content represented as IO stream. Then it uses the Zlib::GzipWriter
to turn the stream into a gzipped version:
# lib/rubygems/package.rb
def gzip_to(io)
gz_io = Zlib::GzipWriter.new io, Zlib::BEST_COMPRESSION
gz_io.mtime = @build_time
yield gz_io
ensure
gz_io.close
end
Since the checksums.yaml.gz is a binary file we need to do something to view its content. Fortunately, we don’t have to unpack it to look inside, as we can use the gzcat utility:
$ gzcat checksums.yaml.gz
What we see inside at the top level are the keys that correspond with the names of the digest algorithms used. In our case, the sha256 and sha512 algorithms. Underneath each algorithm name are pairs of gzipped file name and its checksum. The checksums are digital summaries of the file content generated with the particular algorithm. For example, the file checksums under the sha256 key are represented as a string of 64 hexadecimal encoded characters. Thus it all looks something like this:
---
SHA256:
metadata.gz: bdac75ac0dab99ec3cdb10692ef70f8b0966ef0aff7fb2c40b3c0ebf0440667d
data.tar.gz: f0004274a83869c47e487eeb06c655b13c14601350eb8f6f4299ea6e0259b2d4
SHA512:
metadata.gz: 03d8c338cee67683cc5e032a4e54153c9a73496e8960173a60b74b31fbe372e8786b86979efd6a923e0842dc132b2d4bdf020eff299b13a970018a31b5503325
data.tar.gz: c5c52e2f82d90cc9d249b2dcc3500b3e29216f5db4049557b365f5051f34df19e5152519793b60b9691e1b951ec54fb6fa8e6c39ef673522cc1e19d72bb01bbf
The purpose of the checksums file is to help verify the integrity of the data.tar.gz and metadata.gz files during the gem installation process.
The next file we’re going to zoom in on is the data.tar.gz. Since this is a gzipped tar archive, to see inside we use the familiar technique:
$ tar -ztf data.tar.gz
This file contains the source code files that were listed in the gem specification. Here is the list from emoticon gem:
CHANGELOG.md
LICENSE.txt
README.md
exe/emote
lib/emoticon.rb
lib/emoticon/version.rb
The final metadata.gz file is a YAML
archive that includes all the metadata extracted from the gem’s manifest file. If we look inside the file:
$ gzcat metadata.gz
We’re greeted with a long output akin to the following truncated version:
--- !ruby/object:Gem::Specification
name: emoticon
version: !ruby/object:Gem::Version
version: 0.1.0
platform: ruby
authors:
- Piotr Murach
autorequire:
bindir: exe
cert_chain: []
date: 2020-03-01 00:00:00.000000000 Z
dependencies:
- !ruby/object:Gem::Dependency
name: tomlrb
requirement: !ruby/object:Gem::Requirement
requirements:
- - "~>"
- !ruby/object:Gem::Version
version: '1.2'
type: :runtime
prerelease: false
version_requirements: !ruby/object:Gem::Requirement
requirements:
- - "~>"
- !ruby/object:Gem::Version
version: '1.2'
...
What’s interesting here are all the !ruby/object
notations. What are these? The YAML
format allows us to serialize simple values like strings, arrays or hashes. But it also can serialize pretty much any Ruby object. By default, YAML
will serialize all the object’s instance variables and their values. For example, if we create a Gemspec
class from a Struct
object with some attributes:
require "yaml"
Gemspec = Struct.new(:name, :date, :version)
gemspec = Gemspec.new("emoticon", Time.now, "0.1.0")
And then print the object content in a YAML
format to the console:
puts YAML.dump(gemspec)
We will see all the instance variables serialized:
--- !ruby/struct:Gemspec
name: emoticon
date: 2020-04-10 14:00:41.077868000 +01:00
version: 0.1.0
Yet, we’re not stuck with only serializing object’s attributes. We can control what attributes YAML
is going to use with the to_yaml_properties
method. For example, this is how the Gem::Requirement
specifies it’s YAML
transformation under the hood:
# lib/rubygems/requirement.rb
class Gem::Requirement
def to_yaml_properties # :nodoc:
["@requirements"]
end
end
In our case, the entire Gem::Specification
is serialized this way. All its attributes that may point to other objects are recursively serialized and saved in the YAML
file.
If you wish to take a look at the specification of any gem installed on your system, you can run the gem spec command:
$ gem spec package-name
This will print all the gem’s metadata information in your console.
You should also notice that the gem specification file is only needed when building a gem. It shouldn’t be part of your release. The metadata is extracted to a YAML
file and that’s what is used during a gem installation.
Conclusion
You may have thought of gems as nondescript blobs of functionality that somehow appear on your system under some weird and wonderful name. My hope is that this article clarified the structure of a gem and how it accomplishes its functionality in the Ruby ecosystem.
After all, we learnt that there is nothing new under the sun. The old and trusted Unix tools for archiving and compression serve as the foundation for packaging gems. We experienced how the succinct YAML format keeps the data readable and machine friendly. We saw how the OpenSSL toolkit adds a security layer to make our gems resilient and dependable.
I don’t know about you but I like to know the nitty-gritty details of the tools I use. I’m in awe and have nothing but great admiration for the developers who provide such solutions. It takes an ingenious effort to turn the concept of reusable code into a practical solution that scales and makes millions of Ruby projects possible!
Thanks for taking the time to read the article and I hope you found it useful.
Stay hungry and creative!