Rubinius :heart: Gitter IM

Full disclosure: Gitter neither solicited nor reviewed this post but they do donate the Rubinius organization Gitter account at no cost so we can have full access to private chat and history. We thank them for this sponsorship.

Gitter IM is the sweet spot of open source, GitHub-centered, near-realtime team communication. We've been using it for months on the Rubinius project and I've been meaning to write this post for months because it was immediately obvious how good it is. However, just before we started using it, I didn't get it. I'm hoping this post will help you see why it is such a great tool.

The Need To Communicate

One of the central, critical aspects of an open source project is communication. We ask a question, share an idea, talk over some code, answer a question, coordinate work, and share general life experiences. Since we are distributed in time zones across the world, we need a balance point between communication that is synchronous (I ask a question and you immediately respond) and asynchronous (I ask a question and at some, undetermined, future time you respond).

Historically, IRC has been the medium for this sort of communication. Many people defend its use and some people are adamant that it's the only valid option for open source projects. Unfortunately, it has significant drawbacks for everyone involved in a project, experienced and beginner alike.

I'm comfortable in IRC and have been for years. But I pay for a VPS to maintain my IRC bouncer so I have history for all the channels I'm in even when I'm not physically online. I also need to maintain that server with security patches and upgrades.

There are alternatives to rolling your own IRC bouncer but the point is that there's a significant cost here. And most importantly, as a newcomer, you have to know something about IRC before you can ask your first question. Unless you are contributing to IRC software, I'm certain your question has nothing to do with IRC.

And that's why the limitations of IRC affect everyone on the project. Numerous times I've logged in to find a question from someone and when I go to reply, I notice they are no longer in the channel. Missed connection, missed opportunity. Bummer. That doesn't only happen with newcomers either. The number of experienced developers that I've interacted with for years on IRC who don't use a bouncer or service is high.

So, we have a vital need to communicate but just getting to the point of asking a question is a big hurdle. However, that's not where the difficulty ends. Even when someone has gotten on IRC, there are more challenges waiting. This is where Gitter really shines. It eliminates common barriers to communication and improves the context for the communication.

Barriers To Communication

On Gitter, your nick or handle is your GitHub user name. So simple; so useful. It may not seem like a big thing: you have a GitHub user name and you have an IRC user name and you created them at different times so they are often different. Big deal; who can't remember two names?

Well, me for one. I need to keep track of a lot of people and not having to even wonder what a person's GitHub user is when I'm chatting with them on Gitter is a huge help. It's also a huge help for people who are new to the project and just getting oriented.

Think about this: when you go from one thing to two things, you've increased complexity by 100%. But as programmers, with our fancy loop constructs, we typically don't think twice (no pun intended) about multiplying by two. If you think that's not a huge deal, perhaps try asking for double your salary. In the "real world", multiplying by two is a big deal.

With Gitter, everyone is who they are on GitHub. And since we're working with code hosted on GitHub, that's an obvious, useful, simplication.

Communicating In Context

Communication happens in a context. It's inseparable from that context. This is the key thing that Gitter gets right, and the thing that I didn't get until I created an account and started poking around.

Your GitHub organization and repositories have corresponding Gitter rooms. Boom! Did you see that? Where do you go to talk about Rubinius? Simple, direct, and (in retrospect) obvious.

These are some of the contexts we are already using:

  1. The Rubinius organization room ( This room is public/private. It's available for everyone who is a member of Rubinius (over 100 people), but it's not visible to people outside the organization. This is a perfect place to hold organization-member wide discussions.
  2. The Rubinius team room: This is a private channel accessible only to the Rubinius team members where we can have a safe space to discuss and share.
  3. The main Rubinius room ( This is a public room where anyone with a GitHub account can stop by to ask questions, leave comments, or interact. We have integrations enabled so we can see when issues are opened, comments are made on issues, code is updated, and Travis CI results are posted.
  4. Individual private discussions: These are rooms where two people can hold a conversation in private.
  5. Other project rooms: These are public rooms corresponding to other Rubinius repositories. Rubinius is a complex project with many parts. Having a dedicated room to discuss a specific project (for instance, the Rubinius bytecode compiler), provides a good way to put communication in context and cut down on the cross-communication that happens when a project has a single IRC channel. (Most projects I've been involved with had at most two IRC channels.)
  6. Other private group rooms: These can be created as-needed by people leading specific parts of the Rubinius project. This is not being used extensively yet, but with the work we're doing to define project roles, this will be a very useful tool.

Of course, all of this is possible with IRC. But it requires specific, extra work. It's built-in with Gitter, and that is the best part of the service. It's not all of why it's good, but it's an essential aspect.

Another aspect of context is the consistency of the experience. The Gitter app or the website function equivalently (I'm explicitly excluding the Gitter IRC bridge from this). With IRC, there are such a variety of clients that you don't necessarily know what the other person is seeing. With newcomers, this can impact the ability to communicate well. With Gitter, I can better evaluate if how I'm communicating is helpful or not because I can be reasonably sure they are seeing what I'm seeing.

Communicating About Code

Finally, since we're working on an open source project, a lot of the communication is about code. With GitHub flavo(u)red Markdown, communicating about code is really nice.

It seems like a simple thing, but it's not. For example, HipChat gets this horribly wrong. Communicating about code in HipChat is like using Notepad.exe (sorry, Notepad diehards) to program. It's possible, but only by stretching the meaning of the word beyond recognition.

We often make the assumption that code is concrete, but it's usually quite complex and being able to communicate about it well and visually is tremendously useful. And even when not communicating about code, the formatting available with Markdown makes communicating better and more enjoyable.

So, those are the reasons I think Gitter is awesome. Have you used it? Do you like or dislike it? What do you use instead and why do you like that? We'd love to hear from you:

Who's Using Ruby (or Not), for What, and Why?

If the only constant is change, as Heraclitus said, then we should constantly be trying to understand what's changing, why, and how it affects us.

The way we build, test, deploy, maintain, and support applications has changed a lot in the past five years. Containers, microservices, the growing number of connected devices (IoT, if you must), and the sorts of applications people are building with Ember, Angular, React and Meteor are big, not small, changes.

Containers, in particular, are intriguing because of how central they are becoming to much of what is changing about the application development landscape. At his recent keynote at MesosCon, Adrian Cockcroft showed the following slide when talking about adoption of Docker.

Adoption of Dockero

How does Ruby and Rails fit into this rapidly changing landscape? If you search Google for "what companies use Ruby or Rails" there's not a lot of specific details about what people are doing with Rails. How does Rails fit into a containerized microservices environment? How are people building services with Rails, or are they? I'm curious about this and I think a lot of the new developers learning Ruby in this changing environment would be as well.

So, I've put together a really brief survey; I'd love to hear more about why people are using Ruby, or why they are not.

Survey: Who's Using Ruby (or Not), for What, and Why?

Based on the responses I get, I'd also like to do a few Google hangouts with people on both sides of the question. It's one thing to go to a conference and hear an advocate for Ruby tell you reasons why it's a good choice. It's perhaps more informative and entertaining to watch a couple knowledgeable people debate its merits drawing from specific experiences working with Ruby or competing technologies.

Ruby's support for rapid development and Rails' emphasis on convention over coniguration both seem well-suited for the world of containers and microservices. At the same time, we've seen a lot of public examples of companies switching away from these to scale their applications and infrastructure. I'm curious what people we haven't heard from are doing. Let's look at some data on that.

EDIT: The formatting for the link to the survey has been improved.

The Next 10 Million Programmers

This post is based on a talk given 2015 July 23 at So Coded 2015 in Hamburg, Germany. The video and the slides are also available. Thanks to Enova for sponsoring me to speak.

We live in an amazing time.

Recently, Pluto has been getting a lot of attention. In about 100 years, we went from the Wright brothers and the beginning of manned flight to sending a robotic spaceship billions of miles to fly within a few thousand miles of Pluto, capture around 50gb of data, and start sending it back to Earth. Ponder that for a moment, about 100 years.

Each of us with a smart phone holds more power for digital computation in our pockets than existed in the entire solar system 100 years ago. That means there are hundreds of millions of times more power for digital computation than existed 100 years ago.

But, we're not hundreds of millions of times smarter, healthier, happier, or wealthier. We have definitely made some progress, but by comparison to the power we have at our disposal, it doesn't even seem like modest progress. And in some cases, it's not progress at all. U.S. median household income in 2013 was 1% lower than it was in 1989.

Furthermore, the opinion that we're not very good at building computer systems is not particularly controversial. We are constantly reminded of our failures. It seems like every day there is news of some company's systems being compromised. Just recently it was vulnerabilities in Jeep's systems that would allow an attacker to remotely gain full control of an operating vehicle. Terrifying. This situation demands critical and effective changes in the way we build software systems.

So, this is the problem that faces us: How do we more effectively use this massive power of computation to help people be healthier, happier, and wealthier, making the world a better place to live?

My answer to these problems is that we need to foster a Cambrian explosion of programming languages. The Cambrian explosion was a period in Earth's history where most of the genetic diversity (and actually much more) originated. There are multiple theories about why this happened, but it's clear that some aspect of the biosphere, whether only evolution or whether geological or climatic change (or all of them), supported it.

There are two parts to my proposal: 1. programming languages themselves; and 2. increasing the support for people to experiment on the design of programming languages.

Before we get to that, let's first look at where we spend most of our effort today working with computer systems. Then we'll look at why we should focus on programming languages, and finally, ideas for how to foster this work based on my experiences working on Rubinius.

Programming Software Systems

As programmers, a rather small portion of our time is spent strictly writing lines of code in some particular programming language. Much of our time is spent interacting with or working on other aspects of software systems:

  1. New architectures: The current focus tends to be on a refinement of Service-oriented Architecture called "microservices". This architecture does improve the flexibility of systems but introduces its own set of complexities. Wholesale changes to a system's architecture are disruptive and costly. Also, architecture is about the arrangement of parts; the parts themselves must still be built well, and building these require languages. By itself, improvements in architecture will not help us build better software systems. However, better languages may help us build more effective architectures.
  2. New infrastructures: distributed systems, "Cloud", containers, container OSes, and commodity hardware are all providing opportunities for reduced cost and greater availability and reliability of compute resources. The emphasis here is on opportunity because they don't magically have these benefits, they must be used and managed effectively. They also have significant risk, the most important being that you are dependent on vast areas of the system not under your control when using someone else's "Cloud" infrastructure or platform-as-a-service. As with architecture, this significant potential benefit doesn't help us build better software systems by itself.
  3. New platforms: iOS and Android native apps and "one-page" JavaScript apps are increasingly common today. However, the different platforms impose substantial organizational cost. Different programming languages are needed, typically Java for Android, and Objective-C or Swift for iOS. This likely means different developers for different platforms. Utility libraries are often platform specific. There is also the increased complexity of designing and testing apps for these multiple platforms.
  4. New frameworks: (Or anti-frameworks--the small modules approach currently popular in some areas of the Node/JavaScript community). Frameworks are critically language dependent and consequently are not typically suitable for evolving a system. For example, you won't use Rails and Django together on the same application. So it's not possible to leverage a feature in Django without re-creating it in Rails. Even when frameworks coexist, like Rails and Ember, they do so by staying on opposite sides of the room as it were.
  5. New development methodologies: All development methodologies are substitutes for running code. Sometimes this is a good thing, but usually it's a bad thing. People appear to want to do anything to avoid shipping running code. Activities that help clarify the assumptions about a problem and sketch experiments to validate or invalidate the assumptions should be the limit of development methodology. The artifacts of the development methodology present their own language problem. The point is, development methodologies have only a minor role to play in building improved software systems.
  6. New "general-purpose" programming languages: These must balance competing interests and resolve that tension in a reasonably generic manner. There are fundamental trade-offs in computation, like space versus time, and different circumstances are often best served with different compromises. General purpose languages can't play favorites. Due to their general nature, they typically provide many more language features than are needed for a particular task, but all those features need to be consistent and interoperate, increasing the cost and time to implement a general purpose language.

Despite their limitations, in each of these areas we see progress in more effectively and efficiently solving problems. There are two big issues I see that tend to be tightly coupled to these different areas: 1. the existence of "enshrined problems": components or processes that are "Too Big To Replace" and continually impose the cost of working around their failures; and 2. they don't mix well, forcing wholesale replacement to move forward. For example, you won't successfully build microservices with traditional enterprise architecture or waterfall development methodologies.

Cost Gravity

Looking at these six areas, we can see that languages are an essential aspect of every one. What's the significance of this? Let's step back and ask the question: Why do we have hundreds of millions of times more compute power than 100 years ago?

The answer to that is this idea of cost gravity, an idea I learned from Pieter Hintjen's book Culture & Empire: Digital Revolution. You've likely heard of the idea, too, expressed as Moore's law: the number of transistors on a chip doubles approximately every 18 months.

The idea is simple: over time, the cost of things goes down according to an exponential graph. The reason for that is because living things process and share information. Every living thing does this. And that is why we need to focus on languages; our languages are the most powerful tool we have for processing information.

Elaborating on this idea of cost gravity a bit, there are three key inflection points to consider:

  1. Impossible to possible;
  2. Possible to general purpose; and
  3. General purpose to specialized.

Computers are a good example of this dynamic. At one point, they were essentially impossible as the cost was too high. This doesn't mean just the material costs or cost to manufacture one, but also the cost to do the R&D necessary to even build one. At some point, the cost dropped enough that IBM, for instance, could build custom mainframes for certain companies. Eventually, the cost dropped enough that a middle-class household could afford a "general purpose" personal computer. Costs continue to drop and now a typical computer, for example, your smart phone, is actually a composition of multiple computers: the CPU, GPU, Flash memory controllers, accelerometer, etc.

Programming languages are stuck in the general purpose section of the cost gravity graph. I say, stuck, because at least in the last ten years, programming languages are behind the cost curve. It may be five years or it may be a little more than ten years, but I'm convinced that the cost to create a programming language today is far less than the cost we are paying to create one. Cost gravity assures us that the cost has fallen, but we are not capitalizing on this reduction in cost. In other words, we're paying a tax on all the things we do with computers by not taking advantage of the falling costs to innovate with programming languages.

Why are programming languages stuck behind the cost curve? I argue that they are behind the cost curve because we don't have enough people working on programming languages.

Recently, Business Insider ran a short piece on the 12 most influential programmers today. Here they are:

12 most influential programmers

What's wrong with this picture? It's basically twelve white guys. There are certainly no people of color and especially no women of color. You, whoever you are, are also not in this picture. Equality is extremely important for the benefit of everyone. Diversity of viewpoints is important, but diversity of problems, and empowering people to solve their problems, is even more important.

So, here's where we are: We have this amazing power for processing information and we need languages to use that power effectively. How do we foster creating new languages? There are two parts: 1. who do we need to make programming languages? and 2. what tools do we provide?

The answer to the question of "Who?" is simple: we need as diverse a group of people as possible.

The answer to the question of "What tools?" is also pretty simple: we're mostly already using them and have been for the past 60 years or so. These include parsers, compilers, and virtual machines.

Who We Need To Experiment With Programming Languages

Starting with who, how do we make this technology accessible to the people we want to reach and how do we structure interactions to promote their success? The answers is to build a distributed network of people who collaborate using the advice process.

The Advice Process

I learned this idea from a book, Reinventing Organizations by Frederic Laloux. It's a simple and radical idea. Instead of needing some arbitrary authority to make decisions, people are generally empowered to make decisions guided by the advice process:

  1. one needs to describe the problem;
  2. one needs to explain their solution (decision);
  3. one must seek input from people who would be affected by the decision; and
  4. one must seek advice from people with relevant expertise.

A Network Of Collaboration

In a typical hierarchical organization, one's authority to make decisions (ie solve problems) derives from one's position in the structure. Near the bottom of the hierarchy, one has very little authority. As one's position moves up towards the top of the hierarchy, one has more authority. Changing to a hub-and-spoke structure where one or a few people are at the center and the rest are mostly of equal level around that core doesn't fundamentally change the situation. In fact, it probably distributes even less authority than a typical hierarchy. Open source projects often have this hub-and-spoke structure.

A distributed network contrasts with the typical hierarchy or hub-and-spoke structure of an organization. One's position in the network shows how one collaborates with others. People are explictly empowered to make decisions and supported by a process that helps them do so effectively. If someone makes a poor decision (ie the outcome is not positive), that person or any other person can help fix the problem.


Along with the distributed network structure and the advice process for making decisions, we also need to make roles explicit. Defining the roles doesn't mean that only certain people can do them, nor does it mean that they are fixed and unchanging. One or more roles can be concerned with helping to define roles!

Roles are necessary to give people an opportunity to see if they relate to the work and have, or want to learn, the necessary skills. Some roles may collaborate often and some not as much. Explicit roles are also needed to counter the tendency to see coding as the only important activity. In fact, most of the important roles have nothing to do with coding.

Some examples of roles that I think are needed:

  1. Diversity Architect
  2. Community Wellness Custodian
  3. Accessibility Advocate
  4. Readability Guardian
  5. Under-represented Groups Seeker
  6. Roles Designer
  7. Discussion Facilitator

So, the distributed network structure, decision making with the advice process, and roles help with Who? Now, what tools do we need?

What Tools Foster Programming Language Experimentation

First, here is a list of things that are common to general-purpose programming languages that I don't want:

  1. Don't force choosing between static and dynamic typing unnecessarily;
  2. Don't force using a single type system;
  3. Don't force choosing between compiled to machine code ahead-of-time and using a managed runtime;
  4. Don't force choosing a programming paradigm unnecessarily (eg everything is an object or pure, lazy FP)

These four limitations of most current systems promote certain facilities that are useful in certain contexts above all others and force every solution to a problem to confirm to their limitations. That is wasteful and unnecessary. They also feed endless, and mostly useless, debates and fights about "the right way to do it".

What I do want to promote are these three things:

  1. Enabling a gradual process, over and over again, to build systems that track the cost gravity curve instead of repeatedly building systems that deviate significantly from it over time. In other words, optimize for learning;
  2. Enabling low cost experiments with existing systems rather than needing to completely step outside the system to experiment; and
  3. Building a reliable system out of unreliable parts that we know will have failures and bugs.

These are the things that I think are fundamental to achieving this:

Instruction Set

A "machine" with an instruction set is a ridiculously useful abstraction that we have invented. However, people usually, incorrectly, fixate on an irrelevant attribute of a virtual machine: whether it has a stack or register based design.

In fact, stack and register machines are equivalent, but each has a slight advantage in certain contexts. Stack machines are very easy to write compilers for and the bytecode tends to be somewhat smaller than register bytecode due to the implicit stack locations. Register bytecode tends to be easier to write analysis tools for due to the explicit register locations. In Rubinius we can have both.

There are actually five distinct categories of instructions needed:

  1. Stack
  2. Register
  3. Parsing
  4. Assertion
  5. Instrumentation

Parsing instructions provide optimized features to process a stream of bytes into a structure suitable for generating a stream of instructions. Having first-class parsing instructions significantly simplifies building a system up from simpler parts by parsing a simple language to bootstrap a more complex language.

Assertion instructions can halt the virtual machine or perform some other activity based on the correctness of the computation being performed. They cannot otherwise affect the computation. This makes them suitable for use during development or selectively in production, with the guarantee that the computation itself cannot be changed.

Likewise, instrumentation instructions can provide other dimensions of information about the computation, for example, tracing values or interactions in the code, again with the guarantee that the computation is not being changed. By ensuring these introduce very little performance impact, and not being subject to failure modes like lock inversion, re-entrancy, or unexpected recursion, these instrumentation instructions are safe to use dynamically on production code.

Data and Objects

Functions are for data, objects are for interactions. Parsing a stream of bytes representing HTML or JSON into a structure to manipulate is a thing that functions excel at. You don't need to interoperate with bytes. They don't change. Furthermore, the function is completely defined for any possible byte input.

On the other hand, an agent in a complex system needs to have the absolute minimum coupling with other parts of the system or adaptation of the system as a whole won't be possible. Consider the analogy of ordering your latte at the coffee shop. If the you and the barista must have exactly the same opinions about every aspect of coffee to process your order, you're never going to get your drink.

Both of these abstractions are useful in particular contexts to solve particular problems and switching between them should be easy. Experimenting with a solution may use the loose coupling of objects and once the solution is well understood, it may be possible to define data and functions and isolate this now rigid component in a way that works well within the overall system.

Universal AST and Intermediate Representations

The Clojure compiler doesn't share anything with the Haskell compiler, which doesn't share anything with the Go compiler, which doesn't share anything with the Elixir compiler, and on and on. This is ridiculously wasteful. Fundamentally, compiling Clojure has more in common with compiling Go or Elixir or whatever than any differences. To reduce the cost of language experimentation, we must make the tools needed to build the languages as broadly accessible, and sharable, as possible.

Code Database

The data structure that is created to execute a program (eg machine code, bytecode, expression tree) is only one representation of the code. A code database enables associating an arbitrary number of dimensions of related data. This can support, for example, much more extensive analysis by associating runtime data with a dynamically typed program.

Switching Between Machine Code and Managed Runtime

The path of developing a program is never a straight line. It's a chaotic thread of experimentation, learning, mistakes, and insights. Forcing this complex process to only function within the confines of a completely compiled program wastes too much time. The system should support fluidly, and repeatedly, switching between compiling a program to a single, machine-code executable or experimenting in a managed-code runtime.

Ubiquitous Tools

Tools for doing language analysis and exploring the behavior of running code, without favoring a particular programming paradigm, are essential for programming language experimentation. Measuring, profiling, debugging, monitoring, and analyzing code must be available at all times, during development and in production.

Arguments Against Specialized Languages

The only argument I've heard against language experimentation (besides the obviously false one that language X is all we need) is the idea that if people build many languages, the cost of learning them will be too high. This fear derives from the fallacy that a world of specialized languages is just like our world of "general purpose" languages multiplied N times.

First, most people will never learn most languages, and that's fine. You may create a language to solve a particular problem you have and that's it. And greater experimentation will inevitably finds its way into languages that many people do use.

Second, if the cost for a group to learn the language is more than offset by the increased utility (ie less cost to accomplish the work), then it learning the language is the best thing to do. This is why we learn languages now.

Third, we all already know many languages and dialects even when we don't recognize them as such. Using a CLI is using a language; using a GUI is using a language.

No Free Lunch

There is no free lunch. Programmers seem to love this idea that there's this one weird trick that can make everything work perfectly. Functional programming, static typing, containers, register virtual machines, etc. There is certainly optimization possible, but there are fundamental limits always. You're not going to exceed the speed of light. You can't infinitely compress information. You can't search a document arbitrarily fast. The fascinating thing to me is that in many areas we don't know what these limits are. The way we find out is by bringing to bear more powerful tools. Languages are the most powerful tools we have for processing information. Let's really make them powerful.

And while there is no free lunch, there is this amazing platform we have created called the internet. With the recent announcement of WebAssembly, we have the basis for taking everything I've described here anywhere and to almost any device. That is awesome. Here's to the next 10 million programmers making the world a better place.

Rubinius Metrics meet InfluxDB part II

We live in a containerized world now and after seeing the complexity of setting all up in the first part Mr Joe Eli McIlvain kindly created this this awesome Docker image with everything ready for seeing some neat graphs about the Rubinius VM.


If you are on Mac OS X like me you can install docker with the Homebrew package manager. It's worth to mention that Docker setup in OS X these days is a bit trickier because it depends on Linux kernel features. In order to run it in non-Linux operating systems you have to install boot2docker (which in essence is a Linux based VirtualBox virtual machine in which Docker will run). Having said that, here is an outline of the whole process:

  1. Install VirtualBox.
  2. Install boot2docker and Docker.
  3. Setup boot2docker and Docker.
  4. Setup and start the Rubinius Influxdb-Grafana Docker container.
  5. Run Rubinius enabling the StatsD metrics.

Installing VirtualBox

You can use the Homebrew-cask brew extension to install applications distributed as binaries. You can install it using brew itself:

$ brew install caskroom/cask/brew-cask

And after that use Homebrew-cask to install VirtualBox:

$ brew cask install virtualbox

Alternatively you can download VirtualBox for your operating system from here or install it using your package manager.

Installing boot2docker and Docker

Here is the command for installing both Docker and boot2docker:

$ brew install boot2docker

If you are wondering why there isn't an explicit section in how to install Docker is because it is a dependency of boot2docker in Homebrew. However, you can check Docker's official installation guide for detailed information about how to install Docker in your OS or try installing it with your package manager.

Setting up boot2docker and Docker

On a Linux based operating system you have to start the docker daemon and of course, you can skip this step.

On Mac OS X the first time you have to initialize the boot2docker virtual machine:

$ boot2docker init

The first time the ISO image will be fetched, so probably you will have to wait a bit. If everything goes well you are now able to start the boot2docker VM:

$ boot2docker start

You should see something like this:

Waiting for VM and Docker daemon to start...
Writing /Users/goyox86/.boot2docker/certs/boot2docker-vm/ca.pem
Writing /Users/goyox86/.boot2docker/certs/boot2docker-vm/cert.pem
Writing /Users/goyox86/.boot2docker/certs/boot2docker-vm/key.pem

To connect the Docker client to the Docker daemon, please set:
    export DOCKER_HOST=tcp://
    export DOCKER_CERT_PATH=/Users/goyox86/.boot2docker/certs/boot2docker-vm
    export DOCKER_TLS_VERIFY=1

The last three lines are important because the Docker client uses it to connect to the Docker daemon. You can either copy and paste these lines in your current terminal session or put them in your shell's init file.

Setting up the rubinius/influxdb-grafana Docker container.

When everything is in place you can go to the homepage of the Rubinius InfluxDB-Grafana container read a bit the instructions to start it or just be lazy like me and run:

If you are on a Linux based OS:

 docker run -d \
  -p 8125:8125/udp \
  -p 8086:8086 \
  -p 80:80 \

Or a Mac OS X one:

docker run -d \
  -p 8125:8125/udp \
  -p 8086:8086 \
  -p 80:80 \

Notes for Mac OS X users:

Why an environment variable on Mac OS X?

From rubinius/influxdb-grafana container site:

"Because docker relies on features of the Linux kernel, it does not run containers natively in Mac OS X - it hosts containers inside of a Linux VM called boot2docker. One consequence of this is that ports mapped to the docker host from containers are not mapped to localhost of OS X, but to the boot2docker host. Therefore, in all of the above commands, OS X users should replace localhost with the IP address given by running boot2docker ip."

We have to be able to know the boot2doecker virtual machine ip address in order to allow Grafana (a client side application) to hit the dockerized InfluxDB server.

How to get that boot2docker ip?

$ boot2docker ip

Run Rubinius enabling the StatsD metrics

Enable Rubinius StatsD metrics emitting for your Rubinius process like this:

On Linux based systems:

        -Xsystem.metrics.statsd.server=localhost:8125" \
  rbx # (your app here)

On Mac OS X:

        -Xsystem.metrics.statsd.server=" \
  rbx # (your app here)

Where "" is the boot2docker VM ip address.

Some screenshots while running Rubinius Benchmarks:

RBX Dashboard.png

RBX Dashboard 2.png

RBX Dashboard 3.png

Happy Graphing!

Matz's Ruby Developers Don't Use RubySpec and It's Hurting Ruby

Matz's Ruby team released version 2.2.0 a few days ago. This release fails or errors on multiple RubySpecs, and at least one spec causes a segmentation fault, which is a serious error that can sometimes be a security vulnerability. All of these issues could have been easily avoided if MRI developers used RubySpec.

Full output of MRI 2.2.0 running RubySpec

Ruby is an extremely complex language. It has no formal definition. I created the RubySpec project to provide this definition and to help implementations check correctness. MRI developers have released versions 1.9.3, 2.0.0, 2.1.0 and now 2.2.0 without checking correctness with RubySpec, nor have they created RubySpecs to define the Ruby behaviors they have implemented for these versions, to the detriment of all Ruby programmers and businesses who depend on Ruby.

As of today, I'm ending the RubySpec project. The personal cost of RubySpec over the past eight years, as well as the cost to Rubinius are greater than any benefit derived from the project.

Independent of RubySpec, the Ruby community needs to address this issue. The MRI developers claim to own Ruby, so it's their responsibility to clearly define the semantics of Ruby and ensure the correctness of their implementation.

Instead of contributing to RubySpec, the MRI developers write their own tests. That is their choice to make. As the issues here are complex, I want to provide some historical context and explain why the MRI tests are inadequate.

Ruby Is What Ruby Does

When I first began contributing to Rubinius in late 2006, I knew two things: I wanted Rubinius to be successful, and I wanted it to accurately reflect MRI behavior so that developers could simply switch to Rubinius to run their Ruby programs.

At the time (around version 1.8.3-4), MRI had almost no tests. There were two previous projects to attempt to write tests for Ruby: RubyTest and BFTS. Both of these projects were also very limited and used ad hoc facilities for handling platform differences.

My approach to the specs was simple: The way a Ruby program behaves is the definition of Ruby. So, we began to comprehensively write small snippets of Ruby code to show how Ruby behaved when run by MRI. Then we would match that behavior in Rubinius.

Writing tests for a language as complex as Ruby is tremendously difficult. Even for a single implementation, there are platform issues like endianness and machine word size. Now we needed to also account for different implementations and, with the beginning of Ruby 1.9 development, completely different versions with syntax incompatibilities.

I started developing a consistent set of facilities to handle all these challenges as well as writing a separate spec runner that was compatible with RSpec 2.0 syntax so the specs could be run by RSpec. The custom spec runner used very simple Ruby features to enable nascent Ruby implementations (like Rubinius) to start running specs with very little preparation.

The Birth of RubySpec

On May 10, 2008, just before RailsConf, I created the RubySpec project, by extracting the specs we had been writing for Rubinius, in the hope that MRI and other projects would contribute to it and use it. Some people had already started questioning why MRI did not use the specs.

Later that year, at RubyConf 2008, I gave a talk titled, What Does My Ruby Do about RubySpec. Matz and several other MRI developers attended. Immediately after my talk, contributors to Rubinius sat down with Matz and other MRI developers to discuss their effort to create an ISO specification for Ruby. We asked whether RubySpec could be part of the specification but were told that it was not appropriate to include it.

Fast forward to the pending release of MRI 1.9.2. The transition from 1.8.7 to 1.9 had been torturous. There were a number of behaviors introduced in 1.9.0 and 1.9.1 that were reverted in 1.9.2. The Ruby community was very reluctant to adopt 1.9 because of the confusion about 1.9 behavior, instability of 1.9.0 and 1.9.1, and the cost of migrating code. The MRI developers had started writing more tests, but there was still almost no participation in RubySpec.

That changed suddenly when Yuki Sonoda (@yugui), as the 1.9.2 release maintainer, stated that she would not release 1.9.2 until it passed RubySpec. There was a flurry of activity that all but ceased when 1.9.2 was released.

No release maintainer since then has asserted that requirement. MRI developers have written many MRI tests in the last several years. As described below, there are still Ruby features for which there are no MRI tests, but they are writing tests. However, MRI developers have essentially written no RubySpecs for the 2.0, 2.1, or 2.2 features they have implemented.

The Problem With MRI Tests

Not too long ago, prominent Rubyist Erik Michaels-Ober asked me, "What's wrong with MRI's tests?" I was surprised by his question.

Since Erik, who is an experienced Ruby and Rails developer, asked this question, I imagine other people have wondered the same thing.

From the perspective of adequately defining Ruby semantics and providing a mechanism for other Ruby implementations to check correctness, here are the problems with MRI's tests:

  1. They include MRI implementation details. One very difficult aspect of specifying Ruby involves the boundary between a Fixnum and a Bignum. This impacts almost every method in Array, String, Range, etc. Since MRI is written in C, the machine types supported by C leak into Ruby behavior. One place this happens is the accepted range of values that can be used to index an Array. Since MRI uses a C long, there are some values that are bigger than a Fixnum that can be used to index an Array. For an implementation that isn't dependent on C, these hidden semantics expressed in MRI tests like these for Array make the tests unsuitable.

  2. They include MRI bug details. Rather than improving the general quality and coverage of the tests, MRI adds specific bug conditions to the tests. These bugs are irrelevant to other implementations. Moreover, littering the tests with specific bug cases instead of improving the overall quality makes the test suite harder and harder to maintain over time.

  3. They have no facility for distinguishing versions. The tests are specific to whatever version the MRI code supports. However, it's not that simple. MRI develops features by pushing new commits directly into their Subversion repository trunk (which is somewhat like the master branch under Git), and then periodically pulling certain specific commits into a release branch. This makes it extremely difficult to track the MRI features that are intended for an upcoming version.

  4. They have no facility for distinguishing non-MRI platform issues. As stated in the first point above, there are MRI-specific semantics that are not shared by other implementations. The MRI tests assume these semantics and this makes the tests much more difficult to use.

  5. They include behavior that Matz has explicitly said is undefined. Matz has explicitly said that modifying a collection while iterating is undefined behavior. Yet, the MRI tests include this behavior. Another Ruby implementation must exclude these tests if, for example, they result in an infinite loop. But since MRI can change the tests at any time and add more of these types of tests, this requires constantly checking which tests exhibit undefined behavior and omitting them.

  6. They are not discoverable. The tests are roughly divided into test/ and test/ruby, but locating the tests for a specific class and method is not possible through any standard convention. One must constantly grep the code and attempt to determine if the test is specifically for that method, or if the use of the method is coincidental.

  7. They combine too much behavior into a single test. Ruby includes very complex behaviors and there are many different aspects of those behaviors that may depend, for instance, on which parameters are passed or the kind of parameters passed. This example of Range#bsearch tests with Float values spans 49 lines and is hardly the most confusing test in MRI. Or this test, titled test_misc_0, that asserts something about Arrays. It is extremely difficult to identify the key boundaries of Ruby behavior and implement them correctly.

    Another example is Process.spawn, which has at least 43 distinct forms of passing parameters. It is very difficult to understand that complexity from these MRI tests.

  8. They use ad hoc facilities for setting up state and sub-processes. There are three basic kinds of specs one can write for Ruby: 1) a value computed with no side effects, 2) state change (side effects) internal to the Ruby process, and 3) state change (side effects) external to the Ruby process. The first kind of specs are the simplest and easiest to write. The latter two often require starting a new Ruby process to execute the test. The MRI tests use various different ways to do this, including IO.popen, Process.spawn, system, etc. There are also many different ways for setting up the state for the test.

  9. They are often incomprehensible. This issue of the difficultly of understanding the tests is part of several of the previous points, but it stands on its own as well. It is often not clear at all what behavior the test attempts to show. This makes implementing Ruby much more difficult. For example, what exactly is the behavior here?

  10. They are incomplete. Ruby is complex, so it's not completely surprising that testing Ruby is hard. However, this is MRI. They are defining the behavior of Ruby. Complexity is no excuse whatsoever for incomplete tests. The consequence of incomplete tests is serious. Recently, Rubinius implemented Thread::Backtrace::Location from the documentation in MRI. There are no MRI tests for Thread::Backtrace::Location#path, a method that Rails 4.1 started depending on to initialize Rails. The MRI implementation appears to have several bugs in it. This resulted in a Rails issue, a Rubinius issue, a bunch of time wasted and all our Rails 4.1 applications being broken and requiring a monkey patch while waiting almost a month for Rails 4.1.9 to be released, which still hasn't happened. It is unreasonable that the developers defining Ruby features are not writing tests for them.

The facilities and structure I built for RubySpec address all of these significant problems with MRI tests, except for completeness, which is impossible for RubySpec to solve while MRI continues to change the definition of Ruby without adequately specifying the behavior.

This issue of completeness is extremely difficult given the complex behaviors of Ruby in areas like processing keyword arguments. When calling a method, the object passed as a parameter that is considered to be a candidate for keyword arguments is determined by hidden semantics. A single object will sometimes be split into two separate Hashes, at other times an exception will be raised. The semantics are extremely complex. While writing numerous RubySpecs for this, I encountered several bugs and filed issues with MRI. Some of those bugs persist in Ruby 2.0 and 2.1.

The existing MRI tests are simply inadequate to define the semantics of Ruby and require significantly greater cost than necessary for other implementations to use to implement Ruby behavior.

Developers And Businesses Suffer From Poor Ruby Quality

I think it's unreasonable for the MRI developers not to use and contribute to RubySpec. I don't think it's acceptable to release a stable version that segfaults when running an easily accessible and valuable test suite. Businesses who depend on Ruby and employ developers to write Ruby deserve a more mature, more rigorous design and quality assurance processes.

I have personally discussed RubySpec with Matz on multiple occasions. I have sat with Matz and other MRI developers to discuss RubySpec. I have advocated for using RubySpec in my talk, Toward a Design for Ruby, in the feature request A Ruby Design Process on MRI's feature / bug tracker, in my blog posts explaining my proposed design process (A Ruby Design Process, A Ruby Design Process - Talking Points), and in my petition to adopt the proposed design process.

RubySpec has existed for almost eight years, but has clearly failed to suit the needs of MRI developers. That's disappointing to me, but I no longer view it as a problem I can help solve.

The Future Is Bright

Ultimately, I've made the decision to end RubySpec for the benefit of Rubinius as a project and to support current and future contributors.

There is a significant opportunity cost for Rubinius in supporting RubySpec. We have the ability to experiment with even better approaches to specifying Rubinius features. For example, a literate programming style that combines Rubinius documentation and code examples to serve as tests. Or a custom language that makes the specs easier to write and understand.

Attempting either of these approaches is untenable if broad compatibility across implementations is a requirement. Moreover, Rubinius needs to be free to prioritize efforts toward things that benefit Rubinius, just like all the other Ruby implementations have done.

I am more excited about Rubinius than I have been since early 2007. We continue to write very high quality specifications of the Ruby behavior we support, and all the new features that are being created. RubySpec was born in Rubinius out of a desire for the best Ruby support we could create. Whatever approach we take, that goal is deeply embedded in Rubinius.

Rubinius is getting better every day. It's the easiest way to transition your applications from MRI to a system with a just-in-time machine code compiler, accurate generational garbage collector, an evolving set of comprehensive code analysis tools, and concurrency support for all those CPU cores we have these days. If those things are important to you, we'd love to hear from you.


While the decision to end RubySpec was mine alone and not everyone fully agrees with it, I received tremendously helpful and generous feedback from many people. Special thanks to Chad Slaughter, Tom Mornini, Sophia Shao, Jesse Cooke, Yorick Peterse, Gerlando Piro, Giles Bowkett, Kurt Sussman and Joe Mastey.

Rubinius Metrics meet InfluxDB

A little bit of background

Along with the release of 2.3.0 an exciting feature landed: the basic infrastructure for always-on metrics of Rubinius subsystems. And after reading the 2.3.0 release notes I just fired my IRB session and tried:

=> #<Rubinius::Metrics::Data:0x8c94>

Sweet! Immediately went back to the release notes and again, tried to find how could I access the metrics inside that object:

=> {:""=>0, :""=>0, 
:""=>0, :""=>0, 
:""=>0, :""=>0, 
:"jit.methods.queued"=>0, :""=>0, 
:"jit.methods.compiled"=>0, :""=>0,
:""=>0, :""=>0, 
:""=>0, :""=>0, 
:"memory.young.objects.current"=>0, ...}

And voila! JIT, GC, IO Subsystem, Memory, OS Signals (and more!) stats about the current Rubinius VM running my IRB at my fingertips. I was delighted to see there was so much data accessible in near-real-time through a simple Ruby object. A quick switch back to the release notes revealed something even more promising: built-in support for emitting the metrics to StatsD (a simple daemon for stats aggregation). Are you thinking the same as me? Lets do some graphing!

Looking for a tool for graphing

Ironically, this step was the most difficult. After researching open source tools for graphing everything seemed to point in the same direction: Graphite. The plan was to have Rubinius connected to StatsD, then connect StatsD with Graphite to make some graphs. All of this seemed straightforward except that getting Graphite up and running was in practice, a pain, and after even trying the Docker Image I could not get it up and running. I'm lazy so I gave up on Graphite and started to look for other approaches.

Then, I remembered Brian talking about a time series database InfluxDB on the Rubinius IRC channel. I went to the site and totally fell in love with it, installation was easy, it was working out of the box and seemed that I was in the middle of a lucky strike because while reading InfluxDB documentation I saw a Grafana Dashboards link, followed the rabbit hole and there it was: "An open source, feature rich metrics dashboard and graph editor for Graphite, InfluxDB & OpenTSDB." That sounded like a plan: Rubinius -> StatsD -> InfluxDB -> Graphana. The "StatsD -> InfluxDb" part seemed a little bit scary but after some googling I've had found this handy StatsD backend. I had everything I could wish, lets do some X-RAYS to Rubinius!


Getting InfluxDB

I'm using Mac OS X so for me was matter of:

$ brew install influxdb

But you can check the installation section in the InfluxDB documentation for your operating system. The version used at the time of the writing is v0.8.7.

Starting InfluxDB

Again it can vary depending on the operating system but for Mac OS X you can start it like this:

$ ln -sfv /usr/local/opt/influxdb/*.plist ~/Library/LaunchAgents
$ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.influxdb.plist

Creating the InfluxDB databases

As I mentioned before, all the metrics will be sent to InfluxDB by StatsD to one database (to keep things relatively simple) in our case the "rbx". One of the neat things of InfluxDB is it's admin UI listening at http://localhost:8083. You can read more about how to login into InfluxDB here but the default credentials are user: "root" and password: "root" and should be enough should go there and create a two new databases one named "rbx" and other named "grafana" as I explain later is used by Grafana to store the dashboards.


Getting StatsD

StatsD is a NodeJS package so it can be installed in your system with NPM:

If you don't have NPM installed on your system and you are on OS X you can install NPM (which will install NodeJS which comes with NPM bundled) with this:

$ brew install npm

If you are using another operating system you may find this helpful.

After you have NPM you can:

$ npm -g install statsd

Getting the InfluxDB StatsD backend:

$ npm install -g statsd-influxdb-backend

Creating a configuration file for StatsD

We need a bit of configuration tweaking, but you can use this one:

$ wget -O config.js

This config file has the InfluxDB settings such as enabling the InfluxDB backend, credentials (just use the default ones) and the InfluxDB database to which you will be forwarding the metrics ("rbx"). I've also enabled the debugging option for StatsD because, you know, is extremely difficult for a developer not knowing what's happening ;).

Starting StatsD and testing it

You can now:

$ statsd config.js 

You should see something like:

$ 9 Dec 13:11:37 - reading config file: config.js
$ 9 Dec 13:11:37 - DEBUG: Loading server: ./servers/udp
$ 9 Dec 13:11:37 - server is up
$ 9 Dec 13:11:37 - DEBUG: Loading backend: ./backends/graphite
$ 9 Dec 13:11:37 - DEBUG: Loading backend: statsd-influxdb-backend

Also, you can run a small test to StatsD like this:

$ echo "foo:1|c" | nc -u -w0 8125

After that you can just go to InfluxDB UI follow the "Explore Data" link at the right of the "rbx" database and run the "list series" query. Hopefully, if everything is OK you should be able to see you "foo.counter" there.


Getting Grafana

To get Grafana just go to download the compressed file, place it in the directory of your preference and you can either put the files into the public static files directory of your web server or as I my case just open index.html with your web browser.

For the lazy ones:

$ wget
$ unzip

Configuring Grafana

Grafana has the concept of Datasources from where it pulls the data. It ships with a sample config file we could tweak but as we are lazy here is the one I'm using in my machine. There is nothing esoteric in there we just define two sources: one "rbx" pointing to our "rbx" InfluxDB database (where StatsD is dropping the metrics (remember? :)) and other called "grafana" used internally by Grafana to persist the dashboards.

So the process would be:

$ cd grafana-1.9.0
$ wget -O config.js
$ open index.html

Creating some dashboards

You are almost done! Now you have to go to the Grafana documentation and learn how to make some graphing! (I'm kidding, There is a sample dashboard for seeing Rubinius metrics your only work is to import this dashboard in Grafana (by clicking the small folder 'Open' button in the top right section of the UI, then clicking in 'Import') and start looking at those amazing graphs.

Something like this:

$ wget -O RBX-DASH

Running Rubinius with StatsD support

There will be nothing to draw if you don't start the StatsD metrics emitting in your Rubinius process. You can do this by passing some -X flags to rbx command.

For example (and the most simple would be to fire up an IRB sesion):

$ RBXOPT=" -Xsystem.metrics.statsd.server=" rbx

Or if you are intrepid just clone the (Rubinius Benchmarks) repo and run some benchmarks and see the X-RAYS in action \o/:

$ git clone
$ cd rubinius-benchmark
$ RBXOPT=" -Xsystem.metrics.statsd.server=" ./bin/benchmark -t /Users/goyox86/.rubies/rbx/bin/rbx core

Or even if you are more intrepid start any of your rails apps like this:

$ RBXOPT=" -Xsystem.metrics.statsd.server=" rbx -S bundle exec rails server


There is a bright future coming for all logging, monitoring and analysis for Rubinius, the fact that we now support StatsD directly opens a whole world of possibilities since we can send metrics to anything supported by StatsD, allowing you to X-RAY your Rubinius processes. Imagine sending your Rubinius metrics to NewRelic, Skylight out of the box? Please don't hesitate in sharing your ideas about this even requesting built-in support for other metrics or other targets. Happy Graphing!