1
2
3
4
5
6
7

play synth('midi.mid') { |midi|
  midi.env * midi.tone.ramp.filter(
    :lowpass,
    cutoff: midi.frequency + midi.env(range: 300..1200),
    quality: midi.cc(1, range: 2..10)
  )
}

Hey everyone! In this post I want to talk about a long-standing idea in software engineering for making development easier, called domain-specific languages, or DSLs. The term DSL can apply to diagramming layouts, document markup formats, and more, but I want to focus specifically on domain-specific programming languages. In that context, a DSL is a programming language specifically designed for a single purpose or “domain,” like writing web servers, processing audio, creating machine learning pipelines, etc. DSLs are, in my view, the ultimate form of abstraction and expressiveness.

Consider this: when you’re hungry, do you ask for “a saucer-shaped flatbread of roughly 16 inches in diameter, coated with sauce, topped with meat and cheese,” or do you order “a large pizza?” This is how much of a difference there can be between classical approaches to coding and domain-specific languages.

Here’s an example borrowed from later in the article to give you a sneak peek at the advantage a domain-specific language can provide:

1
2
3

get '/' do
  'Hello'
end

1
2
3
4
5
6
7
8
9

class RootController < HttpServer
  def get(request, response)
    if request.path == '/'
      response.body = 'Hello'
    else
      response.status = 404
    end
  end
end

Read on to learn more about domain-specific languages, see how I used Ruby to implement a domain-specific language for audio processing, and compare some real-world DSLs to more traditional programming approaches. My goal is to show you the power of language, abstraction, and Ruby, and if you’re a programmer, convince you to try using and making domain-specific languages yourself.

Why?

Let’s take a step back. What are we actually accomplishing here? Well, as ideas evolve, spread, and coalesce, some ideas become so valuable that we give them names. Our pizza above is one example. This is a form of abstraction, a concept that is central to software engineering. We are taking a bunch of useful details that are always basically the same, and putting them in a box so we can just say “that box there” instead of having to describe everything inside the box in great detail.

A domain-specific programming language, then, is the programming equivalent of creating a new kind of slang or jargon. It makes people who use the language more productive, and if you design it right, you can even make it much, much easier for newcomers to understand what you’re talking about.

At the time of writing this, there are several competing paradigms for structuring software and designing languages: functional, procedural, object-oriented, etc. Object-oriented programming is, perhaps, the most common in “real world” enterprise software. Think of languages like Java, C#, and C++. These all have the benefit of popularity and standardization. But sometimes they come with a cost of significantly lower productivity for certain tasks.

In a language like Java, there’s a lot of what programmers call “boilerplate” code – class declarations, type signatures, getters and setters, checked exceptions, and so on. Building a data processing pipeline could include creating several classes, instantiating those classes, explicitly managing the communication between those instances, etc. This introduces a lot of cognitive overhead to manage the details the language forces upon us. That’s energy that isn’t spent solving the true root problem.

The core reasons to create a domain-specific programming language, even as a mini-language within another programming language like Java or Ruby, are to clarify the flow and intent of the program, simplify the creative process, and amplify the value created per effort invested. In other words, a DSL lets you focus on solving your actual problems, not on managing the fluff of a given programming language.

Ruby?

Now it’s no secret that I’m a huge fan of the Ruby programming language. One of my favorite things about Ruby is how easy it makes the creation of domain-specific languages. Ruby is far from the only way to create a DSL, of course. Other languages use type systems and implicit arguments for similar ends, and you can create a DSL from scratch just like a regular programming language.

Creating a DSL from scratch requires the same steps as creating any other programming language from scratch. You need to define a grammar, create an interpreter, etc. Ruby’s syntax is flexible and powerful enough that you can create something that looks like an entirely custom programming language within valid Ruby.

This ease of creation within Ruby goes hand-in-hand with a common philosophy in the Ruby world: code can be poetry, and as DHH’s “software writer” term implies from his RailsConf 2014 keynote, code is meant to be read by humans just as much as computers (as it happens I gave a lightning talk at RailsConf 2014 offering an engineering counterpoint to this concept of the software writer). As such the Ruby language and the ecosystem around it are heavily influenced by aesthetics as much as function, and it’s this aesthetic drive that yields the language-promoting features of Ruby.

How?

What is it about Ruby that makes it so amenable to the creation of domain-specific languages? Apart from the poetic aesthetic described above, Ruby has a number of key language design decisions that make it uniquely good at DSLs:

• Flexible syntax: optional parentheses, keyword arguments, multi-line method chaining
• Inline callbacks: blocks, yield, instance_exec
• Operator overloading: custom definitions of what + and * mean
• Metaprogramming: introspection, method_missing, runtime code generation, BasicObject

I’ll provide some contrived examples that aren’t necessarily useful themselves, but that show useful patterns that can be applied elsewhere.

Flexible syntax

First, Ruby’s flexible syntax gives a lot of options for structuring code. These can all be functionally identical, but communicate different intentions or present different aesthetics.

We can start by looking at optional parentheses on method calls, a controversial but useful feature:

1
2
3
4
5
6
7
8
9
10
11

def a_method(some = 1, args = 2)
  puts some, args
end

# These are both the same
a_method
a_method()

# And these are both the same
a_method 3, 4
a_method(3, 4)

The RSpec DSL is built around a bunch of objects that expose the right methods in the right places, returning the next type of object, to allow building up complex test expectations. Optional parentheses give RSpec its readability, though it does take some time to get used to:

1
2
3
4
5
6
7
8

something = 1
nothing = 0

# This ...
expect(something).to be > nothing

# ... behaves the same as this
expect(something).to(be().>(nothing))

Keyword arguments make function calls self-documenting in ways that can look like a completely different language:

1
2
3
4
5
6
7
8
9

# Note that Ruby has a built-in method called 'p', so one would want to be
# careful about shadowing it like this.
def p(style: '', content:)
  # Don't copy/paste this -- you need HTML quoting/escaping, etc.
  %Q[<p style=#{style.inspect}>#{content}</p>]
end

p style: 'color: red', content: "You wouldn't download a color"
# => "<p style=\"color: red\">You wouldn't download a color</p>" 

Method chaining is how ActiveRecord makes querying databases much easier, and Ruby allows you to split the chains across multiple lines:

1
2
3
4

User
  .where(active: true)
  .order(created_at: :desc)
  .first(10)

Inline callbacks

Let’s look at Ruby’s block syntax and yield keyword. This allows the methods you call to trade control back and forth with your own code. The block you give following a method is, in some sense, passed as an argument to the function. When the function yields control back to your block, the arguments to yield become the arguments to your block (in this case the single argument x).

1
2
3
4
5
6
7

def meaning
  yield 42
end

meaning { |x|
  puts "The answer is #{x}"
}

This syntax is used by XML builders to create nested XML based on nested Ruby:

1
2
3
4
5
6
7
8
9

doc = Nokogiri::XML::Builder.new do |xml|
  xml.root {
    xml.example {
      xml.nested
    }
  }
end

puts doc.to_xml

This generates the following XML:

1
2
3
4
5
6

<?xml version="1.0"?>
<root>
  <example>
    <nested/>
  </example>
</root>

A block can be given a different execution context with instance_exec, allowing the code within the block to call methods that are not in scope outside the block:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

module Context
  module_function

  def is_for(&block)
    instance_exec(&block)
  end

  def kings
    puts 'If only Discovery had gone differently'
  end
end

# The kings method is not available outside the block
Context.is_for { kings }

Operator overloading

Common to many programming languages, operator overloading in Ruby allows us to redefine what addition, multiplication, comparison, and other operations do. In my mb-sound node graph DSL, this lets me turn + into an audio mixer node, * into an audio multiplier node, etc.

Here’s a quick example of an overloaded + operator:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

class Z
  attr_reader :name

  def initialize(name)
    @name = name
  end

  def +(other)
    puts "Pleased to meet you, #{other.name}; I'm #{@name}."
  end
end

one = Z.new('Thing One')
two = Z.new('Thing Two')

# Prints "Pleased to meet you, Thing One; I'm Thing Two."
two + one

Metaprogramming

Finally, Ruby allows essentially limitless metaprogramming (analyzing and modifying a program while it is running). You can redefine methods, delete methods, create classes, inspect arguments, etc. This lets code behave differently based on who is calling it or the names of the arguments of a block, allows injecting context into a block, etc. Unrestrained, this flexibility leads to an unmaintainable mess, but when used judiciously, we can create beautiful, readable custom languages.

One particular kind of metaprogramming, “introspection” or “reflection,” means that the program can get information about itself while it is running. Code can look up classes and methods by name without having to compile those references ahead of time, and can even get the names of block parameters. Ruby provides powerful introspection methods: constants, public_methods, method, etc.

We can use argument introspection on a block to change what gets passed into the block. You’d never want to do this in ordinary code, but it’s incredibly useful for a domain-specific language. I have some ideas for how to use this in my audio synthesis DSL, for example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

def create(&block)
  # [0] first parameter; [1] name of the parameter
  case block.parameters[0][1]
  when :meaning
    yield 42

  when :beauty
    yield 'in the eye of the beholder'
  end
end

create { |meaning| puts "The answer is #{meaning}" }
create { |beauty| puts "Beauty is #{beauty}" }

# The answer is 42
# Beauty is in the eye of the beholder

Ruby’s method_missing method is especially powerful for creating document builders, configuration languages, etc. Ruby also provides a BasicObject base class with almost no methods defined on it, giving method_missing even more power. My old Hashformer gem used BasicObject and method_missing to record a sequence of method calls to be replayed later as part of a data transformation DSL.

I’ve omitted respond_to_missing? in the examples below, but you’d likely want to define that as well in a production implementation.

1
2
3
4
5
6
7
8
9

class Q
  # Just prints whatever it's given, as an example
  def method_missing(m, *a)
    puts "#{m}(#{a.map(&:inspect).join(', ')})"
  end
end

# Prints hello("there", "friend")
Q.new.hello 'there', 'friend'

The last bit of metaprogramming we’ll look at is runtime code generation. ActiveRecord uses this to create attribute methods for database tables based on schema, for example. Here’s one way this might look, where we define a new method whenever we hit method_missing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

class M
  def method_missing(m)
    puts "I don't have #{m} yet"
    self.singleton_class.define_method(m) do
      puts "I have the #{m}"
    end
  end
end

m = M.new

# Prints "I don't have power yet"
m.power

# Prints "I have the power"
m.power

# Your turn -- predict what these calls will print
m.money
m.money

What?

Now that we have Ruby’s tools in our toolbox, let’s look at some side-by-side examples of DSL code next to traditional code to see how a DSL can simplify, clarify, and amplify ideas.

Web servers

Here’s an example comparing the Sinatra DSL (Sinatra is a Ruby gem for making web backends) with a hypothetical object-oriented design:

1
2
3
4

# Actual example of a Sinatra server
get '/' do
  'Hello'
end

1
2
3
4
5
6
7
8
9
10
11
12
13

# Hypothetical object-oriented demo
# based on a mishmash of the general
# shapes of EM-HttpServer, Rails,
# Spring, etc.
class RootController < HttpServer
  def get(request, response)
    if request.path == '/'
      response.body = 'Hello'
    else
      response.status = 404
    end
  end
end

Synthesizers

My mb-sound project has evolved into a DSL for synthesizing and processing sound. I’m not the first person to do this by any means; there’s Csound, Sonic Pi, and no doubt countless others. But this one is mine.

The code snippet at the start of the article is a fully functional, single-oscillator, four-voice, subtractive synthesizer that will play a MIDI file. You can paste that snippet into the bin/sound.rb command-line interface that comes with mb-sound and play any MIDI file with that little synthesizer.

For the example below, the “Classic” version here is what my synth scripts used to look like before I added synthesizer creation to the mb-sound DSL.

1
2
3
4

# Play sine waves from a file or from realtime MIDI keyboard input
MB::Sound.play MB::Sound.synth(ARGV[0]) { |midi|
  midi.tone * midi.env
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

jack = MB::Sound::JackFFI[]
midi = MB::Sound::MIDI::MIDIFile.new(ARGV[0]) if ARGV[0]&.end_with?('.mid')
manager = MB::Sound::MIDI::Manager.new(jack: jack, input: midi, connect: ARGV[0])

voices = 4.times.map { |i|
  MB::Sound::MIDI::GraphVoice.new(manager: manager) do |midi|
    midi.tone * midi.env
  end
}

pool = MB::Sound::MIDI::VoicePool.new(
  manager,
  voices
)

MB::Sound.play pool

Documents

We’re focusing on domain-specific programming languages, but just for completeness, let’s look at a non-programming-language DSL, HTML. HTML (hypertext markup language) can be described as a DSL for document structure. HTML itself gets pretty wordy, though, so there are even simpler DSLs for generating HTML. Here’s an example of the Slim template language, which mirrors the structure of HTML but with far fewer characters.

1
2
3
4
5

html
  body
    h1 An example of a DSL for HTML
    p This is the first paragraph.
    p This is the second paragraph.

1
2
3
4
5
6
7

<html>
  <body>
    <h1>An example of a DSL for HTML</h1>
    <p>This is the first paragraph.</p>
    <p>This is the second paragraph.</p>
  </body>
</html>

See how the Slim template has a lot less visual noise to distract the eyes? This is one of the key ways a DSL helps with readability and clarity.

Why not?

I’ve heard that some hiring interviewers will ask candidates what their biggest complaints are about their language of choice. Passion and criticism go hand-in-hand, the reasoning goes. Nothing is perfect, there’s no such thing as a free lunch, etc., so being able to acknowledge the faults of the thing you love is a sign of experience and maturity.

The same goes for concepts of abstraction, including domain-specific languages. So what are some of the reasons not to use a DSL?

Maintainability: while using a DSL is meant to be very easy, maintaining the DSL internals can require much more caution and expertise. The power that enables the creation of a good DSL can just as easily lead to an unmaintainable mess of interdependent spaghetti.

The counter to this risk is using off-the-shelf DSLs, holding high standards for DSL creation, and using both automated and manual reviews to maintain code quality.

Training: I’m not personally swayed by this argument, but it’s not uncommon for teams to choose standard technologies to make it easier to bring new engineers up to speed on a project, regardless of the cost to long-term productivity. A good DSL can reduce training time, but there is still the possibility that having a lot of custom technology will make it harder to find, hire, and train qualified engineers.

This risk can be mitigated by, again, using off-the-shelf DSLs, open sourcing DSLs to build a pre-vetted community of users, and choosing technology for total productivity, not just for popularity.

Overengineering: Sometimes boring, standard code is actually less effort. For example, it’s not worth creating a DSL if it will take more effort to make than it will save over its lifetime. In other words, always think about ROI (return on investment) when making engineering decisions, including when deciding whether to use or create a DSL.

So?

One last question to address about DSLs: if you’re creating a DSL using the features built into an existing programming language, aren’t you just creating an API (application programming interface) like any other code library? What makes something a DSL and not just a clever API?

You’re not exactly wrong, hypothetical questioner. The useful distinction is whether you are creating something that looks like it could be its own language, or is dramatically simpler than traditional API design. Here are some examples of how I personally would distinguish a DSL from “just an API:”

• ActiveRecord: the chainable query API is a DSL. The class hierarchy of models is an API, not a DSL. The methods used within a model to define validators and relationships are a DSL.
• RSpec: the expectation syntax, before/after/let blocks, and describe/context/it hierarchy are a DSL. Some of the other ways of interacting with RSpec are an API, e.g. retrieving and inspecting the example object.
• Fluent interfaces: the method chaining I’ve described above is often called a “fluent” interface. The Wikipedia article explicitly describes this as a DSL, and I agree. Though in languages like Java you don’t have Ruby’s method_missing for unlimited naming flexibility, the method chaining layout is a major improvement over calling a constructor and a bunch of non-chainable setters to create and configure an object.

Are we there yet?

Domain-specific languages, applied judiciously, can give developers a significant advantage over “standard” software design. We are all most productive, and the work is the most fun, when the code clearly expresses engineering intent, has a high signal-to-noise ratio with no boilerplate, and maps well to the underlying domain.

What are your thoughts on DSLs? Are you convinced to try using or creating one? Or, do you have horror stories about DSLs gone wrong? What features in your own favorite language – type systems, implicits, patterns, etc. – could you use to create a DSL? How do you manage the internal complexity of a DSL to present a clean exterior surface?

Let me know down in the – wait, there are no comments. So instead share this with your colleagues and start a conversation!