It's true that when re-architecting a system, most of the benefits you receive (assuming you're successful) stem from all the hindsight you have, a clean slate, clearer requirements and so on. So I won't go into all the cool stuff we've managed to get into our main system.

What I do want to talk about is something that I believe Clojure really excels at, and that is "incremental automation" of our infrastructure via a REPL.

The problem

We run a multi-tenant SaaS platform but we try to be as white label as possible, since our clients are big enterprises with complex IT and security policies. Clients can bring their own domains and their own mail servers, they need to unblock their spam and phishing filters so that emails go through and other very boring but necessary tweaks.

For a very long time, onboarding a new client has been a manual process that involved a lot of email back-and-forth and also a lot of clicking around in the AWS console, the database provider console, the mail service console and also ssh-ing into a bastion host with access to the prod service to run some database commands and other maintenance scripts.

At the same time, we had a lot of ad-hoc requests from our Customer Success people: This client wants a data dump, the other client deleted something by accident, could we please automate a very tedious process for them etc etc. We tried to do some of this as best as we could, but it has been tedious and annoying: having to copy bash & node scripts around to production servers gets old really fast, plus we were always scared that a fat-finger would do bad things to the production database.

Enter Clojure

When we managed to get Clojure setup on a set of servers, one of the first things I did is to figure out how to run a remote REPL that I could SSH-forward to my local machine and connect my editor to. While still there is an issue with fat-fingering and the danger of developing in production, at least I could write and inspect a whole function in a familiar environment before sending the code across the wire.

What then started to happen is that we'd start writing a lot of tiny one-off functions, that we committed to a catch-all repository. We started with small tasks like generating a one-off report, to updating some DNS entries, to filtering some bad data or even business analytics for our sales team to use; all these tasks transformed from a tedious half-day work that you had to do from scratch every.single.time to a fun and very fast experience.

And at some point, when more and more of our onboarding tasks were automated, we took the plunge, set some time aside, and connected everything to a web form that our non-technical colleagues can now use to do a lot of these tasks. This consolidation took only a handful of days, since it was mostly glue code. I don't believe this would ever get done if we tried to do it with Node.js or any other "conventional" language, since even figuring out the requirements would be a monumental task on its own.

In my opinion, this work was enabled by three very important pillars of Clojure:

a) Interactive development
b) Small, composable functions
c) Access to a rich library ecosystem

Interactive Development

With Node.js, we'd have to either edit a one-off script right at the server, and go back to the shell and running it, or edit locally and scp the script across (and then going back to the shell). Sometimes you might fire up a Mongo or a Node shell but then you'd lose your editor affordances – and nobody wants to type for long into a REPL anyway.

This also meant that every time you wanted to change something, you'd have to re-run your script, losing all your state. Sometime this was OK, but sometimes when you have already done an expensive calculation or fetched a lot of data, it was very slow and annoying to make a mistake.

With Clojure, once the REPL was started and connected, we could just start trying things our right in our editor. That function blew up because of something? Just tweak it, re-evaluate it, run it again with a single keystroke. Not only this process has literally a sub-second overhead, but it also allows you to explore the problem, the data and the solution incrementally, building upon your previous calculations.

Also, we could use all the Clojure's data-exploration functions, our IDEs would show a nice visualisation of the returned data and sometimes we'd even fire up a local real to use a visualisation GUI like the Clojure Inspector, Inspector Jay or REBL.

Small, composable functions

As mentioned before, with Node we'd have to write one-off scripts that had to contain "the universe" – for some reason we never created a proper Node project to collect useful scripts. My hypothesis is that since Node doesn't have an interactive mode and encourages an imperative style for this kind of thing, after doing all the work of actually generating the report or doing the task, you'd have to then refactor the reusable bits out to their own modules, which you would do by copy-pasting them from the remote server to your local editor, making sure they don't rely on or mutate any global state and finally making sure they are actually usable. This is just too much work to do after you have handed over your one-off report.

With Clojure, on the other hand, you are encouraged to write small functions – which are usually pure and side-effect free. Which in practice meant that we'd start with a thread macro ((->> something (map #(...)) (filter #(...)) etc) which of course after a few stages became unwieldy. Then we'd extract out the small "domain" predicated functions, like is-draft?, truncate-to-day, created-between that would be pure and actually reusable in a very obvious way. Or perhaps, we'd managed to figure out a tricky AWS API call – that we could then extract out for future use.

And of course, when you are actually developing in your editor, writing small pure functions, and wrapping all your explorations in a "rich comment", you can just git -am "fiddle with the REPL" so that your teammates can not only use these functions in the future, but also benefit from all your exploration.

Access to a rich library ecosystem

Now of course, most of the work we've done usually relies on slicing and dicing data using the Clojure core functions. But when we actually needed to go above and beyond to that, we were able to leverage some very robust Clojure and Java libraries: e.g. Hickory to convert HTML into Clojure data for further analysis, Apache Tika to detect file mime-types, Apache POI to generate Excel reports etc.

While all this is also available in many other ecosystems, including Node.js, the ability to quickly add a Java library to your project (or its Clojure wrapper) and start to experiment with its API with your actual data takes the whole experience just to the next level.

Clojure: Low friction instead of paper cuts

To sum up our experience so far, we've found that Clojure is the inverse of the "death by a thousand paper cuts" phenomenon. Instead of slowing down because of numerous little inconveniences, with Clojure we found that the very very low friction to get things done enables you to do things that you'd otherwise never even consider. We're extremely happy with our decision so far, so a big thank you to all the fantastic people in the Clojure community that make all this possible!

Note: This is based on a considerable amount of research and thought, but very little personal practical experience. I will update this blog post based on suggestions from the Clojure community.

Web applications concerns

Most web applications have to deal with a large set of common use-cases that are inherent to the client/server, stateless, HTTP request/response model. Some web frameworks try to cover a lot of these use cases and provide some value on top, whereas other are more minimal and depend a lot on middleware and other plug-ins or libraries to cover the gaps.

On top of that, the more established frameworks also try to tackle data modelling, persistence and more. These so-called "Monolithic" web frameworks usually try to address most of the following concerns:

• Architecture — there are pre-defined ways for you to write your code. Most, if not all architectural decisions are made by the framework creators, e.g. MVC.
• Basic HTTP interface — request parsing, response generation etc. Most of the time there is an agreed-upon interface like WSGI, Rack or Plug that these frameworks conform to.
• URL Routing — dispatching the correct action based on the URL structure.
• HTML templating — creating HTML documents with common sections (headers, footers, sidebars), links, dynamic content and so on.
• Persistence — reading and writing to some database, usually through some other abstraction, e.g. an ORM.
• Sessions and Users — handling cookies, per-session data, authentication/authorization
• Internationalization/Localization — showing the interface and data in multiple languages
• File handling — accepting form uploads and storing them on persistent storage
• Emails — sending emails via templates
• Forms — rendering and accepting HTML forms, incl. widgets, validation and eventually persistence of data.
• Security — ensuring that the web application isn't vulnerable to the most common security pitfalls, e.g. CSRF, XSS, SQL injection, timing attacks and so on.
• Websockets — or compatible alternatives like long-polling, server-sent events etc.
• Javascript and other Assets — packaging Javascript files, compiling SCSS, optimising images etc.
• Job queues — scheduling jobs to happen out-of-band
• Testing — writing and running tests at different levels of abstraction
• Extensibility — allowing the addition of middleware and plugins that enhance or replace built-in functionality
• Production — deployment, web servers, caching

On the other hand, the so-called "minimalist" frameworks will mostly deal with basic HTTP and perhaps one or two more concerns, while you pick and choose from various libraries to fill in the gaps.

The pros and cons of "Monolithic vs Minimal" depend on your use case, experience with the language and library landscape and your level of comfort when making architectural decisions — I won't go into a comparison here.

The Clojure situation

There is no such thing as a "Monolithic Clojure web framework". However, I'd argue that there quite a few frameworks that offer much more than other minimalistic frameworks in other languages. In addition, the JVM layer on which Clojure is based on is a rock-solid foundation for when the time comes to deploy, have logs, metrics etc.

Rather than going concern-by-concern, I'll present Clojure web development as a series of layers. The crucial different between Clojure and most other common web-dev languages is how easily one can combine and compose libraries to take care of some concerns (Phoenix/Elixir is also good at this).

NOTE: One major thing that Clojure also has is the toll-free interop with Java libraries — and also a lot of historical artefacts of Java Web development, such as servlets, web containers etc. However, most of the time you won't be dealing with those, and a lot of approaches don't rely on them at all.

Ring specification & Java servlets

This is not actually the top-most or bottom layer, but it's central to everything else so it needs to be introduced first.

Ring is a simple, server-agnostic specification that models HTTP requests and response to plain Clojure maps and functions. A Ring handler is a function that takes a Ring request map as an argument and returns a Ring response map. The request and response maps are plain Clojure maps with some well-defined keys like :body, :uri and :headers. There are synchronous and asynchronous flavours (the async version is trivial to implement once a sync version is written).

This abstraction, inspired by the WSGI and Rack specifications of Python/Ruby, supports writing adapters that are responsible for abstracting away the low-level networking and protocol aspects.

The rich I/O facilities of Java allow for some powerful features — for example, the response :body can be a string, a sequence, a file or a generic java.io.InputStream and the server will do the correct thing.

All the servers and frameworks listed below support Ring handlers (which are plain functions) and sometimes will add their own extra features on top. A Ring handler function is portable across all web frameworks and servers, though.

Java Servlets is a 1998-era specification for doing HTTP with Java, and there's a ton of Java web servers out there that support servlets. Ring has some tools that can convert a Ring handler into a generic Servlet, so that they can be embedded with any existing Java web-app.

Frameworks

Continuing this middle-out exploration, we go to frameworks. While you can very easily just use Ring to write a trivial web applications, there are some frameworks out there that are best suited for a more complex app. Some frameworks have an opinionated front-end story, whereas others focus solely on the back-end.

• Luminus — less coherent than a framework, and more a project template that uses a curated collection of libraries. Has a number of options that can bring in databases, i18n, templating, etc. Extensive documentation and a published book.
• Pedestal — originally a full-stack framework created by Cognitect, now focused on server-side APIs. Handles HTTP, Routing, Security, WebSockets, Deployment concerns. Production grade but documentation could be expanded.
• Yada — A framework by JUXT, that aims for 100% HTTP compliance, including advanced or niche areas of the spec. Production ready but the HTTP focus may be limiting. Can only be run on Aleph (see below).
• Duct — An ambitious "configuration-driven" framework, that brings in under its umbrella a number of compatible libraries for concerns such as routing, databases, migrations, front-end assets. The creator is well-respected within the Clojure community, however the project is still young and documentation is sparse.
• Fulcro — a front-end first web framework. Extremely opinionated on the front-end, and somewhat prescriptive on the back-end. A special case since it's aimed 100% at Single-Page Applications, it dictates a server architecture that can be implemented by any of the above frameworks as needed.

There are quite a lot more libraries that can be integrated with any Ring-compatible framework (that is, all of the above). A lot of standalone Ring middleware are already reused by Pedestal and Luminus. Yada is somehow a lone example, as it suggests that Ring middleware go against HTTP semantics and data-driven architectures.

Networking stuff

At the bottom layer, we have the actual TCP/IP stack and the first layer of HTTP protocol implementation, plus web socket support. All the options here have more or less feature parity, so there's no clear winner in my eyes.

• Jetty — Java HTTP/Websocket server, natively supported by Ring.
• HTTP-Kit — Clojure/Java web server.
• Immutant — Clojure HTTP library, based on Undertow. Standalone or embeddable in the WildFly container for cluster support.
• Aleph — Clojure networking library, based on Netty.

All the servers are standalone and can be bundled into a JAR file. This means that the only dependency you have when deploying is a Java Runtime Environment.

Jetty and HTTP-Kit are quite stable and conservative choices. Both support Websockets and streaming, while Jetty also supports HTTP/2 and HTTPS. HTTP-Kit also includes an HTTP client.

Aleph is an ambitious asynchronous networking library, being able to handle TCP & UDP streams, while also exposing an HTTP layer.

Immutant is also far more than a web server — it adds support for Messaging, Scheduling, Caching, Transactions, built on established Java libraries. It also supports clustering when deployed in the WildFly container. You can start out by just using the standalone web server though and not even include features you don't use.

Deciding?

There is definitely a lot of choice, even for a seemingly simple concern such as dealing with HTTP — and then a ton of other choices that relate to persistence, front-end etc. I won't give any opinions here, but I'll try to articulate my thought process since I'm currently evaluating a lot of Clojure libraries for building a web application.

Start with the data

Perhaps the most important decision that will be the hardest to reverse is the data modelling and persistence layer. Perhaps you already have your data in some database, or perhaps you're only comfortable with one particular technology (e.g. SQL or NoSQL), so this narrows down your choices. Fortunately, the decision matrix for persistence in Clojure is much smaller — there are a couple of SQL libraries, and each major NoSQL library has one well-maintained client.

I'd first start by writing some data structures (start with plain maps, perhaps use records if really needed) that show how the data would like "in-flight", that is, when passed around inside a live system. You can use Clojure Spec for this, or take a suggestion from the libraries used in Luminus.

Saving the data

Next you have to decide how the data are going to look like when "at-rest" — I'd create an internal API (just a set of functions, really) that deals with persisting the in-flight data model to the database. Start with simple CRUD operations, then add other actions that might not model well to CRUD. It's a good idea to try and abstract this behind a protocol of sorts, so that it can be mocked out for tests or even entirely replaced.

Showing the data

The next biggest decision is actually how you'd get the in-flight data shown to your users — this usually means either rendering HTML (old-school style) or passing the data to the browser in some format like JSON. You might want to consider GraphQL as there's a really good library for that, or you may want to use a simpler REST-like interface. The possibilities here are quite large and it really depends on what you are aiming to do and what front-end technologies you'll use.

Tying everything together

I'd argue that the above steps are two thirds of the work needed to get a web application off the ground. When the above decisions are made and implemented, picking an HTTP framework should be an easy decision, and things like authentication and authorisation should be quite straightforward to add.

I hope this short overview has helped demystify the landscape of Clojure back-end web development. Please let me know of any errors or omissions on Reddit, Hacker News, Clojureverse or just drop me an email.

Installation

If you're on Mac and use homebrew, just do brew install clojure. Otherwise check out the official instructions.

Check that your installation works fine by typing clj — you should see something like:

Clojure 1.9.0  
user=>  

Hit Ctrl-C to exit this.

Note: you will notice that it takes a few second to get to this prompt. You will suffer through this during this introduction, but when doing proper Clojure development, the startup time becomes a non-issue for most workflows.

Hello world!

Create a file named hello.clj, with these contents:

(println "Hello, world!")

You can run it with clojure hello.clj, and you should see the results printed on your terminal.

So what's going on? println is a function. "Hello, world!" is a String. You invoke a function by enclosing it and its arguments with parentheses. In other languages, you'd do println("Hello world!") which is a bit similar.

Clojure being a dynamic language, you can pass any kind of argument to println — try adding those to hello.clj and invoke it again with clojure hello.clj.

(println 123) ;; this is a comment

(println true) ;; true/false are booleans

(println nil) ;; nil is null

(println [1 2 3]) ;; vector definition, commas are optional

You should see this:

Hello, world!  
123  
true  
nil  
[1 2 3]

So a .clj file can contain one or more top-level expressions. They will be evaluated one by one, top-to-bottom, as in most programming languages. Clojure also has numbers, booleans, nil and a few pretty cool container data structures.

The REPL

Clojure also an interactive prompt; you saw it when you run clojure without a command. You can try all this code in the REPL, like so:

$ clj
Clojure 1.9.0  
user=> (println "hi again")  
hi again  
nil  
user=>  

This time, you see the printed line, but you also see nil — this is because the REPL will print out the return value of each expression as you type it in. In this case, the println function returns nil.

There is an alternative form of println called println-str — instead of printing directly, it will instead return a string. Let's try it out:

user=> (println-str "another try")  
"another try\n"
user=>  

This time, nothing is printed, and we just get a string back. You can see the newline at the end.

Note: moving on, I'll showcase REPL-based code, but you can run all those just by adding them to your hello.clj file, and running clojure hello.clj. Note how we use clojure to run programs but clj to get the REPL.

Variables

Let's try to capture the value returned by println-str and print it later:

user=> (def a "hello")  
#'user/a
user=> a  
"hello"
user=> (println a)  
hello  
nil  
user=>  

Notice how to define a variable, we just call a function — this time the function is def and takes two arguments: the variable name which is a Symbol, and the variable value, which can be anything. Notice how def also returns this weird-looking #'user/a thingy — this is the internal representation of "global" variables in Clojure. However, when you evaluate the Symbol, that is, a, you get the value back, which is "hello". You can use the name of the variable as you would expect when calling functions etc.

Of course, the value of the variable can be pretty any expression — in this case, the expression will be evaluated, and its return value will be used:

user=> (def my-var (println-str "variables are cool"))  
#'user/my-var
user=> my-var  
"variables are cool\n"
user=> (println my-var)  
variables are cool

nil  
user=>  

Notice how we now get two newlines when using println — the one that my-var contains, and another one that println adds.

Also notice how my-var can contain a dash. Clojure is very permissive on what can be used in identifiers (Symbols, technically). You might have guessed that already, since println-str contains a dash.

Note: println-str is a very little used function, but makes sense for this tutorial. We'll cover more useful functions soon!

Functions

Let's define our own function — it will just call the only two functions we have used so far:

user=> (defn say-it*3 [x]  
          (println x)
          (println x)
          (println x))
#'user/say-it*3
user=> (say-it*3 "hi!")  
hi!  
hi!  
hi!  
nil  
user=>  

Let's break this apart — to define a function, just call another function, in this case defn.

Now, defn is technically a macro, but it looks and behaves just like a function for our purposes. Its arguments are:

• a Symbol — the name of the function, say-it*3 (wow! you can have a * in your function name!)
• a vector of arguments, that is, [x], meaning "this functions takes a single argument named x
• a bunch of expressions, that when the function is called, will be evaluated top-to-bottom. The return value of the last expression will be returned (there is no return or similar keyword).

As you can see, defn returns the same kind of var as def returns.

You call this function as if it was any other function, by surrounding it and its arguments with parentheses. Of course, functions can have more than one arguments:

user=> (defn another! [a b c]  
  (println a)
  (println b)
  (println c))
#'user/another!
user=> (another! 1 2 3)  
1  
2  
3  
nil  
user=>  

...or even no arguments:

user=> (defn -*- []  
  (println "konnichi-wa!"))
#'user/-*-
user=> (-*-)  
konnichi-wa!  
nil  
user=>  

And yes, you can have a lot of weirdly named functions — please don't do it :)

The fact that you can use a lot of characters for symbols comes directly from Clojure's minimal syntax — there are very few special operators and rules, so even arithmetic stuff is done by just calling functions:

user=> (+ 1 2)  
3  
user=> (* 3 4)  
12  
user=> (+ 1 2 3 4 5)  
15  
user=> (= 1 2)  
false  
user=> (= 2 (+ 1 1))  
true  
user=> (= 2 2 2)  
true  

In this case +, * and = are just plain functions, and you call them with as many arguments as you want.

You can also have anonymous functions, like so:

user=> (fn [x] (println x))  
#object[user$eval183$fn__184 0x4bff64c2 "user$eval183$fn__184@4bff64c2"]
user=>  

Note: Ignore the weird return value — this is a Clojure-internal representation.

See how fn is really similar to defn — you just don't give it a name. Let's combine fn and def to actually capture the function in a variable:

user=> (def anonymous (fn [x] (println x)))  
#'user/anonymous
user=> anonymous  
#object[user$anonymous 0x4566d049 "user$anonymous@4566d049"]
user=> (anonymous "hi")  
hi  
nil  
user=>  

Conditionals

The fundamental way to do conditionals is by calling the if function:

user=> (if (= 1 2) "wat?" "ok!")  
"ok!"
user=>  

We call if we 3 arguments:

• The condition, in our case, the result of (= 1 2), i.e. false
• The "true" expression, i.e. "wat?"
• The "false" expression, i.e. "ok!"

The return value is either the "true" or "false" expression. Here's a more complicated example:

user=> (if (= 1 "1") (println "wat?") (println "ok!"))  
ok!  
nil  
user=>  

Notice how although only one expression is even evaluated — this is a special property of if that is provided by the Clojure runtime. If if was a normal function, we'd see both wat? and ok! printed.

Notice also that the return value is nil, since this is what println returns.

Another useful special function is do:

user=> (do  
         (println 1)
         (println 2)
         (println 3)
         "done!")
1  
2  
3  
"done!"
user=>  

This looks very similar to defn and fn — it evaluates expressions one-by-one, and returns the value of the last one (in our case, just the string "done!"). By combining if and do you can get more useful constructs:

user=> (if (= 1 1.0)  
(do
 (println "Equality is weird")
 "boo!")
(do
  (println "Equality is sane")
  "yay!"))
Equality is sane  
"yay!"
user=>  

By the way, if you are dealing only with numbers, you can use the == function instead of =:

user=> (== 1 1.0)  
true  

Note: There are more conditionals available, but we'll stick with if for a while

Scoping

Many programming languages have intricate rules about scoping — that is, from where you can access and change a variable. Clojure, on the other hand, is simple:

• Variables defined with def are always global
• Variables defined with let are always local

Note: this is a simplification, but it can get you very far.

In reality, you always use def sparingly, and always at the "root" level of a file. You use let everywhere else. How does let look like? It's just, as you may have guessed, a function:

user=> (let [local 1]  
          (println "This is my scope")
          (println "The value is" local))
This is my scope  
The value is 1  
nil  
user=>  

Let looks a bit similar to defn and fn in that its first argument is a vector, that is [local 1]. The rest of the arguments are just expressions that are evaluated one-by-one, like in do. The return value is, again, the value of the last expression, in our case nil.

So how does this work? In other languages, you'd do local = 1, then in the same "scope" you'd be able to use local to refer to the value 1. This is similar, the scope being the body of let. After the closing ) of let, local is not valid anymore.

A more involved example:

user=> (let [a 1  
             b 2
             c 3]
            (* a b c))
6  
user=>  

Here we assign multiple variables, a, b, and c at the same time. We just multiply them together, and we get the result back. Another one:

user=> (let [a 2  
             b 3
             aa (* a a)
             bb (* b b)
             aabb (+ aa bb)]
             (= aabb 13))
true  
user=>  

In this case, not only we assign multiple variables, we also use previously-assigned variables. The variable assignment is done top-to-bottom, so we just have to make sure everything is in the right order.

Functions everywhere

You might have noticed that to do very common stuff like defining a variable, a function, comparing or even adding two numbers together, we just call a function. This is a common attribute for any Lisp-like language, and there's no other syntax to them. In real-life usage though, there are a handful of functions that are so commonly used that they effectively become the syntax.

The everything-is-a-function is a bit of a simplification. Some things are macros, other things are special forms. But when you use them, usually you don't think about it, and you call them as if they were functions.

To be continued! Rather than let this sit in my drafts, I'm publishing it now in order to get some feedback. I will either add to this or post something new later.

Please post your feedback on this Reddit thread

I'd like to show how I wrote my first functional-style program (with immutable data structures and recursion) when trying to solve the first puzzle of AoC 2015. Spoilers ahead!

The language used doesn't really matter — I first did this in Elixir, and then again in Clojure, so I will stay in the domain of dynamically-typed functional languages. That is, I'll model my data as a combination of basic data structures and won't create type definitions for everything.

The puzzle

AoC puzzles are given in two parts; you solve the first, then the requirements change slightly. The first part of our puzzle is:

Santa is trying to deliver presents in a large apartment building, but he can't find the right floor - the directions he got are a little confusing. He starts on the ground floor (floor 0) and then follows the instructions one character at a time.

An opening parenthesis, (, means he should go up one floor, and a closing parenthesis, ), means he should go down one floor.

The apartment building is very tall, and the basement is very deep; he will never find the top or bottom floors.

For example:

• (()) and ()() both result in floor 0.
• ((( and (()(()( both result in floor 3.
• ))((((( also results in floor 3.
• ()) and ))( both result in floor -1 (the first basement level).
• ))) and )())()) both result in floor -3.

To what floor do the instructions take Santa?

Let's start with some plain, imperative Javascript:

function santa(instructions) {  
  let floor = 0;
  for (let i of instructions) {
    if (i == '(') {
      floor += 1;
    } else {
      floor -= 1;
    }
  }
  return floor;
}

console.log(santa("(())")) // 0  

(Let's assume that we won't get any weird characters in there.)

Keeping the same mindset, let's jump into Elixir. Try as you might, you won't find any loop construct in Elixir — no while, no goto, and the for construct is not what you might think.

The only tool we have to implement a loop is recursion, and we'll use that:

defmodule Santa do  
  defp santa([], floor) do
    floor
  end

  defp santa([h | t], floor) do
    if h == "(" do
      santa(t, floor + 1)
    else
      santa(t, floor - 1)
    end
  end

  def santa(instructions) do
    l = String.codepoints(instructions)
    santa(l, 0)
  end
end

IO.inspect Santa.santa("(())") # 0  

Notice how we have 2 + 1 versions of the function. The first two are private and take two arguments, while the third one is public, takes one argument, and only converts the string into a list. This is all perfectly fine in Elixir.

Then, we use Elixir's pattern matching in function definitions to effectively say:

• If the list is empty, we are done, here's the floor. This is the "stop clause" of our recursive function.
• If the list has elements, get the first one, check what kind is it, then continue with the rest of the elements and the updated floor. This is the "work clause".

Note how floor is initialized in the 1-arity function, then updated and eventually returned by the other two functions. In functional programming, this kind of pattern is usually called "accumulator passing", and is used to ensure that our recursive calls can be optimized by the compiler so we don't suffer a stack overflow (good luck googling for that).

A slightly more idiomatic Elixir code would look like this:

defmodule Santa do  
  defp santa([], floor), do: floor

  defp santa(["(" | t], floor) do
    santa(t, floor + 1)
  end

  defp santa([")" | t], floor) do
    santa(t, floor - 1)
  end

  def santa(instructions) do
    l = String.codepoints(instructions)
    santa(l, 0)
  end
end  

This more specific pattern matching — we are not only getting the first element of the list, but we also check what it is and do the right thing without a conditional. Plus, if we pass something that's not a ( or a ), our process will promptly crash.

The code above represents pretty much the first Elixir code I've ever written. Today I would probably write something like this:

defmodule Santa do  
  def santa(instructions) do
    instructions
    |> String.codepoints
    |> Enum.map(fn("(") -> 1
                  (")") -> -1 end)
    |> Enum.reduce(fn(x, acc) -> x + acc end)
  end
end  

The |> is called the "pipe operator" and allows you to "pipe" the return of a function as the first argument to the next function etc.

We are using the map and reduce functions to go at a higher abstraction level where we reason about the whole problem in one go.

This approach allows you to start thinking about the code in terms of plain language:

• Take the instructions...
• convert them to individual characters...
• replace "(" with 1 and ")" with -1...
• add everything together.

Note that we never even mention the "floor" by any name — the function should probably be named floor. If the computation was more complex, I'd break it into smaller pieces just to name a few variables for clarity.

For fun, here is the same approaches in Clojure:

Recursive version:

(defn santa
  ([instructions]
   (santa instructions 0))
  ([instructions n]
   (if (nil? (seq instructions)) n 
      (if (= (first instructions) \( )
        (recur (rest instructions) (+ n 1))
        (recur (rest instructions) (- n 1))))))

(println (santa "(())"))
;; UPDATE: Added nil? to (seq instructions), thanks /u/nogridbag 

That is dense! (and not idiomatic at all, BTW). First, we are using a multi-arity function definition, same as in Elixir. Then we are checking if the instructions are empty; Clojure's Sequence abstraction allows us to do that without caring what the actual data type is (could be a list, a vector, a Java array...). Then we just get the first character, and recurse with the rest of the instructions. (Instead of (+ n 1) I would have used (inc n), but I left it as-is for clarity.)

Notice since the JVM doesn't support Tail Call Optimization, we have to use the special keyword recur, compared to Elixir, where we just call the function by name. The compiler puts some restrictions on where you can put the recur to ensure you are always recursing from the tail position.

Here is the more idiomatic map/reduce version:

(defn santa [instructions]
  (let [conv {\( 1, \) -1}]
    (->> instructions
         (map conv)
         (reduce +))))

Since Clojure doesn't have a build-in pattern matching, we create a lookup-map named conv from \( and \) (Java Character instances) to the elevator directions.

Clojure has literal support for maps, and they can contain any key or value, e.g. {1 "a"} or {"b" true}. No colons, and commas are optional. You can get the value out of a map by either using get or by just pretending the map is a function and calling it — (get {"a" 1} "a") and ({"a" 1} "a") are equivalent.

Note that Clojure has very explicit scoping rules — we have to use let to define that map, and it's only accessible within its body.

The ->> is equivalent to Elixir's |> only it pipes the values as the last argument, which is what map/reduce expect. Contrary to Elixir, Clojure supports map over a String — it converts into a sequence of Java Characters. Finally, + is just a plain function so we can use it as-is without having to define a wrapper function as we did in Elixir.

This is mostly equivalent with the Elixir version, with the notable difference that if you pass in something unexpected, you will silently get a nil in your sequence and eventually a NullPointerException when you try to add a nil and a number together. This can be cryptic if you are new to Clojure.

How about some map-reduce Javascript?

function santa(instructions) {  
    let conv = {'(': 1, ')': -1}; 
    return [...instructions]
             .map((i) => conv[i])
             .reduce((acc, v) => acc + v);
}

Not half bad! We do have to convert the string to an Array since map is not a first-class function, which is annoying. At least the ... spread operator makes it a bit straightforward.

Note how it suffers the same problem with Clojure — this time, we'll get undefined in our mapped Array, and our returned value will be NaN. Which is arguably worse than just blowing up.

I won't show a recursive Javascript version; Javascript doesn't have support for tail-recursive calls, meaning that while it would work for a few characters, when we'd try this with the actual puzzle input it might result in a stack overflow error.

This is enough for Part 1 — we've solved the problem in 3 different styles, two of them functional. In Part 2 we'll see how each approach adapts to the different requirements.

Clojure Bonus Round — an imperative version, with mutable state:

(defn santa [instructions]
  (let [floor (atom 0)]
    (doseq [i instructions]
      (if (= i \( )
        (swap! floor inc)
        (swap! floor dec)))
    @floor))

Here we are using an atom, which is one of Clojure's primitives for state. We iterate over the instructions, and either increase or decrease the "mutable" floor value. When we're done, we're "dereferencing" the value contained in the atom and return it.

Note that this atom is by default thread-safe; if we wanted, we could spin up multiple threads to work on this collection in parallel, and at their core they would just call (swap! floor inc) without needing to lock the value.

This is a very unsuitable way to solve this problem, but it is worth pointing out that if your algorithm is better expressed with local mutable state, Clojure allows you to do that, whereas Elixir will make it really hard.

In any case, since the customer complained I had to troubleshoot the root cause of the issue. I went into the systemd logs only to realise that all logging of this Elixir umbrella application had stopped a couple of days ago.

Diving back into Wobserver, I saw that the main Logger process had a couple of million messages in its queue, and is effectively dead and not responding (or, at least, not doing any new IO). It seems that the BEAM VM will not enforce a mailbox limit, but adding messages there will be slower.

As per 1.5.3, The Elixir Logger is essentially a GenEvent-based architecture, where each log message is sent to another process for later processing. When the load is low, the Logger will use the asynchronous notify function for faster performance; however, when the message inbox starts to fill up, the Logger will switch to a synchronous mode. This way, each call to Logger.log will block until "[..] after the event has been handled by all event handlers.". This is supposed to add backpressure to give the logger back-end time to flush before new messages are coming in.

I will need to study the Logger code in more detail, but it seems to me that this design is flawed; when using any back-end, there is an inherent limit on how fast you can write to it. For the default console back-end, the limit is related to whoever handles the stdout of the process — in my case, it was systemd.

In this particular failure mode, the systemd could process only X msg/sec. My code produced Y msg/sec, where Y > X. This resulted in Y – X messages accumulating in the Logger queue every second, with no hope ever to be flushed out. What's worse, the Logger would switch to sync mode, slowing down the entire application for ever. This kind of backpressure might be useful in other situations, but for logging I would argue that just dropping the messages (and logging the fact!) would be much better.

In my case, I fixed this by setting the log level to :info (it is :debug by default, another questionable choice) and doing a complete review of all my logging calls. However, I have to check again in a couple of days and see how the system holds up.

Incidentally, I now understand why Clojure's core.async channels by design do not support this unbounded behaviour. You have to specify an explicit window after which the channel becomes blocking, or a custom behaviour that will drop messages if the channel is full.

The Clojure documentation can be hard to decipher, and the clojure.core namespace is huge. ClojureDocs helps, but I believe there's a need for an apropos-like functionality built into the system.

In case you're not familiar with /usr/bin/apropos, here's what it looks like:

~> apropos audio
SoX(1)                   - Sound eXchange, the Swiss Army knife of audio manipulation  
faad(1)                  - Process an Advanced Audio Codec stream  
flac(1)                  - Free Lossless Audio Codec  
gst-visualise-0.10(1), gst-visualise(1) - Run a GStreamer pipeline to display an audio visualisation  
lame(1)                  - create mp3 audio files  
libsox(3)                - SoX, an audio file-format and effect library  
libswresample(3)         - audio resampling library  
sndfile-cmp(1)           - compares two audio files  
sndfile-concat(1)        - concatenates two or more audio files  
soxeffect(7), SoX(7)     - Sound eXchange, the Swiss Army knife of audio manipulation  
soxformat(7), SoX(7)     - Sound eXchange, the Swiss Army knife of audio manipulation  
twolame(1)               - an optimised MPEG Audio Layer 2 (MP2) encoder  
bluetoothaudiod(8)       - The Mac OS X Bluetooth audio daemon  
mount_cddafs(8)          - mount an Audio CD  

This generated a couple of responses via tweets and private emails pointing out that Clojure does indeed have an apropos function:

(apropos str-or-pattern)

Given a regular expression or stringable thing, return a seq of all public definitions in all currently-loaded namespaces that match the str-or-pattern.

However, even in the ClojureDocs apropos example you can see how it's not really what I had in mind:

user=> (apropos "temp")  
()

user=> (require 'clojure.template)  
nil

user=> (apropos "temp")  
(apply-template do-template)

... since it only searches the currently-loaded namespace. If you don't know what you're looking for, you probably won't have loaded the containing namespace in the first place. Of course, most of the time you're looking for something in clojure.core which is loaded by default, so let's move on.

Phrasing

Let's try another example. Let's say I want to add something to a collection. The keywords I'm thinking of are "add", "append", "prepend", "push", "insert". Here's what the output of apropos for all those are:

user=> (apropos "add")  
(clojure.core/add-classpath clojure.core/add-watch clojure.core/unchecked-add clojure.core/unchecked-add-int clojure.java.javadoc/add-local-javadoc clojure.java.javadoc/add-remote-javadoc clojure.tools.nrepl.middleware.session/add-stdin)

user=> (apropos "append")  
(clojure.core/chunk-append)

user=> (apropos "prepend")  
()

user=> (apropos "push")  
(clojure.core/push-thread-bindings clojure.tools.reader.reader-types/indexing-push-back-reader clojure.tools.reader.reader-types/input-stream-push-back-reader clojure.tools.reader.reader-types/push-back-reader clojure.tools.reader.reader-types/source-logging-push-back-reader clojure.tools.reader.reader-types/string-push-back-reader)

user=> (apropos "insert")  
()

Of course, the function I'm looking for is conj. Here's the built-in documentation for it:

user=> (doc conj)  
-------------------------
clojure.core/conj  
([coll x] [coll x & xs])
  conj[oin]. Returns a new collection with the xs
    'added'. (conj nil item) returns (item).  The 'addition' may
    happen at different 'places' depending on the concrete type.

You can see that "add" is mentioned twice, so you'd expect this would show up when doing (apropos "add") — but the fact is that clojure.repl/apropos searches only function names. This, combined with the very short names of most functions, results in very limited functionality.

Update: @mfikes on Twitter pointed out find-doc. It produces a ton of output though and it's not easily skimmable (I had to constrain the height of this code block to avoid messing the whole post up):

  
user=> (find-doc "add")  
-------------------------
clojure.tools.nrepl.middleware.session/add-stdin  
([h])
  stdin middleware.  Returns a handler that supports a "stdin" :op-eration, which
   adds content provided in a :stdin slot to the session's *in* Reader.  Delegates to
   the given handler for other operations.

   Requires the session middleware.
-------------------------
clojure.tools.nrepl.middleware.session/session  
([h])
  Session middleware.  Returns a handler which supports these :op-erations:

   * "ls-sessions", which results in a response message
     containing a list of the IDs of the currently-retained sessions in a
     :session slot.
   * "close", which drops the session indicated by the
     ID in the :session slot.  The response message's :status will include
     :session-closed.
   * "clone", which will cause a new session to be retained.  The ID of this
     new session will be returned in a response message in a :new-session
     slot.  The new session's state (dynamic scope, etc) will be a copy of
     the state of the session identified in the :session slot of the request.

   Messages indicating other operations are delegated to the given handler,
   with the session identified by the :session ID added to the message. If
   no :session ID is found, a new session is created (which will only
   persist for the duration of the handling of the given message).

   Requires the interruptible-eval middleware (specifically, its binding of
   *msg* to the currently-evaluated message so that session-specific *out*
   and *err* content can be associated with the originating message).
-------------------------
clojure.test/add-ns-meta  
([key coll])
  Adds elements in coll to the current namespace metadata as the
  value of key.
-------------------------
clojure.tools.nrepl.server/start-server  
([& {:keys [port bind transport-fn handler ack-port greeting-fn], :or {port 0}}])
  Starts a socket-based nREPL server.  Configuration options include:

   * :port — defaults to 0, which autoselects an open port on localhost
   * :bind — bind address, by default "localhost")
   * :handler — the nREPL message handler to use for each incoming connection;
       defaults to the result of `(default-handler)`
   * :transport-fn — a function that, given a java.net.Socket corresponding
       to an incoming connection, will return an value satisfying the
       clojure.tools.nrepl.Transport protocol for that Socket.
   * :ack-port — if specified, the port of an already-running server
       that will be connected to to inform of the new server's port.
       Useful only by Clojure tooling implementations.

   Returns a (map) handle to the server that is started, which may be stopped
   either via `stop-server`, (.close server), or automatically via `with-open`.
   The port that the server is open on is available in the :port slot of the
   server map (useful if the :port option is 0 or was left unspecified.
-------------------------
clojure.core.server/start-server  
([opts])
  Start a socket server given the specified opts:
    :address Host or address, string, defaults to loopback address
    :port Port, integer, required
    :name Name, required
    :accept Namespaced symbol of the accept function to invoke, required
    :args Vector of args to pass to accept function
    :bind-err Bind *err* to socket out stream?, defaults to true
    :server-daemon Is server thread a daemon?, defaults to true
    :client-daemon Are client threads daemons?, defaults to true
   Returns server socket.
-------------------------
cljs.stacktrace/parse-stacktrace  
  Parse a JavaScript stacktrace string into a canonical data form. The
  arguments:

  repl-env - the repl environment, an optional map with :host and :port keys
             if the stacktrace includes url, not file references
  st       - the original stacktrace string to parse
  err      - an error map. :ua-product key defines the type of stacktrace parser
             to use, for example :chrome
  opts     - additional options. :output-dir maybe given in this argument if
             :host and :port do not apply, for example, a file path

  The canonical stacktrace representation can easily be mapped to a
  ClojureScript one see mapped-stacktrace and mapped-stacktrace-str
-------------------------
cljs.module-graph/add-cljs-base  
([modules])
  Adds :cljs-base module to compiler :modules if not already present.
-------------------------
cljs.module-graph/add-cljs-base-dep  
([modules])
  Adds :cljs-base to any module in compiler :modules with an empty :depends-on.
-------------------------
cljs.module-graph/inputs->assigned-modules  
([inputs modules])
  Given compiler inputs assign each to a single module. This is done by first
  starting with :entries. Dependencies for every entry in a module are also added
  to that module. Inputs may of course be assigned to several modules initially
  but we must eventually choose one. User supplied module :entries are respected
  but all other input assignments are computed automatically via
  deepest-common-parent. This function returns a map assigning all inputs (indexed
  by munged name) to a single module. Any orphan inputs will be assigned to
  :cljs-base.
-------------------------
clojure.pprint/*print-miser-width*  
  The column at which to enter miser style. Depending on the dispatch table,
miser style add newlines in more places to try to keep lines short allowing for further  
levels of nesting.  
-------------------------
clojure.pprint/add-english-scales  
([parts offset])
  Take a sequence of parts, add scale numbers (e.g., million) and combine into a string
offset is a factor of 10^3 to multiply by  
-------------------------
cljs.analyzer/add-consts  
([compiler-state constants-map])
  Given a compiler state and a map from fully qualified symbols to constant
  EDN values, update the compiler state marking these vars as const to support
  direct substitution of these vars in source.
-------------------------
clojure.set/join  
([xrel yrel] [xrel yrel km])
  When passed 2 rels, returns the rel corresponding to the natural
  join. When passed an additional keymap, joins on the corresponding
  keys.
-------------------------
cljs.repl/repl  
([repl-env & opts])
  Generic, reusable, read-eval-print loop. By default, reads from *in* using
  a c.t.r.reader-types/source-logging-push-back-reader,
  writes to *out*, and prints exception summaries to *err*. If you use the
  default :read hook, *in* must either be an instance of
  c.t.r.reader-types/PushbackReader or duplicate its behavior of both supporting
  unread and collapsing CR, LF, and CRLF into a single \newline. Options
  are sequential keyword-value pairs. The first argument is the JavaScript
  evaluation environment, the second argument is an extended version of the
  standard ClojureScript compiler options. In addition to ClojureScript compiler
  build options it also take a set of options similar to clojure.main/repl with
  adjustments for ClojureScript evalution and compilation model:

  Available clojure.main/repl style options and their defaults:

     - :init, function of no arguments, initialization hook called with
       bindings for set!-able vars in place.
       default: #()

     - :need-prompt, function of no arguments, called before each
       read-eval-print except the first, the user will be prompted if it
       returns true.
       default: #(if (c.t.r.readers-types/indexing-reader? *in*)
                   (== (c.t.r.reader-types/get-column-number *in*) 1)
                   (identity true))

     - :prompt, function of no arguments, prompts for more input.
       default: repl-prompt

     - :flush, function of no arguments, flushes output
       default: flush

     - :read, function of two arguments, reads from *in*:
         - returns its first argument to request a fresh prompt
           - depending on need-prompt, this may cause the repl to prompt
             before reading again
         - returns its second argument to request an exit from the repl
         - else returns the next object read from the input stream
       default: repl-read

     - :eval, function of one argument, returns the evaluation of its
       argument. The eval function must take repl-env, the JavaScript evaluation
       environment, env, the ClojureScript analysis environment, the form
       and opts, the standard ClojureScript REPL/compiler options.
       default: eval

     - :print, function of one argument, prints its argument to the output
       default: println

     - :caught, function of three arguments, a throwable, called when
       read, eval, or print throws an exception or error default. The second
       argument is the JavaScript evaluation environment this permits context
       sensitive handling if necessary. The third argument is opts, the standard
       ClojureScript REPL/compiler options. In the case of errors or exception
       in the JavaScript target, these will be thrown as
       clojure.lang.IExceptionInfo instances.
       default: repl-caught

     - :reader, the c.t.r reader to use.
       default: c.t.r.reader-types/source-logging-push-back-reader

     - :print-no-newline, print without a newline.
       default: print

     - :source-map-inline, whether inline source maps should be enabled. Most
       useful in browser context. Implies using a fresh reader for each form.
       default: true
-------------------------
clojure.java.javadoc/add-local-javadoc  
([path])
  Adds to the list of local Javadoc paths.
-------------------------
clojure.java.javadoc/add-remote-javadoc  
([package-prefix url])
  Adds to the list of remote Javadoc URLs.  package-prefix is the
  beginning of the package name that has docs at this URL.
-------------------------
clojure.tools.nrepl/client-session  
([client & {:keys [session clone]}])
  Returns a function of one argument.  Accepts a message that is sent via the
   client provided with a fixed :session id added to it.  Returns the
   head of the client's response seq, filtered to include only
   messages related to the :session id that will terminate when the session is
   closed.
-------------------------
clojure.tools.nrepl/message  
([client {:keys [id], :as msg, :or {id (uuid)}}])
  Sends a message via [client] with a fixed message :id added to it.
   Returns the head of the client's response seq, filtered to include only
   messages related to the message :id that will terminate upon receipt of a
   "done" :status.
-------------------------
clojure.tools.nrepl/url-connect  
  Connects to an nREPL endpoint identified by the given URL/URI.  Valid
   examples include:

      nrepl://192.168.0.12:7889
      telnet://localhost:5000
      http://your-app-name.heroku.com/repl

   This is a multimethod that dispatches on the scheme of the URI provided
   (which can be a string or java.net.URI).  By default, implementations for
   nrepl (corresponding to using the default bencode transport) and
   telnet (using the clojure.tools.nrepl.transport/tty transport) are
   registered.  Alternative implementations may add support for other schemes,
   such as HTTP, HTTPS, JMX, existing message queues, etc.
-------------------------
cljs.closure/add-core-macros-if-cljs-js  
([compiled])
  If a compiled entity is the cljs.js namespace, explicitly
  add the cljs.core macros namespace dependency to it.
-------------------------
cljs.closure/add-dep-string  
([opts input])
  Return a goog.addDependency string for an input.
-------------------------
cljs.closure/add-dependencies  
([opts & inputs])
  Given one or more IJavaScript objects in dependency order, produce
  a new sequence of IJavaScript objects which includes the input list
  plus all dependencies in dependency order.
-------------------------
cljs.closure/add-dependency-sources  
([inputs] [inputs compile-opts])
  Given list of IJavaScript objects, produce a new sequence of IJavaScript objects
  of all dependencies of inputs.
-------------------------
cljs.closure/add-js-sources  
([inputs opts])
  Given list of IJavaScript objects, add foreign-deps, constants-table
   IJavaScript objects to the list.
-------------------------
cljs.closure/add-preloads  
([inputs opts])
  Add :preloads to a given set of inputs (IJavaScript). Returns a new
  list of inputs where the preloaded namespaces and their deps come immediately after
  cljs.core or the constants table depending on the optimization setting. Any
  files needing copying or compilation will be compiled and/or copied to the
  appropiate location.
-------------------------
cljs.closure/watch  
([source opts] [source opts compiler-env] [source opts compiler-env quit])
  Given a source directory, produce runnable JavaScript. Watch the source
   directory for changes rebuilding when necessary. Takes the same arguments as
   cljs.closure/build in addition to some watch-specific options:
    - :watch-fn, a function of no arguments to run after a successful build.
    - :watch-error-fn, a function receiving the exception of a failed build.
-------------------------
clojure.tools.nrepl.middleware/topologically-sort  
([comparator stack])
  Topologically sorts the given middlewares according to the comparator,
  with the added huristic that any middlewares that have no dependencies
  will be sorted toward the end.
-------------------------
clojure.tools.reader.impl.errors/eof-error  
([rdr & msgs])
  Throws an ExceptionInfo with the given message.
   If rdr is an IndexingReader, additional information about column and line number is provided
-------------------------
clojure.tools.reader.impl.errors/illegal-arg-error  
([rdr & msgs])
  Throws an ExceptionInfo with the given message.
   If rdr is an IndexingReader, additional information about column and line number is provided
-------------------------
clojure.tools.reader.impl.errors/reader-error  
([rdr & msgs])
  Throws an ExceptionInfo with the given message.
   If rdr is an IndexingReader, additional information about column and line number is provided
-------------------------
clojure.core/add-classpath  
([url])
  DEPRECATED

  Adds the url (String or URL object) to the classpath per
  URLClassLoader.addURL
-------------------------
clojure.core/add-watch  
([reference key fn])
  Adds a watch function to an agent/atom/var/ref reference. The watch
  fn must be a fn of 4 args: a key, the reference, its old-state, its
  new-state. Whenever the reference's state might have been changed,
  any registered watches will have their functions called. The watch fn
  will be called synchronously, on the agent's thread if an agent,
  before any pending sends if agent or ref. Note that an atom's or
  ref's state may have changed again prior to the fn call, so use
  old/new-state rather than derefing the reference. Note also that watch
  fns may be called from multiple threads simultaneously. Var watchers
  are triggered only by root binding changes, not thread-local
  set!s. Keys must be unique per reference, and can be used to remove
  the watch with remove-watch, but are otherwise considered opaque by
  the watch mechanism.
-------------------------
clojure.core/assoc!  
([coll key val] [coll key val & kvs])
  When applied to a transient map, adds mapping of key(s) to
  val(s). When applied to a transient vector, sets the val at index.
  Note - index must be <= 2="" (count="" vector).="" returns="" coll.="" -------------------------="" clojure.core="" completing="" ([f]="" [f="" cf])="" takes="" a="" reducing="" function="" f="" of="" args="" and="" fn="" suitable="" for="" transduce="" by="" adding="" an="" arity-1="" signature="" that="" calls="" cf="" (default="" -="" identity)="" on="" the="" result="" argument.="" conj="" ([coll="" x]="" [coll="" x="" &="" xs])="" conj[oin].="" new="" collection="" with="" xs="" 'added'.="" (conj="" nil="" item)="" (item).="" 'addition'="" may="" happen="" at="" different="" 'places'="" depending="" concrete="" type.="" conj!="" ([]="" [coll]="" x])="" adds="" to="" transient="" collection,="" return="" defn="" ([name="" doc-string?="" attr-map?="" [params*]="" prepost-map?="" body]="" [name="" ([params*]="" body)="" +="" attr-map?])="" macro="" same="" as="" (def="" name="" (fn="" [params*="" ]="" exprs*))="" or="" ([params*="" exprs*)+))="" any="" doc-string="" attrs="" added="" var="" metadata.="" prepost-map="" defines="" map="" optional="" keys="" :pre="" :post="" contain="" collections="" pre="" post="" conditions.="" defrecord="" [&="" fields]="" opts+specs])="" (defrecord="" [fields*]="" options*="" specs*)="" options="" are="" expressed="" sequential="" keywords="" arguments="" (in="" order).="" supported="" options:="" :load-ns="" if="" true,="" importing="" record="" class="" will="" cause="" namespace="" in="" which="" was="" defined="" be="" loaded.="" defaults="" false.="" each="" spec="" consists="" protocol="" interface="" followed="" zero="" more="" method="" bodies:="" protocol-or-interface-or-object="" (methodname="" [args*]="" body)*="" dynamically="" generates="" compiled="" bytecode="" given="" name,="" package="" current="" namespace,="" fields,="" and,="" optionally,="" methods="" protocols="" interfaces.="" have="" (immutable)="" fields="" named="" can="" type="" hints.="" interfaces="" optional.="" only="" supplied="" those="" declared="" note="" bodies="" not="" closures,="" local="" environment="" includes="" accessed="" directly.="" definitions="" take="" form:="" argument="" types="" hinted="" arg="" methodname="" symbols.="" supplied,="" they="" inferred,="" so="" hints="" should="" reserved="" disambiguation.="" all="" desired="" protocol(s)="" interface(s).="" you="" also="" define="" overrides="" object.="" parameter="" must="" correspond="" target="" object="" ('this'="" java="" parlance).="" thus="" one="" than="" do="" declarations.="" recur="" head="" *not*="" pass="" object,="" it="" automatically="" substituted.="" bodies,="" (unqualified)="" used="" (for="" new,="" instance?="" etc).="" implementations="" several="" (clojure.lang)="" generated="" automatically:="" iobj="" (metadata="" support)="" ipersistentmap,="" their="" superinterfaces.="" addition,="" type-and-value-based="," .hashcode="" .equals="" consistent="" contract="" java.util.map.="" when="" aot="" compiling,="" (a="" symbol),="" prepends="" ns="" package,="" writes="" .class="" file="" *compile-path*="" directory.="" two="" constructors="" defined,="" taking="" designated="" metadata="" (nil="" none)="" extension="" field="" none),="" (using="" meta="" fields).="" names="" __meta="" __extmap="" currently="" defining="" your="" own="" records.="" typename="" ...),="" factory="" functions="" defined:="">TypeName, taking positional parameters for the fields,
  and map->TypeName, taking a map of keywords to field values.
-------------------------
clojure.core/ex-info  
([msg map] [msg map cause])
  Create an instance of ExceptionInfo, a RuntimeException subclass
   that carries a map of additional data.
-------------------------
clojure.core/find-keyword  
([name] [ns name])
  Returns a Keyword with the given namespace and name if one already
  exists.  This function will not intern a new keyword. If the keyword
  has not already been interned, it will return nil.  Do not use :
  in the keyword strings, it will be added automatically.
-------------------------
clojure.core/gen-class  
([& options])
Macro  
  When compiling, generates compiled bytecode for a class with the
  given package-qualified :name (which, as all names in these
  parameters, can be a string or symbol), and writes the .class file
  to the *compile-path* directory.  When not compiling, does
  nothing. The gen-class construct contains no implementation, as the
  implementation will be dynamically sought by the generated class in
  functions in an implementing Clojure namespace. Given a generated
  class org.mydomain.MyClass with a method named mymethod, gen-class
  will generate an implementation that looks for a function named by
  (str prefix mymethod) (default prefix: "-") in a
  Clojure namespace specified by :impl-ns
  (defaults to the current namespace). All inherited methods,
  generated methods, and init and main functions (see :methods, :init,
  and :main below) will be found similarly prefixed. By default, the
  static initializer for the generated class will attempt to load the
  Clojure support code for the class as a resource from the classpath,
  e.g. in the example case, ``org/mydomain/MyClass__init.class``. This
  behavior can be controlled by :load-impl-ns

  Note that methods with a maximum of 18 parameters are supported.

  In all subsequent sections taking types, the primitive types can be
  referred to by their Java names (int, float etc), and classes in the
  java.lang package can be used without a package qualifier. All other
  classes must be fully qualified.

  Options should be a set of key/value pairs, all except for :name are optional:

  :name aname

  The package-qualified name of the class to be generated

  :extends aclass

  Specifies the superclass, the non-private methods of which will be
  overridden by the class. If not provided, defaults to Object.

  :implements [interface ...]

  One or more interfaces, the methods of which will be implemented by the class.

  :init name

  If supplied, names a function that will be called with the arguments
  to the constructor. Must return [ [superclass-constructor-args] state]
  If not supplied, the constructor args are passed directly to
  the superclass constructor and the state will be nil

  :constructors {[param-types] [super-param-types], ...}

  By default, constructors are created for the generated class which
  match the signature(s) of the constructors for the superclass. This
  parameter may be used to explicitly specify constructors, each entry
  providing a mapping from a constructor signature to a superclass
  constructor signature. When you supply this, you must supply an :init
  specifier.

  :post-init name

  If supplied, names a function that will be called with the object as
  the first argument, followed by the arguments to the constructor.
  It will be called every time an object of this class is created,
  immediately after all the inherited constructors have completed.
  Its return value is ignored.

  :methods [ [name [param-types] return-type], ...]

  The generated class automatically defines all of the non-private
  methods of its superclasses/interfaces. This parameter can be used
  to specify the signatures of additional methods of the generated
  class. Static methods can be specified with ^{:static true} in the
  signature's metadata. Do not repeat superclass/interface signatures
  here.

  :main boolean

  If supplied and true, a static public main function will be generated. It will
  pass each string of the String[] argument as a separate argument to
  a function called (str prefix main).

  :factory name

  If supplied, a (set of) public static factory function(s) will be
  created with the given name, and the same signature(s) as the
  constructor(s).

  :state name

  If supplied, a public final instance field with the given name will be
  created. You must supply an :init function in order to provide a
  value for the state. Note that, though final, the state can be a ref
  or agent, supporting the creation of Java objects with transactional
  or asynchronous mutation semantics.

  :exposes {protected-field-name {:get name :set name}, ...}

  Since the implementations of the methods of the generated class
  occur in Clojure functions, they have no access to the inherited
  protected fields of the superclass. This parameter can be used to
  generate public getter/setter methods exposing the protected field(s)
  for use in the implementation.

  :exposes-methods {super-method-name exposed-name, ...}

  It is sometimes necessary to call the superclass' implementation of an
  overridden method.  Those methods may be exposed and referred in
  the new method implementation by a local name.

  :prefix string

  Default: "-" Methods called e.g. Foo will be looked up in vars called
  prefixFoo in the implementing ns.

  :impl-ns name

  Default: the name of the current ns. Implementations of methods will be
  looked up in this namespace.

  :load-impl-ns boolean

  Default: true. Causes the static initializer for the generated class
  to reference the load code for the implementing namespace. Should be
  true when implementing-ns is the default, false if you intend to
  load the code via some other method.
-------------------------
clojure.core/import  
([& import-symbols-or-lists])
Macro  
  import-list => (package-symbol class-name-symbols*)

  For each name in class-name-symbols, adds a mapping from name to the
  class named by package.name to the current namespace. Use :import in the ns
  macro in preference to calling this directly.
-------------------------
clojure.core/init-proxy  
([proxy mappings])
  Takes a proxy instance and a map of strings (which must
  correspond to methods of the proxy superclass/superinterfaces) to
  fns (which must take arguments matching the corresponding method,
  plus an additional (explicit) first arg corresponding to this, and
  sets the proxy's fn map.  Returns the proxy.
-------------------------
clojure.core/keyword  
([name] [ns name])
  Returns a Keyword with the given namespace and name.  Do not use :
  in the keyword strings, it will be added automatically.
-------------------------
clojure.core/partial  
([f] [f arg1] [f arg1 arg2] [f arg1 arg2 arg3] [f arg1 arg2 arg3 & more])
  Takes a function f and fewer than the normal arguments to f, and
  returns a fn that takes a variable number of additional args. When
  called, the returned function calls f with args + additional args.
-------------------------
clojure.core/partition  
([n coll] [n step coll] [n step pad coll])
  Returns a lazy sequence of lists of n items each, at offsets step
  apart. If step is not supplied, defaults to n, i.e. the partitions
  do not overlap. If a pad collection is supplied, use its elements as
  necessary to complete last partition upto n items. In case there are
  not enough padding elements, return a partition with less than n items.
-------------------------
clojure.core/proxy  
([class-and-interfaces args & fs])
Macro  
  class-and-interfaces - a vector of class names

  args - a (possibly empty) vector of arguments to the superclass
  constructor.

  f => (name [params*] body) or
  (name ([params*] body) ([params+] body) ...)

  Expands to code which creates a instance of a proxy class that
  implements the named class/interface(s) by calling the supplied
  fns. A single class, if provided, must be first. If not provided it
  defaults to Object.

  The interfaces names must be valid interface types. If a method fn
  is not provided for a class method, the superclass methd will be
  called. If a method fn is not provided for an interface method, an
  UnsupportedOperationException will be thrown should it be
  called. Method fns are closures and can capture the environment in
  which proxy is called. Each method fn takes an additional implicit
  first arg, which is bound to 'this. Note that while method fns can
  be provided to override protected methods, they have no other access
  to protected members, nor to super, as these capabilities cannot be
  proxied.
-------------------------
clojure.core/refer  
([ns-sym & filters])
  refers to all public vars of ns, subject to filters.
  filters can include at most one each of:

  :exclude list-of-symbols
  :only list-of-symbols
  :rename map-of-fromsymbol-tosymbol

  For each public interned var in the namespace named by the symbol,
  adds a mapping from the name of the var to the var to the current
  namespace.  Throws an exception if name is already mapped to
  something else in the current namespace. Filters can be used to
  select a subset, via inclusion or exclusion, or to provide a mapping
  to a symbol different from the var's name, in order to prevent
  clashes. Use :use in the ns macro in preference to calling this directly.
-------------------------
clojure.core/remove-watch  
([reference key])
  Removes a watch (set by add-watch) from a reference
-------------------------
clojure.core/require  
([& args])
  Loads libs, skipping any that are already loaded. Each argument is
  either a libspec that identifies a lib, a prefix list that identifies
  multiple libs whose names share a common prefix, or a flag that modifies
  how all the identified libs are loaded. Use :require in the ns macro
  in preference to calling this directly.

  Libs

  A 'lib' is a named set of resources in classpath whose contents define a
  library of Clojure code. Lib names are symbols and each lib is associated
  with a Clojure namespace and a Java package that share its name. A lib's
  name also locates its root directory within classpath using Java's
  package name to classpath-relative path mapping. All resources in a lib
  should be contained in the directory structure under its root directory.
  All definitions a lib makes should be in its associated namespace.

  'require loads a lib by loading its root resource. The root resource path
  is derived from the lib name in the following manner:
  Consider a lib named by the symbol 'x.y.z; it has the root directory
  /x/y/, and its root resource is /x/y/z.clj. The root
  resource should contain code to create the lib's namespace (usually by using
  the ns macro) and load any additional lib resources.

  Libspecs

  A libspec is a lib name or a vector containing a lib name followed by
  options expressed as sequential keywords and arguments.

  Recognized options:
  :as takes a symbol as its argument and makes that symbol an alias to the
    lib's namespace in the current namespace.
  :refer takes a list of symbols to refer from the namespace or the :all
    keyword to bring in all public vars.

  Prefix Lists

  It's common for Clojure code to depend on several libs whose names have
  the same prefix. When specifying libs, prefix lists can be used to reduce
  repetition. A prefix list contains the shared prefix followed by libspecs
  with the shared prefix removed from the lib names. After removing the
  prefix, the names that remain must not contain any periods.

  Flags

  A flag is a keyword.
  Recognized flags: :reload, :reload-all, :verbose
  :reload forces loading of all the identified libs even if they are
    already loaded
  :reload-all implies :reload and also forces loading of all libs that the
    identified libs directly or indirectly load via require or use
  :verbose triggers printing information about each load, alias, and refer

  Example:

  The following would load the libraries clojure.zip and clojure.set
  abbreviated as 's'.

  (require '(clojure zip [set :as s]))
-------------------------
clojure.core/transduce  
([xform f coll] [xform f init coll])
  reduce with a transformation of f (xf). If init is not
  supplied, (f) will be called to produce it. f should be a reducing
  step function that accepts both 1 and 2 arguments, if it accepts
  only 2 you can add the arity-1 with 'completing'. Returns the result
  of applying (the transformed) xf to init and the first item in coll,
  then applying xf to that result and the 2nd item, etc. If coll
  contains no items, returns init and f is not called. Note that
  certain transforms may inject or skip items.
-------------------------
clojure.core/unchecked-add  
([x y])
  Returns the sum of x and y, both long.
  Note - uses a primitive operator subject to overflow.
-------------------------
clojure.core/unchecked-add-int  
([x y])
  Returns the sum of x and y, both int.
  Note - uses a primitive operator subject to overflow.
-------------------------
clojure.core/update-proxy  
([proxy mappings])
  Takes a proxy instance and a map of strings (which must
  correspond to methods of the proxy superclass/superinterfaces) to
  fns (which must take arguments matching the corresponding method,
  plus an additional (explicit) first arg corresponding to this, and
  updates (via assoc) the proxy's fn map. nil can be passed instead of
  a fn, in which case the corresponding method will revert to the
  default behavior. Note that this function can be used to update the
  behavior of an existing instance without changing its identity.
  Returns the proxy.
-------------------------
clojure.core/use  
([& args])
  Like 'require, but also refers to each lib's namespace using
  clojure.core/refer. Use :use in the ns macro in preference to calling
  this directly.

  'use accepts additional options in libspecs: :exclude, :only, :rename.
  The arguments and semantics for :exclude, :only, and :rename are the same
  as those documented for clojure.core/refer.
-------------------------
cljs.core/defn  
([name doc-string? attr-map? [params*] prepost-map? body] [name doc-string? attr-map? ([params*] prepost-map? body) + attr-map?])
Macro  
  Same as (def name (core/fn [params* ] exprs*)) or (def
    name (core/fn ([params* ] exprs*)+)) with any doc-string or attrs added
    to the var metadata. prepost-map defines a map with optional keys
    :pre and :post that contain collections of pre or post conditions.
-------------------------
cljs.core/defrecord  
([rsym fields & impls])
Macro  
  (defrecord name [fields*]  options* specs*)

  Currently there are no options.

  Each spec consists of a protocol or interface name followed by zero
  or more method bodies:

  protocol-or-Object
  (methodName [args*] body)*

  The record will have the (immutable) fields named by
  fields, which can have type hints. Protocols and methods
  are optional. The only methods that can be supplied are those
  declared in the protocols.  Note that method bodies are
  not closures, the local environment includes only the named fields,
  and those fields can be accessed directly.

  Method definitions take the form:

  (methodname [args*] body)

  The argument and return types can be hinted on the arg and
  methodname symbols. If not supplied, they will be inferred, so type
  hints should be reserved for disambiguation.

  Methods should be supplied for all methods of the desired
  protocol(s). You can also define overrides for
  methods of Object. Note that a parameter must be supplied to
  correspond to the target object ('this' in JavaScript parlance). Note also
  that recur calls to the method head should *not* pass the target object, it
  will be supplied automatically and can not be substituted.

  In the method bodies, the (unqualified) name can be used to name the
  class (for calls to new, instance? etc).

  The type will have implementations of several ClojureScript
  protocol generated automatically: IMeta/IWithMeta (metadata support) and
  IMap, etc.

  In addition, defrecord will define type-and-value-based =,
  and will define ClojureScript IHash and IEquiv.

  Two constructors will be defined, one taking the designated fields
  followed by a metadata map (nil for none) and an extension field
  map (nil for none), and one taking only the fields (using nil for
  meta and extension fields). Note that the field names __meta
  and __extmap are currently reserved and should not be used when
  defining your own records.

  Given (defrecord TypeName ...), two factory functions will be
  defined: ->TypeName, taking positional parameters for the fields,
  and map->TypeName, taking a map of keywords to field values.
-------------------------
cljs.core/import  
([& import-symbols-or-lists])
Macro  
  import-list => (closure-namespace constructor-name-symbols*)

  For each name in constructor-name-symbols, adds a mapping from name to the
  constructor named by closure-namespace to the current namespace. Use :import in the ns
  macro in preference to calling this directly.
-------------------------
cljs.core/require  
([& args])
Macro  
  Loads libs, skipping any that are already loaded. Each argument is
  either a libspec that identifies a lib or a flag that modifies how all the identified
  libs are loaded. Use :require in the ns macro in preference to calling this
  directly.

  Libs

  A 'lib' is a named set of resources in classpath whose contents define a
  library of ClojureScript code. Lib names are symbols and each lib is associated
  with a ClojureScript namespace. A lib's name also locates its root directory
  within classpath using Java's package name to classpath-relative path mapping.
  All resources in a lib should be contained in the directory structure under its
  root directory. All definitions a lib makes should be in its associated namespace.

  'require loads a lib by loading its root resource. The root resource path
  is derived from the lib name in the following manner:
  Consider a lib named by the symbol 'x.y.z; it has the root directory
  /x/y/, and its root resource is /x/y/z.clj. The root
  resource should contain code to create the lib's namespace (usually by using
  the ns macro) and load any additional lib resources.

  Libspecs

  A libspec is a lib name or a vector containing a lib name followed by
  options expressed as sequential keywords and arguments.

  Recognized options:
  :as takes a symbol as its argument and makes that symbol an alias to the
    lib's namespace in the current namespace.
  :refer takes a list of symbols to refer from the namespace.
  :refer-macros takes a list of macro symbols to refer from the namespace.
  :include-macros true causes macros from the namespace to be required.
  :rename specifies a map from referred var names to different
    symbols (and can be used to prevent clashes)


  Flags

  A flag is a keyword.
  Recognized flags: :reload, :reload-all, :verbose
  :reload forces loading of all the identified libs even if they are
    already loaded
  :reload-all implies :reload and also forces loading of all libs that the
    identified libs directly or indirectly load via require or use
  :verbose triggers printing information about each load, alias, and refer

  Example:

  The following would load the library clojure.string :as string.

  (require '[clojure.string :as string])
-------------------------
clojure.test  
  A unit testing framework.

   ASSERTIONS

   The core of the library is the "is" macro, which lets you make
   assertions of any arbitrary expression:

   (is (= 4 (+ 2 2)))
   (is (instance? Integer 256))
   (is (.startsWith "abcde" "ab"))

   You can type an "is" expression directly at the REPL, which will
   print a message if it fails.

       user> (is (= 5 (+ 2 2)))

       FAIL in  (:1)
       expected: (= 5 (+ 2 2))
         actual: (not (= 5 4))
       false

   The "expected:" line shows you the original expression, and the
   "actual:" shows you what actually happened.  In this case, it
   shows that (+ 2 2) returned 4, which is not = to 5.  Finally, the
   "false" on the last line is the value returned from the
   expression.  The "is" macro always returns the result of the
   inner expression.

   There are two special assertions for testing exceptions.  The
   "(is (thrown? c ...))" form tests if an exception of class c is
   thrown:

   (is (thrown? ArithmeticException (/ 1 0)))

   "(is (thrown-with-msg? c re ...))" does the same thing and also
   tests that the message on the exception matches the regular
   expression re:

   (is (thrown-with-msg? ArithmeticException #"Divide by zero"
                         (/ 1 0)))

   DOCUMENTING TESTS

   "is" takes an optional second argument, a string describing the
   assertion.  This message will be included in the error report.

   (is (= 5 (+ 2 2)) "Crazy arithmetic")

   In addition, you can document groups of assertions with the
   "testing" macro, which takes a string followed by any number of
   assertions.  The string will be included in failure reports.
   Calls to "testing" may be nested, and all of the strings will be
   joined together with spaces in the final report, in a style
   similar to RSpec 

   (testing "Arithmetic"
     (testing "with positive integers"
       (is (= 4 (+ 2 2)))
       (is (= 7 (+ 3 4))))
     (testing "with negative integers"
       (is (= -4 (+ -2 -2)))
       (is (= -1 (+ 3 -4)))))

   Note that, unlike RSpec, the "testing" macro may only be used
   INSIDE a "deftest" or "with-test" form (see below).


   DEFINING TESTS

   There are two ways to define tests.  The "with-test" macro takes
   a defn or def form as its first argument, followed by any number
   of assertions.  The tests will be stored as metadata on the
   definition.

   (with-test
       (defn my-function [x y]
         (+ x y))
     (is (= 4 (my-function 2 2)))
     (is (= 7 (my-function 3 4))))

   As of Clojure SVN rev. 1221, this does not work with defmacro.
   See http://code.google.com/p/clojure/issues/detail?id=51

   The other way lets you define tests separately from the rest of
   your code, even in a different namespace:

   (deftest addition
     (is (= 4 (+ 2 2)))
     (is (= 7 (+ 3 4))))

   (deftest subtraction
     (is (= 1 (- 4 3)))
     (is (= 3 (- 7 4))))

   This creates functions named "addition" and "subtraction", which
   can be called like any other function.  Therefore, tests can be
   grouped and composed, in a style similar to the test framework in
   Peter Seibel's "Practical Common Lisp"
   

   (deftest arithmetic
     (addition)
     (subtraction))

   The names of the nested tests will be joined in a list, like
   "(arithmetic addition)", in failure reports.  You can use nested
   tests to set up a context shared by several tests.


   RUNNING TESTS

   Run tests with the function "(run-tests namespaces...)":

   (run-tests 'your.namespace 'some.other.namespace)

   If you don't specify any namespaces, the current namespace is
   used.  To run all tests in all namespaces, use "(run-all-tests)".

   By default, these functions will search for all tests defined in
   a namespace and run them in an undefined order.  However, if you
   are composing tests, as in the "arithmetic" example above, you
   probably do not want the "addition" and "subtraction" tests run
   separately.  In that case, you must define a special function
   named "test-ns-hook" that runs your tests in the correct order:

   (defn test-ns-hook []
     (arithmetic))

   Note: test-ns-hook prevents execution of fixtures (see below).


   OMITTING TESTS FROM PRODUCTION CODE

   You can bind the variable "*load-tests*" to false when loading or
   compiling code in production.  This will prevent any tests from
   being created by "with-test" or "deftest".


   FIXTURES

   Fixtures allow you to run code before and after tests, to set up
   the context in which tests should be run.

   A fixture is just a function that calls another function passed as
   an argument.  It looks like this:

   (defn my-fixture [f]
      Perform setup, establish bindings, whatever.
     (f)  Then call the function we were passed.
      Tear-down / clean-up code here.
    )

   Fixtures are attached to namespaces in one of two ways.  "each"
   fixtures are run repeatedly, once for each test function created
   with "deftest" or "with-test".  "each" fixtures are useful for
   establishing a consistent before/after state for each test, like
   clearing out database tables.

   "each" fixtures can be attached to the current namespace like this:
   (use-fixtures :each fixture1 fixture2 ...)
   The fixture1, fixture2 are just functions like the example above.
   They can also be anonymous functions, like this:
   (use-fixtures :each (fn [f] setup... (f) cleanup...))

   The other kind of fixture, a "once" fixture, is only run once,
   around ALL the tests in the namespace.  "once" fixtures are useful
   for tasks that only need to be performed once, like establishing
   database connections, or for time-consuming tasks.

   Attach "once" fixtures to the current namespace like this:
   (use-fixtures :once fixture1 fixture2 ...)

   Note: Fixtures and test-ns-hook are mutually incompatible.  If you
   are using test-ns-hook, fixture functions will *never* be run.


   SAVING TEST OUTPUT TO A FILE

   All the test reporting functions write to the var *test-out*.  By
   default, this is the same as *out*, but you can rebind it to any
   PrintWriter.  For example, it could be a file opened with
   clojure.java.io/writer.


   EXTENDING TEST-IS (ADVANCED)

   You can extend the behavior of the "is" macro by defining new
   methods for the "assert-expr" multimethod.  These methods are
   called during expansion of the "is" macro, so they should return
   quoted forms to be evaluated.

   You can plug in your own test-reporting framework by rebinding
   the "report" function: (report event)

   The 'event' argument is a map.  It will always have a :type key,
   whose value will be a keyword signaling the type of event being
   reported.  Standard events with :type value of :pass, :fail, and
   :error are called when an assertion passes, fails, and throws an
   exception, respectively.  In that case, the event will also have
   the following keys:

     :expected   The form that was expected to be true
     :actual     A form representing what actually occurred
     :message    The string message given as an argument to 'is'

   The "testing" strings will be a list in "*testing-contexts*", and
   the vars being tested will be a list in "*testing-vars*".

   Your "report" function should wrap any printing calls in the
   "with-test-out" macro, which rebinds *out* to the current value
   of *test-out*.

   For additional event types, see the examples in the code.

-------------------------
clojure.pprint  
  A Pretty Printer for Clojure

clojure.pprint implements a flexible system for printing structured data  
in a pleasing, easy-to-understand format. Basic use of the pretty printer is  
simple, just call pprint instead of println. More advanced users can use  
the building blocks provided to create custom output formats.

Out of the box, pprint supports a simple structured format for basic data  
and a specialized format for Clojure source code. More advanced formats,  
including formats that don't look like Clojure data at all like XML and  
JSON, can be rendered by creating custom dispatch functions.

In addition to the pprint function, this module contains cl-format, a text  
formatting function which is fully compatible with the format function in  
Common Lisp. Because pretty printing directives are directly integrated with  
cl-format, it supports very concise custom dispatch. It also provides  
a more powerful alternative to Clojure's standard format function.

See documentation for pprint and cl-format for more information or  
complete documentation on the the clojure web site on github.  
-------------------------
cljs.closure  
  Compile ClojureScript to JavaScript with optimizations from Google
   Closure Compiler producing runnable JavaScript.

   The Closure Compiler (compiler.jar) must be on the classpath.

   Use the 'build' function for end-to-end compilation.

   build = find-sources -> add-dependencies -> compile -> optimize -> output

   Two protocols are defined: IJavaScript and Compilable. The
   Compilable protocol is satisfied by something which can return one
   or more IJavaScripts.

   With IJavaScript objects in hand, calling add-dependencies will
   produce a sequence of IJavaScript objects which includes all
   required dependencies from the Closure library and ClojureScript,
   in dependency order. This function replaces the closurebuilder
   tool.

   The optimize function converts one or more IJavaScripts into a
   single string of JavaScript source code using the Closure Compiler
   API.

   The produced output is either a single string of optimized
   JavaScript or a deps file for use during development.

nil  

Grouping

Since I'm talking about function discoverability, here's another thing I complained about:

... the clojure.core namespace is huge...

How huge?

user=> (count (keys (ns-publics 'clojure.core)))  
621  

(I tried to figure this out on my own, but in the end I had to resort to StackOverflow)

This includes:

• special forms
• arithmetic, bitwise, boolean operators
• global variables
• some string functions (but others are in clojure.string
• regex functions
• flow control functions & macros
• introspection & type stuff
• concurrency stuff (though core.async is separate)
• stuff that deal with functions (e.g. fn, defn, partial)
• Java interop
• Collections and related functions
• IO
• and much more — a more complete overview can be found on the ClojureDocs quickref.

I understand the reasoning behind this; you need most of those for day to day work, and making you require a namespace just to get map/filter would get old really quick. However, this makes it really difficult for a newcomer to discover what they are looking for.

Proposal

I believe that fixing the above stated problems is quite doable and can be even done in parallel:

• Expand or add to the somewhat dry and too-concise function documentation. Use a format that can link to other functions, show in-line examples etc.
• Group the various functions into related functionality. The ClojureDocs quick reference linked above goes a long way towards this, but needs to be officially adopted and maintained.
• Link the API docs back to the Reference and Guides (the Reference already links to the API docs).
• Do all this in a way that lends itself to better doc generation, incl. integration with 3rd-party tools like Dash.

Update: See also the 2017 ClojuTRE talk Towards Awesome Clojure Documentation by Bozhidar Batsov that goes into some more examples and proposals.

Meanwhile

I'd like to thank all the Clojure contributors for their work on the language. I hope I will have time in the future to contribute my own time towards some of those ideas, as this is the only feasible way to grow a language.

In the meantime, I advise people that might be bitten by all the shortcomings in the API docs, to do what I did: read a free book, read the Reference, then take some time to go over all the clojure.core functions once or twice. The last bit shouldn't take more than a couple of hours of focused work, and will be tough if you're not used to math-like precise writing, but it will pay off in many ways.

Diagnosing a process leak (that manifested as a memory leak) using the remote console - fantastic! #myelixirstatus

— Orestis Markou (@orestis) December 29, 2017

For the benefit of @AndrewShatnyy who inquired, I'll describe here the problem, how I went about diagnosing it, and the solution.

The system

For a recent gig, I built a "Show Control" system that periodically (every minute) reaches out to various machines (could be via a HTTP endpoint or a raw TCP/IP connection) and runs a few commands, to ensure each machine is at the expected state. The most basic usage scenario is ensuring that machines are on or off depending on the venue's schedule of the day.

To model this, I used a GenServer process for each machine, which are discovered via a Registry. Since the system will have at most hundreds of machines, and the Erlang process limit by default is 250k, this seemed like a feasible approach.

My reasoning behind having a persistent process per machine was mainly observability and avoiding an accidental DOS of a remote machine if the server for some reason made a whole bunch of requests — having a single process meant those requests would be serialized. I realise now that I could go with a simpler architecture, but this was my first design in Elixir. In any case, this architecture was not the problem, so let's move on.

The problem

I received a complaint from a user that the system had stopped responding: "I get a spinning wheel but no response". (The web UI was served by a different machine, so it was only the control commands that had an issue).

I have already setup a Prometheus + Grafana dashboard via prometheus_ex so after some digging around the available metrics, I had this (dates include a system restart and a permanent fix):

We see that the process memory has reached 9GB while the system memory was just over 3GB, and they seem to be correlated. Not really visible in this graph, but available when mousing over the data, was that the number of processes was steadily climbing: 125000 at the time of the restart, whereas just after the restart the process count hovered around only 800 but still growing.

Amazingly, even after consuming 12GB of memory, the system was still running, albeit very slow; the VM this was running on had 16GB allocated so it wasn't swapping to disk yet. A quick restart resolved the issue but the leak was still present.

More info needed

Seeing that the processes were steadily climbing, together with the inherent periodicity of the system, immediately pointed to something that spawned processes and never killed them.

In order to get some more insight on the runtime characteristics of the system, I installed Wobserver — this is a read-only observer clone exposed via a web server. This saves me from trying to setup an ssh tunnel to attach a remote console so I can run the real observer.

I then let the processes climb to around 10k, and went over each Application tree and see whether the problem was in my architecture:

The picture above shows one Supervisor with many children — however, not that many. Those are the GenServers that correspond to the physical machines, around 200 of them, while I'm looking for thousands of processes.

So if the extra processes didn't show up as part of an applications, it means they were never linked to a parent process.

The next step was going over to the Processes tab of Wobserver:

The picture above shows all the processes running in that node, their "initial function" (the one used to spawn them), the number of reductions, their memory usage, the number of messages in their mailbox queue and the currently running function.

NB: Both of the above screens are interactive; you can click on a process PID to get more information about it. However, exploring this via a web server will increase the load of the system considerably. This is why I tried it with a reduced number of stray processes.

Using that list, I browsed around trying to spot a pattern. One particular name, let's call it Pinger was statistically more present, however, since the list is live-updating it's hard to see if some processes are ephemeral or really stray. The PID numbers give some indication of process age but they are mostly opaque and cannot be relied upon.

Digging deeper

I got some hints via Wobserver, and now it was the time to attach a console to the remote node and dig around manually.

To do this, I ssh'ed into the remote machine. This node was deployed as a Release (which you should always use), via Distillery. This gives you a command line script that you can use to do various tasks:

# ./my-release/bin/my_app
Usage: my_app <task>

Service Control  
=======================
start                          # start my_app as a daemon  
start_boot <file>              # start my_app as a daemon, but supply a custom .boot file  
foreground                     # start my_app in the foreground  
console                        # start my_app with a console attached  
console_clean                  # start a console with code paths set but no apps loaded/started  
console_boot <file>            # start my_app with a console attached, but supply a custom .boot file  
stop                           # stop the my_app daemon  
restart                        # restart the my_app daemon without shutting down the VM  
reboot                         # restart the my_app daemon  
reload_config                  # reload the current system's configuration from disk

Upgrades  
=======================
upgrade <version>              # upgrade my_app to <version>  
downgrade <version>            # downgrade my_app to <version>  
install <version>              # install the my_app-<version> release, but do not upgrade to it

Utilities  
=======================
attach                         # attach the current TTY to my_app's console  
remote_console                 # remote shell to my_app's console  
pid                            # get the pid of the running my_app instance  
ping                           # checks if my_app is running, pong is returned if successful  
pingpeer <peer>                # check if a peer node is running, pong is returned if successful  
escript <file>                 # execute an escript  
rpc <mod> <fun> [<args..>]     # execute an RPC call using the given MFA  
rpcterms <mod> <fun> [<expr>]  # execute an RPC call using the given Erlang expression for args  
eval <expr>                    # execute the given Erlang expression on the running node  
command <mod> <fun> [<args..>] # execute the given MFA  
describe                       # print useful information about the my_app release

Custom Commands  
=======================
No custom commands found.  

What we want is a remote_console:

# ./my_release/bin/my_app remote_console
Erlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.5.0) - press Ctrl+C to exit (type h() ENTER for help)  

And we're in! Now we can start not only introspecting but also messing around with our live system. Be careful!

I have since fixed the bug, and I don't want to re-introduce it, so I'll fudge the output of the commands for illustrative purposes.

The first thing is to get a process list:

> Process.list()
[#PID<0.0.0>, #PID<0.1.0>, #PID<0.2.0>, #PID<0.3.0>, #PID<0.6.0>,
 #PID<0.1131.0>, #PID<0.1132.0>, #PID<0.1134.0>, #PID<0.1135.0>, #PID<0.1136.0>,
 #PID<0.1137.0>, #PID<0.1139.0>, #PID<0.1140.0>, #PID<0.1141.0>, #PID<0.1142.0>,
 #PID<0.1143.0>, #PID<0.1144.0>, #PID<0.1145.0>, #PID<0.1146.0>, #PID<0.1147.0>,
 #PID<0.1148.0>, #PID<0.1149.0>, #PID<0.1150.0>, #PID<0.1151.0>, #PID<0.1152.0>,
 #PID<0.1153.0>, #PID<0.1154.0>, #PID<0.1155.0>, #PID<0.1156.0>, #PID<0.1157.0>,
 #PID<0.1158.0>, #PID<0.1168.0>, #PID<0.1169.0>, #PID<0.1170.0>, #PID<0.1171.0>,
 #PID<0.1172.0>, #PID<0.1173.0>, #PID<0.1174.0>, #PID<0.1175.0>, #PID<0.1176.0>,
 #PID<0.1177.0>, #PID<0.1178.0>, #PID<0.1185.0>, #PID<0.1186.0>, #PID<0.1187.0>,
 #PID<0.1188.0>, #PID<0.1191.0>, #PID<0.1192.0>, #PID<0.1193.0>, #PID<0.1194.0>,
 ...]

Note how the shell won't print all ten thousand members of that list.

> Process.list() |> Enum.count()
10782  

You can use Process.info/1 to get information for a process:

> Process.list() |> Enum.at(0) |> Process.info()
[registered_name: :init, current_function: {:init, :loop, 1},
 initial_call: {:otp_ring0, :start, 2}, status: :waiting, message_queue_len: 0,
 messages: [], links: [#PID<0.1131.0>, #PID<0.1132.0>, #PID<0.6.0>],
 dictionary: [], trap_exit: true, error_handler: :error_handler,
 priority: :normal, group_leader: #PID<0.0.0>, total_heap_size: 2585,
 heap_size: 1598, stack_size: 2, reductions: 4304,
 garbage_collection: [max_heap_size: %{error_logger: true, kill: true, size: 0},
  min_bin_vheap_size: 46422, min_heap_size: 233, fullsweep_after: 65535,
  minor_gcs: 3], suspending: []]

or if you know what you want, you can use Process.info/2:

> Process.list() |> Enum.at(23) |> Process.info([:initial_call])
[initial_call: {:proc_lib, :init_p, 5}]

Annoyingly, this :initial_call doesn't match the "Pinger" name I was seeing in Wobserver. After some fussing around, I got to this:

> Process.list() |> Enum.at(23) |> Process.info([:dictionary]) |> Keyword.get(:dictionary) |> Keyword.get(:"$initial_call")
{MyApp.Pinger, :init, 1}

It turns out that since all OTP GenServers go through the same init call, the actual init callback is stored in the process dictionary as :"$initial_call" (this is an atom with special characters, so it needs to be quoted). The result is an "MFA" or a {Module, Function, Arity} tuple.

Let's make a filter function that will give us all those processes:

init_call_filter = &(Process.info(&1, [:dictionary])[:dictionary] |> Keyword.get(:"$initial_call") == {MyApp.Pinger, :init, 1})  

This is an example of Elixir's shorthand function definition syntax; &1 is the first argument of the function.

Now we can get a list of all the relevant functions:

> Process.list() |> Enum.filter(init_call_filter) |> Enum.count()
9856  

Bingo! At this point, I dove into the code to find the root cause of the problem. After seeing that indeed there were cases that this process was launched but never exited, I wanted to confirm my hypothesis:

> Process.list() |> Enum.filter(init_call_filter) |> Enum.map(&(Process.exit(&1, :kill)))
[true, true, true, true, true, true, true, true, true, true, true, true, true,
 true, true, true, true, true, true, true, true, true, true, true, true, true,
 true, true, true, true, true, true, true, true, true, true, true, true, true,
 true, true, true, true, true, true, true, true, true, true, true, ...]

This immediately killed our 10k processes via Process.exit/2. Note that you have to use :kill as the reason, :normal will get ignored.

then:

> Process.list() |> Enum.filter(init_call_filter) |> Enum.count()
0  

wait a bit:

> Process.list() |> Enum.filter(init_call_filter) |> Enum.count()
7  

...and bit more...

> Process.list() |> Enum.filter(init_call_filter) |> Enum.count()
194  

Ah! A bit later:

> Process.list() |> Enum.filter(init_call_filter) |> Enum.count()
14  

So the problem was that out of roughly 200 pings a minute, 7 were never exiting. Funnily, even after killing 10k processes, the process memory didn't go down to what I expected. Perhaps there wasn't enough time to trigger that part of Garbage Collection. In any case, I had to deploy a fix so it didn't matter for my case as the node would be restarted.

The solution

The Pinger code was added at the early stages of the system, mainly to see some stats in the logs back when there were only 5 machines online. I now trust the system and network much more, so the Pinger module was unceremoniously evicted from the code base.

Since the fix, the systems hovers around the 780 process mark, with a total memory usage of 80MB. Not too shabby!

I hope you enjoyed my little adventure. Lessons learned:

• Add some alerting for the various metrics exposed by the BEAM. I should have received an alert way before users complained.
• Think before you spawn!
• A better architecture might have precluded this problem in the first place.

Happy new year!

Learning a new language by solving those puzzles has a few notable benefits:

• A nice 25 day streak, so that you can build a habit.
• Small and self contained, usually solved in 30-60 minutes without needing to bring in any dependencies.
• You get to exercise the standard library and built-in data structures a lot.
• You get to implement some well-known algorithms (e.g. A*, depth- or breadth-first tree search, graph walking etc.)
• You get a feel of the performance characteristics, since some puzzles require some intense calculations.
• You can read other solutions in your language of choice in order to learn new tricks.
• It's fun!

In 2016, I tried Elixir. For 2017, it was Clojure. In both cases, I've started a couple of weeks in advance by actually learning the language and libraries, solving older puzzles, so I can be ready on December 1st for the new puzzles.

Here is a rough list of my takeaways, after two solid months of daily Clojure puzzle-solving:

• It took me roughly two weeks to get used to the (parentheses). I don't notice them any more.
• The Clojure community is very friendly and very experienced.
• It helps to have some functional programming experience before trying Clojure. In my case, Elixir had many many similarities so at times I felt like just learning a new syntax, rather than a new way of programming.
• Clojure is fast, even when using the functional and immutable data structures. If you need to go faster, you can drop down to the transient versions, to the Java collections, to the unboxed math operations.
• Clojure is flexible. You get immutability by default, but if an algorithm is better expressed by having a mutable data structure you can also have that.
• Having a REPL that is integrated with your editor (in my case, Spacemacs + CIDER) is eye-opening. I will try and make a screencast or stream of what this looks like when solving those puzzles — it's really hard to put into writing, but it makes other REPLs seem downright primitive.
• The Clojure documentation can be hard to decipher, and the clojure.core namespace is huge. ClojureDocs helps, but I believe there's a need for an apropos-like functionality built into the system.
• In the first couple of weeks, the error messages where a bit cryptic. They still are, but now I can decipher the common ones. Doing incremental work with a REPL-connected editor mitigates this somewhat. Also, CIDER has a built-in debugger that's very easy to use and has saved me from littering my code with println statements.
• Java is huge, and the JVM is powerful. I haven't touched the ecosystem since 2007, and I remember now why most enterprise software is written in Java: There's a robust library for pretty much everything you want to do. Being able to explore and use those libraries from a higher-level language is very very interesting and unlocks a huge amount of potential. Even the JDK itself has a great standard library and can do pretty much everything.
• Reading other people's code can be either an enlightening experience or an exercise in frustration. This doesn't seem to be dependant on the author of the code, but rather on whether they are using the same 80% of the language you are using. To be fair, most people that are solving those puzzles are not aiming for readability or maintainability.

In the end, my verdict on Clojure is whole-heartedly positive. However, the next avenue I want to explore is actually ClojureScript, both on client side (probably via re-frame), but also on server side (on Node.js).

In case anyone is interested, you can find my 2017 solutions on GitHub. You can also find a community-curated list of Advent of Code solutions in Clojure. Finally, a special mention to Mike Fikes's solutions that are in cross-platform Clojure and ClojureScript.

I want to preface this by saying that of course there are niches (small and large) and "killer libraries" for those niches that may render all of my arguments moot; I'm talking mostly in the context of doing "general-purpose" services and applications that run on a few servers somewhere. Certainly web services, but also other not-so-web services.

Consider Python in the context of web development. It has a few very mature web frameworks out there, a fantastic asynchronous networking library, an excellent and vibrant community — yet always there is a "ceiling" of load that a web application can handle before getting rewritten to something else, usually Go, perhaps Java; certainly a less dynamic and closer to the metal language that has "real concurrency".

Even by replacing the CPython VM with something like PyPy, you still have to the GIL and therefore you have to have some event-loop based mechanism to do concurrency (sadly other VMs like Jython and IronPython didn't survive the Python 3 transition, so we'll never know what a multi-core Python could work like in practice).

I don't mean to single out Python; Ruby and NodeJS also suffer from this ceiling; The moment you want your web service to do something more "interesting", you start to pull in the dependencies: Redis, RabbitMQ, Memcached and a host of similar tools that are capable of running for a long time, serve multiple concurrent connections in one process, maintain state and not leak memory.

My first a-ha moment: if you know one of these languages, there's no point in learning the others; you'll hit the same ceiling sooner or later. I think this is obvious by the perceived lack of lateral movement between the Python, Ruby and Javascript communities.

Even by taking those out of the counting, there's certainly no shortage of languages to consider. You can easily spend a week going over the TIOBE's 100-most-popular languages index, trying to compare strengths and weaknesses for each.

Assuming though that you need something cross-platform and general-purpose enough, here's my thinking:

1. Pick the ones with a runtime (or VM);
2. Evaluate the runtime (or VM) instead;
3. Pick a language that targets the runtime that suits you.

Why runtimes are important:

• A runtime usually only gains in performance, security, stability and you get all that for free, without having to rewrite your code.

• Using a multi-language runtime means you benefit from existing libraries in other languages, assuming a sane interop strategy.

• New languages will appear that target your runtime, and you can choose to write parts of your system that make sense in them, without ditching your entire codebase.

I think this whole exercise narrows it down to only a handful of contenders:

• JVM
• CLR
• BEAM

My second a-ha moment: You want to focus on the runtime and not the language, because the runtime is what makes or breaks your software in production. The obvious point here is where are the languages that target the Go runtime? Does a Go runtime even exist?

All three are old and battle-tested enough across a whole range of companies and services that should be safe bets. Also all three have major corporations (Oracle, Microsoft, Ericsson) funding their maintenance and development. You really can't go wrong by picking one of those three. Perhaps you'll rule out CLR since it only recently (partly) became Open Source.

From those three, BEAM is quite interesting; It's the only VM I know of that provides pre-emptive concurrency, so that you don't have to write "cooperative" code. Of course, that comes at the tradeoff of performance and expressiveness (in the sense that you can never have shared memory, if you need it). But still you can go quite far.

Elixir has quickly gathered a community of developers that are giving back a lot of useful libraries; plus, as mentioned before, you get access to all the existing and battle-tested Erlang libraries. I personally think that if you're developing web-services without crazy number-crunching requirements, you should really look into Elixir and OTP.

If you want something more general purpose, you might want to look into Clojure; my tip for sticking with it is to keep counting the parentheses: it's always the same number as with your current language, it's just that foo(a, b) looks more familiar than (foo a b) - plus, there's fewer commas and semicolons :)

So finally we come to the elephant in the room: Javascript.

It's true that Javascript is the queen of languages - it can move everywhere. And indeed in has. Apart from being, of course, the only language that can run in the browser.

There was a point around a couple of years ago that ES2015 (aka ES6) had gained 100% support in most browsers, meaning you could ditch most of the transpiling and general complexity, at least during development. The future was bright; finally a modern Javascript dialect that can help us write more expressive programs without bringing in a billion dependencies and a convoluted build chain

Unfortunately, upon closer inspection, soon after ES2015 was finalised, Javascript became a "living" language. Meaning new syntax would be added all the time, and browsers and tool maintainers would need to play the catch-up game. And that would be OK if the focus was on actual filling in the Standard Library gaps, but instead the focus was mostly on new syntax sugar.

This was the point where I gave up and switched new development to Typescript. I felt that it was a more stable platform I could build on. Soon after though, Typescript started moving rapidly again, slowly gaining new features on its type system while also trying to keep up with Javascript (since it aims to be a superset).

At that point, I knew that the cycle of churn would soon begin anew; and I turned to look for other "compiles-to-JS" languages that treat Javascript as byte code and not as something that needs to be readable.

Out of many many contenders, I focused on two:

• ReasonML/Bucklescript/OCaml
• ClojureScript

I spent a month or so learning Reason and OCaml, but it felt that the ecosystem itself was much too young to be a robust platform, and even more crucially, the Standard Library story was just not there - it was not the goal of ReasonML to create and maintain a sane Standard Library for the web. Certainly though given the pace of development, it is worth another look in a year or so.

On the other hand, the ClojureScript approach is to build upon the existing Clojure Standard Library, and rely on the Google Closure compiler doing dead code elimination to ensure the delivered bundles are not bloated. I had a very pleasant experience in using the (Javascript version) of Google Closure: only 4MB of node_modules, ES2015 to ES3 transpilation (great for even very old browsers) and all the performance optimisations you can handle. In addition, the ecosystem has some top-notch developer productivity tool and has recently seen some major improvements in their interop story.

My third a-ha moment: For the browser, pick an advanced language that compiles to Javascript, and you'll never experience JS fatigue again.

Soppy note: While writing this post I felt a bit bittersweet because I realised I'll probably won't use Python again in a production setting. Python was the language that propelled my software career and it does feel weird to be moving on. I would like to thank all the wonderful people that work on Python, and all the fantastic, caring and thoughtful community. You are the best!

As an example, let's say we have a list of entries. Each entry has a title and one or more authors. In my particular case, we fetch the entries from a JSON endpoint and render each one using Handlebars, but it could be easily server-side rendered. The entries are not enough to warrant pagination or server-side searching, but enough to make it difficult to find a single entry you're looking for.

We turn out to client side filtering: render all the data in the DOM, but allow the user to narrow it down as they type. In our case, to keep things explicit, we want two separate search boxes, one that filters only by title, and one that filters by author. We also only care about filtering while maintaining the server-side ordering (so no client side sorting).

Here is what a pseudo-handlebars template could look like for each row:

<tr class="entry-row">  
  <td>{{name}}</td>
  <td>{{#each authors}}{{this.name}}{{#unless @last}}, {{/unless}}{{/each}}</td>
</tr>  

What we are going to do is add a couple of attributes to the row:

<tr class="entry-row" data-filterable data-name="{{lowercase name}}" data-authors="{{#each authors}}{{lowercase this.name}} {{/each}}">  
  <td>{{name}}</td>
  <td>{{#each authors}}{{this.name}}{{#unless @last}}, {{/unless}}{{/each}}</td>
</tr>  

We are adding:

• data-filterable - a generic property that all the entries in this table will share
• data-name, data-authors - the information that will control the search

Up until now this is a very common approach, decorate-then-search. The cool part comes next:

Let's say the user has typed "terry" in the author search box. It turns out we can write two simple CSS selectors:

const matchSelector = '[data-filterable][data-authors*="terry"]';  
const reverseSelector = '[data-filterable]:not([data-authors*="terry"])'  

First, we narrow down the selectors to only elements that have a data-filterable attribute. Then, we search for elements that have "terry" anywhere in the value of the data-authors attribute. For the reverse we use the :not() pseudo-selector to just reverse it - it's almost symmetrical to the first one so really trivial to create.

We can then do:

const matchingElements = document.querySelectorAll(matchSelector);  
const notMatchingElements = document.querySelectorAll(reverseSelector);  

...to get two mutually exclusive NodeLists. You can then iterate over them and add/remove classes that show/hide them as wanted. To clear the search, just combine them and remove all the search-related classes.

What about searching entries that have both "terry" in their authors and "omens" in their title?

const matchSelector = '[data-filterable][data-authors*="terry"][data-title*="omens"]';  

You can go on adding selectors as needed, perhaps adding a simple query language to search entries that have "terry" AND "neil" in their authors. The resulting selector would look like this:

const matchSelector = '[data-filterable][data-authors*="terry"][data-authors*="neil"]';  

The nice thing about all this is that all the logic is moved to the CSS selectors, and all the search function cares about is really two mutually-exclusive lists of elements.

There are obvious limitations but for my use case it was extremely useful and performance seems lightning fast (as the actual searching is done by the browser and by Javascript traversing the DOM to check every element).

That perception, while indeed having a dose of truth, is magnified out of proportion because in today's "literature"¹ web development means different things to different people. So when talking or thinking about web development, you have to use a qualifier to talk about which kind of web development is meant.

What does "web development" even mean?

At its core, the term web development means "delivering an experience through the browser".²

However, the term experience means different things to different people and organizations. One primitive categorization could be:

• Static websites (so-called "brochures")
• Semi-dynamic websites (news, blogs)
• Dynamic websites (e-shops, social networking)
• Public applications (CMS interfaces, Project management tools)
• Private applications (Intranet software)
• Highly-interactive applications (drawing tools, image editing, code editors)
• Games, emulators, visualizations

Furthermore, we can also define various axes that we can use to score a web-based experience. Those might be requirements or restrictions:

• Loading time
• Network load & size
• Local memory & CPU usage
• Interface complexity
• Content size
• Interface performance
• Need for control over presentation
• Interactivity
• Browser independence
• Viewport-size adaptability
• Offline functionality
• Security

It's obvious how different categories of web-based experiences will have wildly different scores on those various axes.

Compare how intranet applications...

..look against static sites:

And that's without even going to the the "hidden" axes of budget, developer productivity, build complexity, maintainability. These are just for what the end-user sees.

Of course, they are related -- everything has a cost. The classic law of software engineering applies to web development too:

Featureful, bug-free, low cost -- pick two.

It's clear that even the largest companies in the field right now cannot deliver an across the board 10/10 experience, and that's with an army of engineers or even with their own browsers. To try and do so in a startup or an IT environment is a recipe for frustration, or worse.

The web in its current state & form forces us to prioritize various axes over others.

If you're developing an intranet application you can afford including a huge widget library if it increases your productivity, even if it adds 1.5MB to your page weight and 6 seconds to your loading time.

On the other hand, your news site should better be snappy and light, even if that means you agonizing over Javascript parse times, HTTP/2 and invalidating the layout.

The broader community does itself a disservice by not acknowledging the tradeoffs when discussing the relative merits of web technologies. Both experienced and junior developers that approach web development are bombarded by contradicting arguments on different approaches, because there is a tacit agreement that there is a "one-size-fits-all" approach to web development.

Going forward

Define up-front the priorities of any web development project. For measurable elements, you can define easily-testable limits. For others, you have to resort to manual testing. Compare those priorities to your budget or deadline, and see if some are unrealistic.

When evaluating technologies, think hard if they are the right fit for your specific needs, instead of selecting them because they have "momentum" or are used by big companies. ³

Finally, when you're writing a blog post, a comment or you are open-sourcing your own micro (or mega) framework, use a qualifier! Don't lead people astray by implicitly claiming that your approach might work for them.

Thanks for reading. Submit / Comment on Hacker News

You might want to follow me on Twitter.

1. I have been working from home for more than 7 years, and I needed to manage my time more aggressively.
2. I have been enjoying my physical life much more, with a small circle of good friends, a partner, and a dog.
3. As my knowledge grew, I have become more humble and self-conscious. A lot of things now seem obvious so I don't feel the need to share them.
4. I am a hobbyist by nature, and I drifted to other fun activities that all require time and devotion (all of them were of a non-digital nature).
5. I have become wary of sharing personal thoughts on the Internet.

While I am not the regretting type, there's a few things I have missed all this time:

• Exchanging thoughts, views and experiences both within the broader community but also within a small circle of people I have met and value their opinions.
• Putting on my "teaching" hat and engaging deeply with a language, framework or programming approach, in the way that only preparing a presentation or tutorial provides.
• Enjoying the buzzing feeling that you're a member of an open-source community, you've met interesting people online and you're going to physically meet them in a conference.

So with that in mind, it's time to reboot my online presence. I'll see you soon.

Get started

Install ember using Ember CLI -- Ember is similar to Django by making a lot of decisions for you

Create a new project by doing ember new mysite -- Like Django, Ember has helper tools to kickstart a project. This will take some time, because npm will go and download half the internet for you.

The development server

ember serve will start a nice auto-reloading server. Note when you install a new add-on you have to manually restart it. --The similarities with Django continue

Creating the Polls app

Ember doesn't really have the notion of separate apps, so we'll skip this.

Write your first view

Here's where the similarities start to break down. Django, being a server-side framework, has a relatively simple architecture:

• Request comes in
• Passes through some middleware
• Use the URL to try to find a handler (could be a function or a class)
• The handler processes the request, returns a Response
• Middleware
• Response gets served to the client.

Ember on the other hand, being client-side has a more desktop app feel to it, in the sense that there's a lot of (unavoidable, to some extent) state going on (and a run loop, somewhere deep down). Ember tries to organize this state so it's easier to reason about:

• The app loads (once)
• Use the URL to extract a Route
• Load the route, load its model, setup the (singleton) controller, render the template
• Process events that update state, refresh the UI to reflect new state
• Repeat previous step (and transition/redirect to new Routes as needed)

The closest thing therefore to Django's view is a Route.

Note: Ember tries (too hard, IMO) to go "convention-over-configuration" so at some point it does seem like magic. When you start an empty project, and visit the page as served by ember serve in the browser, you'll see a page saying "Welcome to Ember". It follows that a Route must be there somewhere, but the routes directory is empty. Ember will often "auto-generate" missing classes for you, and here it auto generated the application Route which is the "root" of your application. Indeed if you see the templates folder, you'll find application.hbs which is what gets rendered. You can ember generate route application to make this explicit.

To follow the Django tutorial, we need to create a Route under /polls. This is generated for us easily by doing ember g route polls. Note: Ember also is fussy about naming things. Stick to plural nouns with the routes.

Edit the templates/polls.hbs template and open /polls in the browser. You should see the contents of your template. Observe how as you edit the template the page auto reloads.

part 2

Database setup

Obviously Ember.js doesn't have a database component. Instead it has Ember Data which is an "ORM-like" component. As in Django, it is completely optional and you could use whatever you want in there.

The idea is that you have a client-side Store that you query for records. The store delegates the actual work to an Adapter that connects to your actual back-end. It also maintains its own cache (which you can bypass by doing explicit reloads or background reloads).

Note: Here's another big deviation from Django (or any server-side framework). Because of the split concerns, there's going to be duplication of concerns. You do client-side validation, then you have to do server-side validation. You save a record on the client, and you save it on the server. Any error on the server should be propagated to the client, and any background state change in the server should be displayed in the client as well. It's a mess that I believe still hasn't been resolved cleanly. The most promising area there is a client-side framework that integrates tightly with a server-side backend (so probably something in Node.js) but I'm not familiar with the landscape to know if this is going to happen.

I've documented how to setup a Django-specific backend for the built-in JSONAPIAdapter of Ember.js. You should probably follow that guide and come back here when the API is working.

Let's create the models by doing ember g model question and ember g model choice -- routes are plural, models are singular

Edit the two new files:

// models/question.js
export default DS.Model.extend({  
  questionText: DS.attr('string'),
  pubDate: DS.attr('date'),
  choices: DS.hasMany('choice')
});

// models/choice.js
export default DS.Model.extend({  
    choiceText: DS.attr('string'),
    votes: DS.attr('number'),
    question: DS.belongsTo('question')
});

Note: Javascript/Ember convention dictates camelCase for fields.

Obviously there is no database involved so far, but you can install the Ember Inspector Google Chrome extension to get a console of sorts.

Go to routes, find any route, click on the $E and you will have it in the console under $E.

var q = $E.store.createRecord('question');  
q.set("questionName", "What is your name?");  
q.set("pubDate", new Date());  
q.toString(); // UGLY!  
q.toJSON(); // observe how Date is serialized to a string.  

Note: You must use these clunky get/set calls instead of doing normal JS property access. The worst thing is that proper property access will still work but you will get weird bugs as soon as you use other helpful Ember features like computed properties.

The toString produces some ugly stuff. You can either override in the model entirely, or add this:

toStringExtension: function() {  
    return this.get('questionText');
}

Now the toString will have a last component which can help you identify your records a bit better.

Let's add the wasPublishedRecently method too:

  wasPublishedRecently: function() {
      var p = this.get('pubDate');
      var n = new Date();
      var diffInMillis = n - p;
      var dayInMillis = 24 * 60 * 60 * 1000;
      return diffInMillis > dayInMillis;
  }

Let's use the console to add a couple of choices:

var c1 = $E.store.createRecord('choice');  
c1.set("choiceText", "Brave Sir Robin");  
c1.set("votes", 1);  
c1.set("question", q); //from our previous session

var c2 = $E.store.createRecord('choice');  
c2.set("choiceText", "Lancelot the Pure");  
c2.set("votes", 2);  
c2.set("question", q); //from our previous session


q.get("choices").then(function (c){console.log(c.length)}); // prints 2

c2.deleteRecord(); //mark as deleted, don't push to backend yet

q.get("choices").then(function (c){console.log(c.length)}); // prints 1

Note: While Django relations are synchronous, Ember Data relationships are async, meaning when you do q.get("choices") you get a promise back that you have to attach a callback on. This is annoying as well, because the attributes themselves are not promises so you will have to use both ways of accessing a model in your code. Templates, however, natively support this async nature of relations.

Introducing the Django Admin

Here's where ember completely lacks anything like the Django Admin. There might be add-ons out there but nothing really complete. Good thing you can keep using the native Django admin if you use a Django back-end!

Part 3, creating the public interface

The Django tutorial creates 3 similar views - show a question, see results, vote on a question. They are all mapped under a dynamic "question_id". For ember, we will create those as routes:

ember g route polls/view --path /:question_id  
ember g route polls/vote --path /:question_id/vote  
ember g route polls/results --path /:question_id/results  

Because of Ember's magic and a side effect of how "nested routes" work, we are going to need to also explicitly generate a previously auto-generated Route: the index:

ember g route polls/index --path /

So our router.js file should look like this:

// router.js
import Ember from 'ember';  
import config from './config/environment';

const Router = Ember.Router.extend({  
  location: config.locationType
});

Router.map(function() {  
  this.route('polls', function() {
    this.route('index', {
        path: '/'
    });
    this.route('view', {
      path: '/:question_id'
    });
    this.route('vote', {
      path: '/:question_id/vote'
    });
    this.route('results', {
      path: '/:question_id/results'
    });

  });
});

export default Router;

It is conceptually similar to the Django urls.py: you define a parent namespace, polls and then inside you created nested paths, with some dynamic element (:question_id) in particular.

Let's go and edit the auto-generated templates and try to navigate some urls:

<!-- templates/polls.hbs -->  
Polls namespace  
{{outlet}}

<!-- templates/polls/index.hbs -->  
INDEX  

<!-- templates/polls/view.hbs -->  
VIEW  

<!-- templates/polls/vote.hbs -->  
VOTE  

<!-- templates/polls/results.hbs -->  
RESULTS  

Then try to navigate to /polls, /polls/1, /polls/1/vote, and /polls/1/results. You should see "Polls namespace INDEX" and so on. This means that we got our URL handling correct, and for our polls namespace we now have 4 different Routes we can customize.

Writing views that actually do something

Ember Routes have a special model method that you use to get a model and pass it to the template for rendering. Ember does some introspection to make this appear magic, but I don't like magic so I prefer to define this explicitly. Let's first do the index:

// routes/polls/index.js
export default Ember.Route.extend({  
    model() {
        return this.store.findAll('question');
    }
});

<!-- templates/polls/index.hbs -->

{{#if model}}
    <ul>
{{# each model as |question| }}
        <li>
{{#link-to "polls.view" question.id }} {{ question.questionText }} {{/link-to}}
</li>  
{{/each}}
    </ul>
{{else}}
<p>No polls are available.</p>  
{{/if}}

Django here constructs link manually (although it replaces that with the url template tag later on). You could do the same in Ember but this would incur a full app reload, so you instead use the link-to helper, passing it the name of the Route (polls.view) and the parameters to load a model question.id.

Let's also update the polls.view route:

// routes/polls/view.js
export default Ember.Route.extend({  
    model(params) {
        return this.store.findRecord('question', params.question_id);
    }
});

<!-- templates/polls/view.hbs -->

<h1>{{ model.questionText }}</h1>  
<ul>  
{{#each model.choices as |choice| }}
    <li>{{ choice.choiceText }}</li>
{{/each}}
</ul>

As you can see, the fact that model.choices is an async call doesn't matter as the templates are smart enough to update when the promise actually resolves.

Part 4, allowing visitors to vote

The Django tutorial here defines a form to allow to send data back to the server. In our case we don't really need that, but instead we'll use "actions". Edit the view route:

<!-- templates/polls/view.hbs -->  
    <li>{{ choice.choiceText }} -- {{choices.votes}} vote(s) <button {{action "vote" choice}}>Vote!</button></li>

// routes/polls/view.js
export default Ember.Route.extend({  
    model(params) {
        return this.store.findRecord('question', params.question_id);
    },
    actions: {
        vote(choice) {
            console.log("VOTING");
            console.log(choice.toJSON());
            choice.incrementProperty('votes');
            choice.save();
        }
    }

});

Now how everything updates almost immediately - however the results are also saved in the backend as well. Try editing the various records in Django admin and updating the page to see how everything is connected.

Welp, it seems we don't need the other routes at all! This is something that will happen a lot of times when moving from "server-side" to "client-side" thinking. In Django, you had to define URLs and views for pretty much every action you had to do. In Ember, you define URLs only if conceptually the view changes enough to warrant a new URL (or if you can imagine a visitor would like to bookmark this specific page/view for later).

Go ahead and delete the previously defined routes:

ember destroy route polls/vote  
ember destroy route polls/results  

Part 5, testing

Just visit /tests to run your automatically generated tests. I am not well versed on the approaches to testing in Ember, but again batteries are included so you could just dive in writing the tests in the pre-generated files Ember CLI gives you.

I have had good luck using the Django Rest Framework JSON API library on top of Django Rest Framework.

This post will try to document how to get things working and forget about your back-end for a while. I'm going to follow the Django Tutorial schema, with an important addition of adding a related_name to the ForeignKey field.

from django.db import models


class Question(models.Model):  
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')


class Choice(models.Model):  
    question = models.ForeignKey(Question, on_delete=models.CASCADE, related_name="choices")
    choice_text = models.CharField(max_length=200)
    votes = models.IntegerField(default=0)

Do the usual makemigrations and migrate then also register the models in the Admin.

Install Django Rest Framework and the aforementioned JSON API adapter. Also install django-filter. In the settings.py, make sure you add rest_framework.

pip install django-rest-framework  
pip install git+https://github.com/django-json-api/django-rest-framework-json-api.git@develop --upgrade # FIXME once 2.0 is properly released it should be enough  
pip install django-filter  

Now configure the DRF like so:

REST_FRAMEWORK = {  
    #'PAGE_SIZE': 10,
    # Use Django's standard `django.contrib.auth` permissions,
    # or allow read-only access for unauthenticated users.
# FIXME do permission and authentication as you see fit.
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.AllowAny'
    ],
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'rest_framework.authentication.BasicAuthentication',
        #'rest_framework.authentication.SessionAuthentication',
    ],
    'EXCEPTION_HANDLER': 'rest_framework_json_api.exceptions.exception_handler',
    'DEFAULT_PAGINATION_CLASS':
        'rest_framework_json_api.pagination.PageNumberPagination',
    'DEFAULT_PARSER_CLASSES': (
        'rest_framework_json_api.parsers.JSONParser',
        'rest_framework.parsers.FormParser',
        'rest_framework.parsers.MultiPartParser'
    ),
    'DEFAULT_RENDERER_CLASSES': (
        'rest_framework_json_api.renderers.JSONRenderer',
        'rest_framework.renderers.BrowsableAPIRenderer',
    ),
    'DEFAULT_METADATA_CLASS': 'rest_framework_json_api.metadata.JSONAPIMetadata',

# this is optional but very useful
    'DEFAULT_FILTER_BACKENDS': ('rest_framework.filters.DjangoFilterBackend',),

}


APPEND_SLASH=False  
JSON_API_FORMAT_KEYS = 'dasherize'  
JSON_API_FORMAT_RELATION_KEYS = 'dasherize'  
JSON_API_PLURALIZE_RELATION_TYPE = True

Now create a new rest.py file:

from .models import Question, Choice  
from rest_framework_json_api import serializers, relations  
from rest_framework import viewsets, views, response

import django_filters  
from rest_framework import filters

class QuestionSerializer(serializers.ModelSerializer):  
    choices = relations.ResourceRelatedField(read_only=True, many=True)
    class Meta:
        model = Question

class ChoiceSerializer(serializers.ModelSerializer):  
    class Meta:
        model = Choice


class QuestionViewSet(viewsets.ModelViewSet):  
    queryset = Question.objects.all()
    serializer_class = QuestionSerializer

class ChoiceViewSet(viewsets.ModelViewSet):  
    queryset = Choice.objects.all()
    serializer_class = ChoiceSerializer

# this is should plural and dasherized names
ROUTES = {  
    'questions': QuestionViewSet,
    'choices': ChoiceViewSet
}

Then in the global urls.py:

from rest_framework import routers  
router = routers.DefaultRouter(trailing_slash=False)  
from polls.rest import ROUTES  
for key, viewset in ROUTES.items():  
    router.register(key, viewset)

urlpatterns = [  
    url(r'^admin/', admin.site.urls),
    url(r'^api/v1/', include(router.urls)),
    url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework'))
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

The only Ember change you have to do is generate an application adapter with this content:

import DS from 'ember-data';

export default DS.JSONAPIAdapter.extend({  
    namespace: 'api/v1',
    isInvalid(status, headers, payload) {
        return status === 422 || status === 400;
    },
});

Now run the Django dev server, and don't forget to use ember serve --proxy http://127.0.0.1:8000 to ensure your Ember app can easily make Ajax requests there.

The usual approach nowadays is to use WebSockets for real-time communication between browser clients and web servers. The usual way to do that would be to use a server capable of handling many concurrent connections, and use a message bus from the WSGI app to communicate to that service. That is a lot of moving parts.

In my use case where I build a lot of intranet applications, deploying and maintaining all this infrastructure is a very big burden, so the result is usually to not even explore this kind of functionality.

However, given that I deploy on Twisted, I wanted to explore what kind of cool things I could build on it.

Enter SSE

Server-Sent Events aren't that new - they have just been shadowed by WebSockets. They are a simple data format that is send from the server to the client via a plain HTTP connection. The Javascript API is quite simple, and it even handles retries for you. It's compatible with a lot of recent browsers, but I haven't really done a lot of research on it.

Sample Code

Here is a very simple WSGI app (using bottle.py). It just has a form and a form POST handler.

And a basic twisted server to run it:

And a SSE-savvy twisted.web resource:

And a very simple index.html

How it works

The WSGI app just calls some Python code. Through crochet we ensure that it gets back a useful result (though in this case, we just throw it away). We use a plain POST to send data to the server. Converting that to an AJAX request is left as an exercise to the reader. The SSE handler is a singleton that keeps track of all the listeners that are connected to it, and broadcasts messages to it.

Does it scale?

It should! I have no experience running Twisted Web under heavy load but it's more than enough for intranet-style apps (even when I have 50 machines hitting some API endpoints quite frequently). If someone wants to run some testing, please get in touch.

What's next?

I would like to make this a bit more reusable, with some better discovery than the current "inject a global function into the namespace". Also, Django integration is something I'd like to investigate. And why not try if the same approach can be extended to web sockets as well?