Of Herbs and Stewed Rabbit

Clean up a messy namespace with cider and clj-refactor

Sun, 6 Dec 2015 00:00:00 +0000

On clojureX this year we almost had a panel discussion on editors; it was cancelled at the last minute unfortunately. So I thought why not publish the longer version of the screencast I prepared demonstrating how to use some clj-refactor features in Emacs to clean up a messy namespace.

I highly recommend Parens of the dead to everybody by the way. The series is real joy to watch and demonstrates many things, among them is how to use Emacs with cider and clj-refactor to work with clojure code in a very effective way. On the other hand Magnar uses a certain bunch of clj-refactor features possibly because he builds his project up from scratch. In my brief demo/screencast I'd like to rather demo some features which are very helpful in an other typical refactoring scenario: when you want to clean up an already existing (legacy?!) namespace.

Also both cider and clj-refactor has so many features now that discoverability is an issue. This screencast and the demonstrated use of hydras tries to offer some remedies for that too.

`ml` move to let

Move the current form to the closest let. More details here. This is an elisp only feature meaning that you don't even need a running REPL to use it.

`rs` rename (local) symbol

Rename all occurrences of a local symbol. More details here. This is a feature which uses the analyzer to figure out the location of the local symbol therefore you need a running REPL with refactor-nrepl middleware added to it.

`is` inline symbol

Inline all occurrences of the symbol at point. Can be used both globally and locally. See screencast gif here. Again this is feature needing the analyzer to figure out the location of the occurrences of the symbol you want to inline.

`pf` promote function

Can promote #(foo %) style anonymus function to (fn [arg1] (foo arg1)) or all the way to first level defn or a (fn [arg1] (foo arg1)) style anonymus function into a first level defn. See details here. Again this needs the analyzer.

`ef` extract function

Extract the form at point, or the nearest enclosing form, into a toplevel defn. Newly created function is either private or public, depending on cljr-favor-private-functions. See a screencast here. This needs the analyzer as well to figure out which locally bound vars the s-expression uses that need to be turned into arguments in the new function.

Java8 stream API vs Clojure fizzbuzzed

Fri, 6 Feb 2015 00:00:00 +0000

Lately I came across a vanilla java solution for the good old fizzbuzz problem. This gave me the idea to try (finally) the java8 stream API: fizzbuzz seemed a small but fitting problem. And while at it why not code up a solution in clojure too so I can compare the two. This post documents the one afternoon adventure.

Let's also spice fizzbuzz up a bit with adding two extra requirements:

- if the number in question has 3 in it we print 'bingo'
- at the end we print a kind of statistics where we show the number of occurrences of 'Fizz', 'Buzz', 'FizzBuzz', 'Bingo' and not changed numbers

Please note that although the first one is just an other condition it is also an other type of condition not related to any modulo of or any arithmetic operation with the given number. The second is interesting too because the solution needs say something about the transformed list. A bit of state might need to be maintened while doing the transformation, right? ;)

So first of all as a starting point let's see the vanilla solution in java:

public class FizzBuzz {

    public void fizzBuzz(int rangeStart, int rangeEnd, PrintStream out) {

        Map<String, AtomicInteger> statMap = new HashMap<String, AtomicInteger>();

        List<String> transformedNumbers = new ArrayList<String>();
        for(int i = rangeStart; i <= rangeEnd; i++){
            String elem = "Integer";
            String iAsString = String.valueOf(i);
            if (iAsString.contains("3")) {
                elem = "Bingo";
            } else if (i % 15 == 0){
                elem = "FizzBuzz";
            } else if(i % 3 == 0){
                elem = "Fizz";
            } else if(i % 5 == 0){
                elem = "Buzz";
            }
            transformedNumbers.add(("Integer".equals(elem))? iAsString : elem);
            statMap.computeIfAbsent(elem, k -> new AtomicInteger(0));
            statMap.get(elem).getAndIncrement();
        }

        out.println(transformedNumbers);
        out.println();
        out.println(statMap);
    }
}

Pretty naive approach but hey-ho we just started. Let's see the (kinda) equivalent in clojure:

(defn fizzbuzz [start end]
  (letfn [(calc-stats [cmap elem] (let [key (if (string? elem) (keyword elem) :integer)]
                                    (update-in cmap [key] #(if % (inc %) 1))))]
    (let [fizzbuzzed (->> end
                          inc
                          (range start)
                          (map #(cond
                                 (.contains (str %) "3") "bingo"
                                 (= (mod % 15) 0) "fizzbuzz"
                                 (= (mod % 3) 0) "fizz"
                                 (= (mod % 5) 0) "buzz"
                                 :default %)))
          stats (reduce calc-stats {} fizzbuzzed)]
      [fizzbuzzed stats])))

Just to point out the obvious differences in clojure you don't need to iterate through the list when fizzbuzzing it: just give a recipe in the anonymous function (line 7.) how to fizzbuzz a given element. In the java code you iterate through all the elements and update a bit of local state as a side effect to get the new list and the statistics. The usage of a REPL is obvious too when you look at the clojure code. You don't really need to worry about how to print your results while developing, playing with the code: the REPL will take care of it. My last point is types: in java you have to convert the original int to string while clojure being a dynamic language you are fine with a list with partly string and partly number elements.

In clojure after you had done the fizzbuzzing you can easily reduce the new list into the stats. That is a bit awkward though, you might want a solution which does the stats and the fizzbuzzing in one step. Also using the cond is equivalent with the java if/else-if/else construct but not really nice either. So bit of work on these points:

(defn- fizzbuzz&stats [list&stats n]
  (let [mod-pred (fn [n word] #(when (= (mod % n) 0) word))
        safe-inc (fn [n] (if n (inc n) 1))]
    (if-let [s (some #(% n) [#(when (.contains (str %) "3") "bingo")
                             (mod-pred 15 "fizzbuzz")
                             (mod-pred 3 "fizz")
                             (mod-pred 5 "buzz")])]
      (-> list&stats
          (assoc :list (concat (:list list&stats) (seq [s])))
          (update-in [(keyword s)] safe-inc))
      (-> list&stats
          (assoc :list (concat (:list list&stats) (seq [n])))
          (update-in [:integer] safe-inc)))))

(defn fizzbuzz-enhanced [start end]
  (->> end
       inc
       (range start)
       (reduce fizzbuzz&stats {})))

The fizzbuzzed list and the stats is constructed in one reduce. Also eliminated the cond and use some with a vector of predicates: the first which applies wins and returns the fizzbuzz string for the element. If there is no winner stick with the original number. And of course either case add the element after fizzbuzzing to the result list inside the result hash. Reduce takes care of passing a fresh version of it to the next round.

Now it is time to get to the cake: code up something similar using the java8 stream API. I am sure there are other nice, very likely better solutions, don't hesitate to ping them to me on twitter ;)

public void fizzBuzz(int rangeStart, int rangeEnd, PrintStream out) {

    String[] fizzbuzzed = rangeClosed(rangeStart, rangeEnd).mapToObj(FizzBuzz::intToFizzBuzz).toArray(String[]::new);
    System.out.println("start ---");
    System.out.println(Arrays.toString(fizzbuzzed));
    System.out.println("end   ---");
    System.out.println("stats start ---");
    System.out.print(Arrays.stream(fizzbuzzed).collect(Stats::new, Stats::accept, Stats::combine));
    System.out.println("stats end   ---");
}

class Stats implements Consumer<String> {

    private int lucky, fizz, buzz, fizzbuzz, integer = 0;

    public void accept(String elem) {
        if ("lucky".equals(elem)) lucky++;
        else if ("fizz".equals(elem)) fizz++;
        else if ("buzz".equals(elem)) buzz++;
        else if ("fizzbuzz".equals(elem))
            fizzbuzz++;
        else
            integer++;
    }

    public void combine(Stats other) {
        lucky += other.lucky;
        fizz += other.fizz;
        buzz += other.buzz;
        fizzbuzz += other.fizzbuzz;
        integer += other.integer;
    }

    public String toString() {
        return "lucky: " + lucky +
            "\n fizz: " + fizz +
            "\n buzz: " + buzz +
            "\n fizzbuzz: " + fizzbuzz +
            "\n integer: " + integer;
    }
}

public static String intToFizzBuzz(int nr) {
    if (String.valueOf(nr).contains("3")) return "lucky";
    if (nr > 0 && nr % (3 * 5) == 0) return "fizzbuzz";
    if (nr > 0 && nr % 3 == 0) return "fizz";
    if (nr > 0 && nr % 5 == 0) return "buzz";
    return String.valueOf(nr);
}

Couple of notes. This solution is rather equivalent of the first iteration of clojure as it first creates the fizzbuzzed list and the reduces/consumes it to calculate the statistics. That is a bit awkwardly though as a stream which you run a terminal operation against you can't consume again. Although this makes sense it also means that the stream needs to be recreated for the reduction into the statistics because reducing it for printing is also a terminal operation. In Clojure there is no such problem as all the collections are persistent therefore you can easily reuse them. See the first clojure iteration code: the fizbuzzed list can be safely returned and reduced into the statistics.

That said the stream API felt something I would not mind to work with on a more realistic problem. And mainly because of this API I tend to agree with Sean Corfield:

I think I can safely say that I would no longer rather fall on a sword than program in Java – as long as it’s Java 8!

[from here]

As wrapping up I mention two other fizzbuzz solutions. One I came across on the rosette code and found it really elegant:

(map #(let [s (str %2 %3) ] (if (seq s) s (inc %)))
        (range)
        (cycle ["" "" "Fizz"])
        (cycle ["" "" "" "" "Buzz"]))

I think this is my favourite as it does not even use mod just simply exploits positions in lists and the fact that map can go through multiple collections. Also laziness shines through this example: just mind blowingly awesome.

The other solution shows how much fizzbuzz is a rabbit hole and also well... funny? sad? humbling? ;)

Webapp regression testing for cheap

Fri, 16 Jan 2015 00:00:00 +0000

An automated regression test suite is a very valuable asset, however most of the times it does not come cheap. You need to write your unit and/or functional tests and add them to the regression suite and then, and this is the real pain: maintain them. Almost every single refactoring means some work on the test code too. We've found a way around this at the MailOnline, and get a really valuable regression test suite for cheap. Perhaps it's worth to share the details.

Our regression test suite was not originally created as such. We aimed to solve a different testing problem. Since our project started off as a rewrite project, ie. rewriting the application producing the MailOnline website in clojure -- it was originally implemented in java and spring MVC --, the problem was how to make sure and prove that we are producing the same output. Mind you, even defining what is the same output here is not trivial as we did not care about whitespaces or other not really important, minor differences in the html output. Later we realised that the same idea can be reused to catch regressions and solve other testing problems too. More on this later.

The key idea of solving this was to use enlive for two crucial features:

its ability to turn html into data structures we can work easily with in clojure,
its ability to write selectors to easily filter out parts of html we are interested in.

With this at hand what needed to be done became quite straight forward:

parse the html output for both the legacy and the new application
select some parts of the html from both the legacy's and the new application's output. We can even drill down to certain attributes
preprocess the selected data optionally. For example for an img tag we are only interested in the src and class attributes: the other attributes are not crucial. Or can munge the data a bit, let's say because we have timestamps here and there: they won't be exactly the same ever.
compare the preselected and preprocessed data
produce some human or machine readable output of the comparison

In fact we came up with a protocol defining what we exactly need to do after the html is parsed already:

(ns fe-adm.ab-test.detective)

(defprotocol DetectiveProtocol
  "An investigator for A/B testing. Understands what parts of a page are
   interesting and how to compare those parts"
  (title [self]
    "Descriptive id")

  (extract [self page-html]
    "Extract the interesting part of the page")

  (keyfn [self]
    "Prime elements for use in sort-by")

  (render [self a-html b-html]
    "Print HTML diff")

  (render-unit [self a-html b-html]
    "Print unit compliant xml")

  (diff [self a b]
    "Calculate missing and surplus"))

One simple example of using this protocol for checking divs across the page:

(ns fe-adm.ab-test.div
  (:require [fe-adm.ab-test.detective :refer :all]
            [net.cgrand.enlive-html :as html]
            [fe-adm.ab-test.utils :as ab-utils]))

(defn- ignore-divs [div]
  (empty? (-> div :attrs)))

(defn- munge-divs [div]
  (-> div
      (ab-utils/trim-attrs-str :class)
      (ab-utils/replace-attrs-str :id #"^.*$" "{{some-id}}")))

(defrecord Detective []
  DetectiveProtocol
  (title [self] :div)

  (extract [self html]
    (->> (html/select html [:div])
         (remove ignore-divs)
         (map :attrs)
         (map munge-divs)
         set))

  (keyfn [self]
    (fn [attrs] (vec (sort (dissoc attrs :content)))))

  (render [self a-html b-html]
    (ab-utils/render-diff self a-html b-html))

  (render-unit [self a-html b-html]
    (ab-utils/render-unit-xml self a-html b-html))

  (diff [self a b]
    (clojure.set/difference a b)))

Note that the really interesting part happens in line 18. extract function. All div tags are selected although some are ignored (this part could be done by enhancing the selector actually). Then the value of the class attribute is trimmed: bit of clean up. And since div ids are uuid-s of some sort they are replaced with a constant so they won't fail the test needlessly.

As I said above after going live with the new application and retiring the legacy one we quickly realised that we can reuse the same idea and code for regression testing. We put together a box in production (not actually serving real traffic) getting a build for every single commit and compared its output with our production output. Presto, we have a regression test suite catching possible differences in our output for every single commit. If there is a difference we need to check if the change was intentional. If it was we use our CI tool's test muting feature to mark and silence it. Easy to do as you can see from the render-unit function we are producing unit compliant output so our CI tool nicely listed our passed and failed tests. After a new production release all the test cases will pass naturally: that actually removes the mutes: clean sheet again.

Bigger pieces of work, refactorings frequently done on branches. With this approach it was easy to push a build on our prod test box from the branch and test the produced output against the production output to get some feedback of the state of the refactoring.

Another use case is when we decided to set up another cluster of our application in a new data center. We just run this test suite comparing the output of one of the new nodes against the output of a live node. Again presto: we were able to say when the new cluster was ready to server real traffic.

In retrospective my main take away from this story is how much value you can get for how less if you take the context of your application into consideration when you create your test suites.

Laziness driven changelogs

Mon, 5 Jan 2015 00:00:00 +0000

At the MailOnline we have not been doing microservices exactly but something definitely far from big, monolithic applications. This resulted in quite a few applications across the board. Many written in clojure or in node.js, javascript for the browser, or some other language. We developers take care of releasing our software which is perhaps the right way to go most of the times, using some CI tool and always trying to push for continuous delivery. Nevertheless the time had come when we were asked to provide some human readable information about the versions of the software we release. The request come from multiple sources: devs, devops and technical management all wanted this info. For a short time we could repel the requests by pointing people to our CI tool and/or gitlogs either in the command line or on github which we use to host almost all our projects -- rest is on bitbucket, just to make things even harder to track. On the top of that we use multiple issue tracking systems, yeah I know, don't ask: historical reasons. As these issues have very valuable information on what we are working on they should be referenced in these changelogs too.

When we realised that we can't really escape the changelog, release note request and to track things would actually help ourselves too we had practically to ways to go: either write these changelogs manually or write some code to generate them for us. Luckily almost from the beginning we were fairly good writing commit messages -- I read that some people actually use git hooks to validate commit messages and enforce certain rules: we are not that good but everybody in the group writes a meaningful one liner at least, referencing github or JIRA issues using tags like [wip] or [refactor]. That is fairly good base to build on.

On an other note a long annoyance of mine the release creation on github: why, oh why can't you automatically generate some changelog on github based on your commit messages? I am sure that would not work for everybody who wants to use the release facility on github but it should do it for many projects. So first round: let's do a bit of googling if the tool is already out there. Well... kinda... gitchangelog looks like exactly what we need. But looking at its sample config file: not that sure anymore. So I basically figured writing something with a similar set of features with simpler config, less code and easy to use for any project but even easier for clojure projects would be fun.

Hence happened gargamel which can be used as a leiningen plugin or with its CLI interface. It accepts any git refs to generate a changelog in between or can generate release notes between the two latest occurrence of some tags with a specific format (for example v\d\.\d\.\d). It uses mustache templates to generate practically any output format (tested with html and markdown) and with simple configuration based on regexes it generates custom sections and munges certain objects (like creating links to github/jira issues from issue references) in the commit messages. On the top of that it is able to create changelogs of project dependencies in projects where those dependencies are managed in a bower.json file.

So now we are able to plugin in change log and release note generation into our release process. We have two types of release note generation jobs on our CI tool: manual one where you can provide from and to git refs: this is used to generate changelog for release candidates and publish them in the appropriate chatrooms (we use slack heavily) for affected people to review. And we have an automated release note generation step which runs right after our production release. Integrating gargamel with CI and slack is the real win in this story. On the top of that on those open source projects I am involved in I can use gargamel now to create markdown changelogs for github releases. for example gargamel's own release notes.

Clean clojure files en masse continued

Mon, 24 Mar 2014 00:00:00 +0000

The project clean mentioned at the end of the previous post got integrated into clj-refactor and changed a bit during the process. Just a quicky about the code.

(defun cljr-project-clean ()
  (interactive)
  (when (or (not cljr-project-clean-prompt)
            (yes-or-no-p "Cleaning your project might change many of your clj files. Do you want to proceed?"))
    (dolist (filename (cljr--project-files))
      (when (s-ends-with? "clj" filename)
        (let ((buffer (get-file-buffer filename))
              find-file-p)
          (if buffer
              (set-buffer buffer)
            (setq find-file-p t)
            (find-file filename))
          (ignore-errors (-map 'funcall cljr-project-clean-functions))
          (save-buffer)
          (when find-file-p
            (kill-buffer)))))))

There are a few small changes basically just to make the this more configurable and more comfortable to use.

Line 3: There is a defcustom controlling if clj-refactor should make sure that we want to proceed with a possible destructive change (en masse). Easy to get rid of this: (setq cljr-project-clean-prompt nil) if we know what we are doing.
Line 7: clj-refactor nicely tries to find the buffer first. If it is found it uses it and then won't get killed when all the changes are done (see line 15): there is a good guy. Otherwise we open the file with find-file and kill it when done. Would be interesting to find a way to use temp-buffer and/or temp-file actually.
Line 13: There is an other defcustom to configure the list of functions to run on a given clojure file in the project. By default this is removing unusued requires and sort ns forms but easy to remove one of these or add others, even perhaps some project dependent cleaning functions implemented in the project itself.

Hope that really helps out there to fix some broken windows.

Clean clojure files en masse

Sun, 16 Mar 2014 00:00:00 +0000

clj-refactor is a very lightweight elisp library for Emacs to support every day refactorings for clojure. I've even read that it is an über-paredit: kinda fair. It definitely melds into the editing experience defined by the mix of cider, clojure-mode, paredit well. Adopting it you get loads of nice transformations which help you with your every day clojure coding. There are simpler ones like add require to namespace declaration ar and cycle collection type cc or cycle privacy cp. This latter sounds silly first: why an earth you would not just go to your defn and add that '-' to make it private. But once your muscle memory has remembered cp it is just natural to turn a function private with one easy key combo when you figured out it does not make sense to keep it public -- you don't even need to think about what to press. There are more arcane ones which give the wtf-just-happened-with-my-code experience first but then they start making absolute sense. The threading macro related ones for example: wrap in thread first tf and fully unwind threaded expression ua or the introduce let, expand let, move to let triumvirate. Used in a correct way the latter one helps you to handle let expressions in a very clean way across your code. These are only examples of course, check out the full list on the project's really cleanly written github readme.

Just a side note on the shortcuts. Well, that is the usual Emacs story. People tend to freak out: 'not that many extra key combination I need to learn again'. I find that my muscle memory kicks in very fast and what is even more helpful it is contextual. I don't even try certain key combos when in the repl or when editing a text file. So after a bit of practice you won't need to think how to move the form after the cursor to the already existing let expression.

I also mentioned that clj-refactor is lightweight. It is in the sense that it does not use any code analyzer library or such and it does not enforce you to use cider either. (Cider still can come in handy. For example you can use completion when adding something to the require section in your ns declaration.) It is an emacs-lisp library -- started out as tiny -- which uses some other elisp libraries like paredit, yasnippet, dash.el, s.el, multiple cursors (the good work of Magnar Sveen and others just like clj-refactor itself). So clj-refactor does not really 'understand' your code, but by using paredit it does understand the structure of your clojure files. This with some more elisp foo gives it enough power to be extremely handy: clj-refactor is not so tiny anymore. There are plans to add some code analyzer to the mix but to be honest I would prefer at least to keep that if ever added as an optional feature even with the drawback of falling back to simpler behaviour. Lightweight means fast and speed is extremely important in order to avoid the productivity killer IDE experience.

That leads me to the next group of offered refactorings, the ones which need to be performant because it makes sense to run them against a whole bunch of clojure files in one go. These are the 'clean up', house keeping features:

replace use with :refer :all
sort use, require and import in your ns form
remove unused requires

When working on a sort of larger clojure project or multiple of those or as a member of a team of six, eight plus developers you might find running these against a whole project periodically a really nice thing as it helps fixing the broken window effect. I personally am not really a big fan of coding conventions, rules of how to write code or code style checkers but the number of namespaces which have unused requires is a good indicator of how much the code started to get disorganised. Better to fix those as fast as possible.

Here comes a small elisp function which does the trick:

(defun cleanup-project-clj-files ()
  (interactive)
  (dolist (filename (cljr--project-files))
    (when (s-ends-with? "clj" filename)
      (find-file filename)
      (ignore-errors (cljr-remove-unused-requires))
      (save-buffer)
      (kill-buffer))))

Some quick points about the above code. It uses clj-refactor's own cljr--project-files to find the clojure files. There are other ways but I thought the best is to use this as we are anyway using this library. One caveat that it will only work with leiningen managed projects for now. There are plans in the project as far as I know to eventually improve this. When that is done the above script will start working for any kind of projects. ignore errors is needed because there is an error thrown if the given file does not have an ns declaration (as the project file itself does not). save buffer might also trigger some additional clean up if you some hooks defined on save. For example my Emacs configuration turns all tabs into spaces when I save. Adding the above function to your Emacs is easy: either just copy it into your Emacs init file or save it in a separate file and make sure that file is loaded via your init.el. If that is done you can just open any files in the project and M-x cleanup-project-clj-files.

I saw something very similar originally in Alex Baranosky's Emacs config repo on github and simplified it a bit. Also lots of information in the post comes from Alex and the small team around clj-refactor while chatting with them about my pull requests. A very nice, fun team around a fun project.