Don't Repeat Yourself, Use Metaprogramming

Dec 27, 2017 · 703 words · 4 minutes read juliametaprogrammingpackages

While working on AlphaVantage.jl I found myself breaking one of the fundamental software development principles: don’t repeat yourself (DRY).

The reason I was repeating myself is because many of the functions I was writing took in the same set of arguments (cryptocurrency, exchange market, etc.). Subsequently these functions were largely doing the exact same thing: checking a set of arguments and passing them to the API request.

Take the digital cryptocurrencies for example. The Alpha Vantage API supports four different “functions” for getting digital currency data: digital_currency_intraday, digital_currency_daily, digital_currency_weekly, and digital_currency_monthly. The only difference between these are functions are the interval between observations.

After awhile I became frustrated. My code worked fine but it was bothersome that it was so needlessly repetitive. But then I remembered metaprogramming!

Metaprogramming is code (a program) that writes more code (another program). For example, you can use metaprogramming to programmatically write functions given a set of arguments. Under the hood, Julia represents its code as a data structure. This data can be edited programmatically.

Writing functions using metaprogramming: AlphaVantage.jl

Returning to the example above, the Alpha Vantage APIs are grouped into four categories: Stock Time Series Data, Physical and Digital/Crypto Currencies, Stock Technical Indicators, and Sector Performances. However, there is actually only a single endpoint that accepts a query parameter, function, that determines actual data request. This function also determines which other query parameters are required.

Because different data requests may require different query parameters there is an AlphaVantage.jl function for every Alpha Vantage function. However many functions share the same arguments. For example, the time_series_daily function accepts the same arguments as time_series_daily_adjusted and time_series_weekly.

It would be cumbersome and inefficient to define each function manually. Fortunately this is easy with metaprogramming – generating the functions through code.

for func in (:intraday, :daily, :weekly, :monthly)
    x = "digital_currency_$(func)"
    fname = Symbol(x)
    @eval begin
        function ($fname)(symbol::String; market::String="USD", outputsize::String="compact", datatype::String="json")
            @argcheck in(outputsize, ["compact", "full"])
            @argcheck in(datatype, ["json", "csv"])
            uri = "$(alphavantage_api)query?function=" *
                uppercase($x) *
                "&symbol=$symbol&market=$market&outputsize=$outputsize&datatype=$datatype&apikey=" *
            data = _get_request(uri)
            return _parse_response(data, datatype)
        export $fname

This code loops through an array of symbols (representing the code as data). For each symbol, the code creates a function that accepts several arguments and makes an API request. The trick here is that the function query parameter is assigned a value using the symbol. Within the @eval block the symbols are evaluated to their corresponding representation (e.g. when $func is intraday then $fname becomes digital_currency_intraday).

Benchmarking a package using metaprogramming: AdventOfCode2017.jl

Another example of metaprogramming in Julia is in the AdventOfCode2017.jl README. Advent of Code is a collection of small programming problems. New problems are released each year during the first 25 days of December.

AdventOfCode2017.jl contains my Julia solutions to the puzzles. The README is generated from README.jl using Weave–a package for converting Julia to markdown (similar to knitr or Sweave in R)–and contains performance benchmarks for each puzzle solution.

The first few lines of a non-metaprogramming approach might look something like this:

@time puzzle01()
@time puzzle02()
@time puzzle03()
@time puzzle04()

In this case the code violates the DRY principle. Fortunately metaprogramming provides tools to express this more succinctly. The code from README.jl is below:

table = ["Day" "Elapsed (s)" "Allocated (MiB)" "Garbage Collection (s)"]
puzzles = filter(x-> x≠ :AdventOfCode2017, names(AdventOfCode2017, false))
for puzzle in puzzles
    m = match(r"[0-9]+", String(puzzle))
    day = string(parse(Int, m.match))
    @eval begin
        val, t, bytes, gctime, memallocs = @timed $puzzle()
        t = round(t, 4)
        kib = round(bytes/1024^2, 4)
        gctime = round(gctime, 4)
        row = [$day t kib gctime]
        table = vcat(table, row)

Instead of just measuring performance of each function using @time, I decided to store results in a table. The names function returns an array of symbols for each function exported by the package. Then for each symbol, I measure the performance of the function and append that row to the table.

Metaprogramming can help you stay DRY

Metaprogramming is code (a program) that writes more code (another program). Under the hood, Julia represents its code as a data structure and this can be edited programmatically. For example, you can use metaprogramming to write functions that take a set of similar arguments or evaluate many functions to measure their performance.