Don't Repeat Yourself, Use Metaprogramming
Dec 27, 2017 · 703 words · 4 minutes read
While working on AlphaVantage.jl I found myself breaking one of the fundamental software development principles: don’t repeat yourself (DRY).
The reason I was repeating myself is because many of the functions I was writing took in the same set of arguments (cryptocurrency, exchange market, etc.). Subsequently these functions were largely doing the exact same thing: checking a set of arguments and passing them to the API request.
Take the digital cryptocurrencies for example. The Alpha Vantage API supports four different “functions” for getting digital currency data:
digital_currency_monthly. The only difference between these are functions are the interval between observations.
After awhile I became frustrated. My code worked fine but it was bothersome that it was so needlessly repetitive. But then I remembered metaprogramming!
Metaprogramming is code (a program) that writes more code (another program). For example, you can use metaprogramming to programmatically write functions given a set of arguments. Under the hood, Julia represents its code as a data structure. This data can be edited programmatically.
Writing functions using metaprogramming: AlphaVantage.jl
Returning to the example above, the Alpha Vantage APIs are grouped into four categories: Stock Time Series Data, Physical and Digital/Crypto Currencies, Stock Technical Indicators, and Sector Performances. However, there is actually only a single endpoint that accepts a query parameter,
function, that determines actual data request. This function also determines which other query parameters are required.
Because different data requests may require different query parameters there is an AlphaVantage.jl function for every Alpha Vantage function. However many functions share the same arguments. For example, the
time_series_daily function accepts the same arguments as
It would be cumbersome and inefficient to define each function manually. Fortunately this is easy with metaprogramming – generating the functions through code.
for func in (:intraday, :daily, :weekly, :monthly) x = "digital_currency_$(func)" fname = Symbol(x) @eval begin function ($fname)(symbol::String; market::String="USD", outputsize::String="compact", datatype::String="json") @argcheck in(outputsize, ["compact", "full"]) @argcheck in(datatype, ["json", "csv"]) uri = "$(alphavantage_api)query?function=" * uppercase($x) * "&symbol=$symbol&market=$market&outputsize=$outputsize&datatype=$datatype&apikey=" * ENV["ALPHA_VANTAGE_API_KEY"] data = _get_request(uri) return _parse_response(data, datatype) end export $fname end end
This code loops through an array of symbols (representing the code as data). For each symbol, the code creates a function that accepts several arguments and makes an API request. The trick here is that the
function query parameter is assigned a value using the symbol. Within the
@eval block the symbols are evaluated to their corresponding representation (e.g. when
Benchmarking a package using metaprogramming: AdventOfCode2017.jl
Another example of metaprogramming in Julia is in the AdventOfCode2017.jl README. Advent of Code is a collection of small programming problems. New problems are released each year during the first 25 days of December.
AdventOfCode2017.jl contains my Julia solutions to the puzzles. The README is generated from README.jl using Weave–a package for converting Julia to markdown (similar to knitr or Sweave in R)–and contains performance benchmarks for each puzzle solution.
The first few lines of a non-metaprogramming approach might look something like this:
@time puzzle01() @time puzzle02() @time puzzle03() @time puzzle04() ...
In this case the code violates the DRY principle. Fortunately metaprogramming provides tools to express this more succinctly. The code from README.jl is below:
table = ["Day" "Elapsed (s)" "Allocated (MiB)" "Garbage Collection (s)"] puzzles = filter(x-> x≠ :AdventOfCode2017, names(AdventOfCode2017, false)) for puzzle in puzzles m = match(r"[0-9]+", String(puzzle)) day = string(parse(Int, m.match)) @eval begin val, t, bytes, gctime, memallocs = @timed $puzzle() t = round(t, 4) kib = round(bytes/1024^2, 4) gctime = round(gctime, 4) row = [$day t kib gctime] table = vcat(table, row) end end
Instead of just measuring performance of each function using
@time, I decided to store results in a table. The
names function returns an array of symbols for each function exported by the package. Then for each symbol, I measure the performance of the function and append that row to the table.
Metaprogramming can help you stay DRY
Metaprogramming is code (a program) that writes more code (another program). Under the hood, Julia represents its code as a data structure and this can be edited programmatically. For example, you can use metaprogramming to write functions that take a set of similar arguments or evaluate many functions to measure their performance.