Erik Rigtorp

Efficient rolling statistics with NumPy

2011-01-01T00:00:00-05:00

Efficient rolling statistics with NumPy

When working with time series data with NumPy I often find myself needing to compute rolling or moving statistics such as mean and standard deviation. The simplest way compute that is to use a for loop:

def rolling_apply(fun, a, w):
    r = np.empty(a.shape)
    r.fill(np.nan)
    for i in range(w - 1, a.shape[0]):
        r[i] = fun(a[(i-w+1):i+1])
    return r

A loop in Python are however very slow compared to a loop in C code. Fortunately there is a trick to make NumPy perform this looping internally in C code. This is achieved by adding an extra dimension with the same size as the window and an appropriate stride:

def rolling_window(a, window):
    shape = a.shape[:-1] + (a.shape[-1] - window + 1, window)
    strides = a.strides + (a.strides[-1],)
    return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)

Using this function it is easy to calculate for example a rolling mean without looping in Python:

>>> x=np.arange(10).reshape((2,5))
>>> rolling_window(x, 3)
array([[[0, 1, 2], [1, 2, 3], [2, 3, 4]],
       [[5, 6, 7], [6, 7, 8], [7, 8, 9]]])
    
>>> np.mean(rolling_window(x, 3), -1)
array([[ 1.,  2.,  3.],
       [ 6.,  7.,  8.]])

More about the “stride trick”: SegmentAxis, GameOfLifeStrides

Automatically generating Erlang release files

2010-03-17T00:00:00-04:00

Automatically generating Erlang release files

If you’ve ever worked with Erlang and worked with it’s release handling functionality you’ve probably been frustrated by the requirement to explicitly specify version numbers of applications included in the release resource file (.rel file). I understand why it is useful to specify application version explicity when you are make a release for a specific target platform, but in many cases you want to use the version currently installed on your system.

In order to make it easier to create a release file and boot script I created a make target to automatically generate these from an application resource file (.app file):

$(APP).boot: $(APP).rel Makefile
	$(ERL) -pa ebin -noshell +B -eval \
	'case systools:make_script("'$(basename $@)'",[local]) of \
         ok -> halt(0); _ -> halt(1) end.'

$(APP).rel: ebin/$(APP).app Makefile
	$(ERL) -pa ebin -noshell +B -eval \
	"ok = application:load($(basename $@)), \
	 {ok, Apps} = application:get_key($(basename $@), applications), \
	 {ok, F} = file:open("$@", [write]), \
	 io:format(F, \"~p.~n\", [{release, {\"$(basename $@)\", \"$(SRCREV)\"}, \
	 {erts, erlang:system_info(version)}, \
	 lists:map(fun (App) -> application:load(App), \
	 {ok, Vsn} = application:get_key(App, vsn), \
         {App, Vsn} end, Apps ++ [$(basename $@)])}]), \
	 file:close(F), halt(0)."

You have to define $(APP) to be the name of your application.

Using jQuery to display a Yahoo! Pipe

2010-03-14T00:00:00-05:00

Using jQuery to display a Yahoo! Pipe

I found myself wanting to display a feed on a webpage and it turned out to be a peace of cake using Yahoo! Pipes and jQuery:

<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript">
  $(document).ready(function() {
    $.getJSON("http://pipes.yahoo.com/pipes/pipe.run?_id=PIPEID&\
_render=json&_callback=?", 
      function(data) {
        $.each(data.value.items, function(i, item) {
          $('#items').append('<li><a href="' + 
	  item.link + '">' + item.title + '</a></li>');
        });
      });
  });
</script>

<ul id="items">
</ul>

Just replace PIPEID with the id of your pipe.

Erlang Latency Guide

2009-11-20T00:00:00-05:00

Erlang Latency Guide

Introduction

Latency is a tricky subject, sometimes it's not even clear what or how to measure it. I've had the experience of writing a fairly complex system requiring low latencies in Erlang. Fortunately Erlang provides really good baseline performance. Most of the time you simply write your program and it will perform well. There are however a few tricks that can be used to lower the latencies of a specific path in the system. This document describes a few of these tricks.

Yield

Erlang allows you to design efficient concurrent systems without caring how processes are scheduled or how many cores the system is running on. When running Erlang with multiple schedulers (generally one per CPU-core) the runtime will balance the load between the schedulers by migrating processes to starved schedulers. There is no way to bind processes to schedulers or control how processes are migrated between schedulers. This introduces a non-deterministic behavior in the system and makes it hard to control latency.

A common pattern is to have a demultiplexer that receives a message, sends it to some other process/processes and then performs some additional processing on the message:


loop(State) ->
    receive 
        Msg ->
            Pid = lookup_pid(Msg, State),
	    Pid ! Msg,
	    State2 = update_state(Msg, State),
	    loop(State2)
    end.

After the message has been sent the receiving process will be ready to execute, but unless the receiving process is on a different scheduler the demultiplexer will first finish executing. Ideally we would bind the demultiplexer to one scheduler and bind the receiving processes to the other schedulers, but that's not allowed in Erlang.

Erlang provides only one simple, but powerful way to control scheduling: The built-in function (BIF) erlang:yield/0 lets processes voluntarily give up execution and let other processes get a chance to execute.

The demultiplexer pattern can be modified by adding erlang:yield() after sending the message:


loop(State) ->
    receive 
        Msg ->
            Pid = lookup_pid(Msg, State),
	    Pid ! Msg,
            erlang:yield(),
	    State2 = update_state(Msg, State),
	    loop(State2)
    end.

After the message has been sent the demultiplexer will give up execution. If the demultiplexer and the receiver are on the same scheduler the receiver will execute before the demultiplexer finishes executing, if they are on different schedulers they will execute in parallel.

Using the erlang:yield/0 BIF it's possible to control the scheduling of Erlang processes. If used correctly this can reduce the latency in a system.

Network

All network I/O in Erlang is implemented as an Erlang driver. The driver is interfaced by the module prim_inet which in turn is interfaced by the network related modules in the kernel application.

There is a performance issue with the prim_inet:send/2 and prim_inet:recv/2 functions affecting all the network related modules. When calling prim_inet:send/2 or prim_inet:recv/2 the process will do a selective receive. If the process's message queue is long there will be a performance penalty from doing this selective receive.

For receiving there is a simple solution to this problem: use the {active, once} socket option.

A simple selective receive-free TCP receiver:


loop(Sock) ->
    inet:setopts(Sock, [{active, once}]),
    receive
        {tcp, Sock, Data} ->
            loop(Sock);
        {tcp_error, Sock, Reason} ->
            exit(Reason);
        {tcp_closed, Sock} ->
            exit()
    end.

To implement sending without doing a selective receive it is necessary to use the low-level port interface function erlang:port_command/2. Calling erlang:port_command(Sock, Data) on a TCP socket would send the data Data on the socket and return a reference Ref. The socket will reply by sending {inet_reply, Ref, Status} to the process that called erlang:port_command.

A simple selective receive-free TCP writer:


loop(Sock) ->
    receive
        {inet_reply, _, ok} ->
            loop(Sock);
        {inet_reply, _, Status} ->
            exit(Status);
        Msg ->
            try erlang:port_command(Sock, Msg)
            catch error:Error -> exit(Error)
            end,
            loop(Sock)
    end.

Though not Erlang specific it is important to remember to tune the send and receive buffer sizes. If the TCP receive window is full data may be delayed up to one network round trip. For UDP, packets will be dropped.

Distribution

Erlang allows you to send messages between processes at different nodes on the same or different computers. It is also possible to interact with C-nodes (Erlang nodes implemented in C). The communication is done over TCP/IP and obviously this introduces latencies, especially when communicating between nodes on a network.

Even when the nodes are running on the same computer they communicate using TCP/IP over the loopback interface. Different operating systems have widely different loopback performance (Solaris has lower latency than Linux). If your system uses the loopback interface it's a good idea to consider this.

Erik Rigtorp

Efficient rolling statistics with NumPy

Efficient rolling statistics with NumPy

Automatically generating Erlang release files

Automatically generating Erlang release files

Using jQuery to display a Yahoo! Pipe

Using jQuery to display a Yahoo! Pipe

Erlang Latency Guide

Erlang Latency Guide

Introduction

Yield

Network

Distribution

Further Reading