It started with NATS. I wanted to cluster my server, NATS seemed like a good fit, and I thought: how hard would it be to write a Zig client? Turns out, not that hard. nats.zig came together well, but it was using threads and blocking sockets. I felt like I stepped into the past, I really wanted asynchronous I/O for this. Every approach I tried (wrapping libuv, wrapping libxev) ended up requiring constant allocations and reference counting. It was not nice code. I really wanted Go-style networking APIs.
So I asked myself the stupid question again, how hard could that be? And that led to a fun, but very long journey. I’ve already written about it.
With Zio working, I needed an HTTP server. I was previously using http.zig by Karl Seguin, which is a great project, but it runs its own event loop and I wanted multiple network services sharing the same runtime. So I wrote Dusty, with an API inspired by http.zig but built on Zio. Dusty is more than just an HTTP server, it’s a full HTTP package. It includes an easy to use HTTP client with connection pooling, natively supports WebSocket (both server and client), Server-Sent Events, has a router with parameter and wildcard support, and a middleware system. I used llhttp for parsing rather than writing my own, because while HTTP/1.1 is an easy to read protocol, there are many edge cases and why not use something that already covers them all.
At this point I had a runtime and an HTTP server, which meant I could actually build something. I still needed some more client libraries. The first one was for Memcached. That was fairly easy to build, it uses the same rendezvous hashing algorithm as the Python client library, I was happy with it. Then came the PostgreSQL client. I adapted pg.zig to use my own I/O layer, it was a small change and worked well. And then I took the Memcached client, removed hashing, replaced the protocol parser, and turned it into a small Redis client. This one is very far from feature-complete, but I don’t use Redis for much, so it covers just what I needed.
I never expected to be writing networking code in Zig when I started. But it turns out I now have a stack I actually enjoy working with. Maybe I need to stop the yak shaving at this point, and finish the AcoustID project that started it all.
Writing backends in Zig is really not for everything. If you’re building a CRUD app, Go or Python will get you there faster. But for the kind of work where performance matters at a low level, like databases, streaming services, or audio processing, I now consider Zig a good option. You don’t have to drop down to a different language for the networking layer.
Once Zig 0.16 is released, I’ll probably work on migrating the
libraries to use std.Io, but there is still some missing
functionality in the APIs that makes it impractical at this point.
I’ve been planning to rewrite AcoustID’s inverted index for a long time, I had a couple of prototypes, but none of the approaches felt right. I was going through some rough times, wanted to learn something new, so I decided to use the project as an opportunity to learn Zig. And it was great, writing Zig is a joy. The new version was faster and more scalable than the previous C++ one. I was happy, until I wanted to add a server interface.
In the previous C++ version, I used Qt, which might seem very strange for a server software, but I wanted a nice way of doing asynchronous I/O and Qt allowed me to do that. It was callback-based, but Qt has a lot of support for making callbacks usable. In the newer prototypes, I used Go, specifically for the ease of networking and concurrency. With Zig, I was stuck. There are some Zig HTTP servers, so I could use those. I wanted to implement my legacy TCP server as well, and that’s a lot harder, unless I want to spawn a lot of threads. Then I made a crazy decision, to use Zig also for implementing a clustered layer on top of my server, using NATS as a messaging system, so I wrote a Zig NATS client, and that gave me a lot of experience with Zig’s networking capabilities.
Fast forward to today, I’m happy to introduce Zio, an asynchronous I/O and concurrency library for Zig.
If you look at the examples, you will not really see where is the asynchronous I/O, but it’s there, in the background and that’s
the point. Writing asynchronous code with callbacks is a pain. Not only that, it requires a lot of allocations, because you need
state to survive across callbacks. Zio is an implementation of Go style concurrency, but limited to what’s possible in Zig.
Zio tasks are stackful coroutines with fixed-size stacks. When you run stream.read(), this will initiate the I/O operation in the background
and then suspend the current task until the I/O operation is done. When it’s done, the task will be resumed, and the result will be returned.
That gives you the illusion of synchronous code, allowing for much simpler state management.
Zio support fully asynchronous network and file I/O, has synchronization primitives (mutexes, condition variables, etc.) that work with the cooperative runtime, has Go-style channels, OS signal watches and more. Tasks can run in single-threaded mode, or multi-threaded, in which case they can migrate from thread to thread for lower latency and better load balancing.
And it’s FAST. I don’t want to be posting benchmarks here, maybe later when I have more complex ones, but the single-threaded mode is beating any framework I’ve tried so far. It’s much faster than both Go and Rust’s Tokio. Context switching is virtually free, comparable to a function call. The multi-threaded mode, while still not being as robust as Go/Tokio, has comparable performance. It’s still a bit faster than either of them, but that performance might go down as I add more fairness features.
Because it implements the standard interfaces for reader/writer, you can actually use external libraries that are unaware they are running within Zio. Here is an example of a HTTP server:
const std = @import("std");
const zio = @import("zio");
const MAX_REQUEST_HEADER_SIZE = 64 * 1024;
fn connectionTask(rt: *zio.Runtime, stream: zio.net.Stream) !void {
defer stream.close(rt);
var read_buffer: [MAX_REQUEST_HEADER_SIZE]u8 = undefined;
var reader = stream.reader(rt, &read_buffer);
var write_buffer: [4096]u8 = undefined;
var writer = stream.writer(rt, &write_buffer);
var server = std.http.Server.init(
&reader.interface,
&writer.interface,
);
while (true) {
var request = try server.receiveHead();
try request.respond("hello", .{ .status = .ok });
if (!request.head.keep_alive) break;
}
}
fn serverTask(rt: *zio.Runtime) !void {
const addr = try zio.net.IpAddress.parse("127.0.0.1", 8080);
const server = try addr.listen(rt, .{});
defer server.close(rt);
while (true) {
const stream = try server.accept(rt);
errdefer stream.close(rt);
var task = try rt.spawn(
connectionTask, .{ rt, stream }, .{}
);
task.deinit();
}
}
pub fn main() !void {
var gpa = std.heap.GeneralPurposeAllocator(.{}){};
defer _ = gpa.deinit();
const allocator = gpa.allocator();
var runtime = try zio.Runtime.init(allocator, .{});
defer runtime.deinit();
try runtime.runUntilComplete(serverTask, .{&runtime}, .{});
}
When I started working with Zig, I really thought it’s going to be a niche language to write the fast code in, and then I’ll need a layer on top of that in a different language. With Zio, that changed. The next step for me is to update my NATS client to use Zio internally. And after that, I’m going to work on a HTTP client/server library based on Zio.
]]>Last year, I discovered Cody from SourceGraph. I’ve tried the trial and I was hooked. It had so much more context about the code I’m working on. I could just select a function, tell it to refactor something on it, and it would do it directly in my editor. Writing documentation, generating tests, writing new code, everything become easier. I’ve used it last year to write a replacement of the acoustid-index server, something I’ve been planning for a long time, but I decided to also learn a new language, Zig, on the project. Cody made the process really effortless. It included countless refactoring, as I was still learning the right patterns in the language, and I was doing most of the work without actually writing the code myself. This year, I’ve started using the chat with thinking models a lot more often, and Cody’s ability to apply the code blocks from the chat to the editor. Even better, I’m actually using this for free, as part of their support for open source. It’s such a good tool that I’d be happy to pay for now, and will definitely start doing that once my current free license expires.
And this year I discovered CodeRabbit for automated code reviews. I was super skeptical about this, but they also have a free plan for open source projects, do I decided to give it a try. I’m maintaining AcoustID alone, so having another set of eyes looking at the code, even if mechanical ones, is welcome. And I was blown away. On the first pull request, it actually found a small logical error I had in the code. And this kept happening again and again. After some time, I switched it to the assertive profile, and now I actually enjoy opening a pull request and going through the suggestions it makes. Yes, sometimes they are obsessive, but that’s OK. I’ve tried alternatives, like Gemini or Copilot, both having options to do code reviews, but the level of quality is somewhere completely elsewhere. Gemini and Copilot feel like useless toys compared to CodeRabbit.
The last four years have completely changed my approach to programming, and for the better. As good as all these new AI tools are, I don’t really expect them to be replacing technical programming jobs. You really need to evaluate their outputs, and if you are not able to do that critically, you will deal with a lot of bullshit code. But if you can judge the quality of the output, these are great helpers and I’m really looking forward to what the future brings.
]]>The library can be only used with static schemas, defined using Zig’s type system. There are many options for generating compact messages, almost competing with protobuf, but without separate proto files and protoc. I’m quite happy with the API. It’s mainly possible due to Zig’s comptime type reflection.
This is the most basic usage:
const std = @import("std");
const msgpack = @import("msgpack");
const Message = struct {
name: []const u8,
age: u8,
};
var buffer = std.ArrayList(u8).init(allocator);
defer buffer.deinit();
try msgpack.encode(Message{
.name = "John",
.age = 20,
}, buffer.writer());
const decoded = try msgpack.decodeFromSlice(Message, allocator, buffer.items);
defer decoded.deinit();
std.debug.assert(std.mem.eql(u8, decoded.name, "John"));
std.debug.assert(decoded.age == 20);
See the project on GitHub for more details.
]]>So what’s new?
The biggest feature is that all components of audio fingerprinting process now work in a streaming fashion and can provide partial results at any time. That means that it’s now possible to feed a continuous audio stream to the process and get back partial fingerprints. This is useful if you want to fingerprint e.g. an internet radio stream and do not want to explicitly split the stream into small chunks. There are also side effects of these changes and the whole process is now faster and uses less memory.
This change is also reflected in the fpcalc utility, which can now work on streams. In fact, the fpcalc utility has been completely rewritten. It now also supports JSON output for easier integration, since you can find JSON parser in the standard library of almost every new programing language.
Here is an example using an online radio:
$ fpcalc -ts -chunk 10 -overlap -json http://icecast2.play.cz/radio1.mp3
{"timestamp": 1480789416.14, "duration": 12.60, "fingerprint": "AQAAUFSSTEnCRBKeZsR0..."}
{"timestamp": 1480789423.68, "duration": 12.60, "fingerprint": "AQAAUZOSRIsUTkGz9SCZ..."}
{"timestamp": 1480789433.68, "duration": 12.60, "fingerprint": "AQAAUYmWfJIoiNIRHzsl..."}
Or you can fingerprint raw pcm data from an external audio input:
$ arecord --format=S16 --rate=44100 --channels=1 --file-type=raw \
| fpcalc -format s16le -rate 44100 -channels 1 -ts -chunk 10 -overlap -json -
{"timestamp": 1480789965.67, "duration": 12.60, "fingerprint": "AQAAUFqkMVmyJMHWBa97..."}
{"timestamp": 1480789975.81, "duration": 12.60, "fingerprint": "AQAAUVGYSEkXJvit4ce1..."}
{"timestamp": 1480789985.72, "duration": 12.60, "fingerprint": "AQAAUcoSpRunwN6Op_hy..."}
This opens up many options how Chromaprint can be used outside of AcoustID.
There has also been a big source code cleanup. A lot of old code has been removed. We started using C++11 features, so it’s no longer possible to compile Chromaprintwith old C++ compilers. Unit tests no longer depend on boost. KissFFT is now bundled in the package and used as a fallback if no other FFT library is found. That means Chromaprint can be now build without any external dependencies.
The public C API now uses standard fixed-size int types from stdint.h. This breaks API-level backwards compatibility, you will need to modify your programs. The binary interface has not changed and programs compiled against the old version of the library will continue working.
All source code from Chromaprint written by me has been relicensed from LGPL to MIT. Note that this does not mean much for the library as a whole, since it still depends on LGPL code, but it’s now easier to reuse parts of the code in other projects, if needed.
We also have fpcalc binaries for ARM processors now. They were built and tested on Raspberry Pi, but might work on other ARM devices.
Download:
fpcalc tool
This version updates the SQLAlchemy models to the latest MusicBrainz database schema. It also adds support for Python 3 in addition to Python 2.
Install:
pip install mbdata==2016.7.17
Changes since version 1.3.1:
Download:
fpcalc tool
Changes since version 1.3:
fpcalc -length to actually restrict fingerprints the requested length.Download:
fpcalc tool
Changes since version 1.2:
fpcalc -length 0 to get the full fingerprintchromaprint_get_fingerprint_hash for calculating SimHash from the fingerprint dataDownload:
fpcalc tool