Peter Alexander

Posted: • 13 min read

There’s a current trend of LLM-authored software moving away from programming languages designed for human ergonomics (e.g. Python) towards those with other upsides such as higher safety, performance, or iteration speed. From what I’ve seen, many people are rewriting various parts of the software stack in Rust for this reason (Bun, Pokémon Showdown, the TypeScript compiler): it produces fast, safe code and the usual frustrations around having to satisfy the borrow checker may matter less to AI agents. The agentic experience (AX) does not necessarily match the human experience, so we have to revisit our assumptions around what matters in language design.

I’ve been a long-time fan of the D programming language, so my natural inclination here is to look at how D fares as a language for agent engineering. I do believe it has the potential to be a great language for agents (fast, safe code, rapid compilation, powerful abstractions). I’ve been using D with Claude for a long time now and the vibes are good, but vibes are not data: how well does Claude actually work with D? Where does it trip up? How can D evolve to be better for agentic software development?

Fortunately, unlike with humans, it’s quite easy to run relatively large experiments with agents in a controlled manner. In this post, I’ll be exploring how Claude performs with D compared to Go on a set of synthetic problems from langbench.

The setup:

  • 40 synthetic real-world problems such as a JSON Schema validator library, or a card game simulator.
  • All using Claude Sonnet 5.5 in Claude Code.
  • Full test suite per problem available.
  • Asked to “structure it the way a well-maintained open-source D project of this kind would be” (or Go project), complete with unit tests.
  • Instructed not to use an existing library that solves the problem.

Overall, D does well, but does fall behind Go in terms of required output tokens and turns to produce a working solution. The difference is real and meaningful, but modest, and much of the difference is solvable.

D relative to Go: output tokens 1.13×, turns 1.38×, cost 1.19×, with 95% intervals

The rest of this post will get into details of where the differences are and what the D language and ecosystem can do about it.

D relative to Go

The ratios are computed by taking the median over repetitions of the same problem, dividing D by Go, and taking the geometric mean across all problems with a 95% bootstrap interval. Output tokens are the headline number here since they measure how much the agent had to write and think, independently of how it batches commands into turns. D took quite significantly more turns largely due to language-specific costs, but those tended to be cheap edit-build-test loops rather than e.g. significantly more thinking. Ultimately, D costs more because the agent was doing more edit-build-test loops.

D relative to Go Value
Output tokens (paired per problem, 95% CI) 1.13× (1.08–1.19)
Turns 1.38× (1.26–1.52)
Cost 1.19× (1.13–1.25)
Calls spent on language-specific costs D 290, Go 26

Where the extra calls go

The cause of the extra turns was identified via agentic reviewers reading the D and Go session transcripts side-by-side and categorising each tool call into one of four buckets:

  1. Productive work, e.g. writing code.
  2. Language-specific costs directly attributable to a language-specific papercut.
  3. Language-agnostic costs, e.g. wrong expectations in an agent’s own tests, shell slip-ups, etc.
  4. Discretionary work after the task was done.

The key takeaways are that while D does end up using extra calls for productive work, the majority of the difference is D-specific papercuts. Every D session hit at least one D-specific cost.

Tool calls by kind: D 894 productive, 290 language-specific, 293 any language, 154 discretionary (1,631 total); Go 785, 26, 259, 111 (1,181 total)

D’s specific costs

Within the language-specific costs for D, we further categorised them into 14 sub-buckets to understand common D-specific papercuts that agents were hitting. The top four account for nearly two thirds of D’s language-specific costs. Some were broad-and-shallow, affecting many sessions lightly, while others were narrow-and-deep, impacting a small number of sessions significantly.

Note: many of the issues are actually in the tooling rather than the language itself.

D-specific overhead in calls by category, with the number of sessions that hit each

Unit-test failures without values

By far the biggest issue is that, by default, D unit test failures only tell you that an assertion failed (e.g. assert(x == y)) and do not tell you what the values of the compared variables were that led to the failure. This caused the LLM to add extra print statements or grep output to find what actually happened. Not expensive, but it does cost an extra few turns and slows things down.

D has a solution to this with -checkaction=context but Claude didn’t find it on its own. A clear indication of how important defaults are!

In addition to missing context, each test module stops at its first failure, so working through N failures often takes N iterations. By comparison, Go’s test runner lists every failure.

D example (default):

core.exception.AssertError@tests/test_cpu.d(103):
  Assertion failure
----------------
??:? _d_assert [0x104f7d7ef]
…/tests/test_cpu.d:103 void
  test_cpu.__unittest_L100_C1()
??:? int rt.minfo.moduleinfos_apply(…)

D with -checkaction=context:

ca.d(2): [unittest] 3 != 4
1/1 modules FAILED unittests

Go example:

--- FAIL: TestMemoryOps (0.00s)
    chip8_test.go:156: BCD memory: 234 1 2
--- FAIL: TestFX1EAndFX29 (0.00s)
    chip8_test.go:167: I=0x10fe, want 0x10e
--- FAIL: TestTimerTick (0.00s)
    chip8_test.go:208: DT=7 ST=0, want 7 7
--- FAIL: TestDrawCollisionAndZeroRows (0.00s)
    chip8_test.go:244: VF=0 lit=2

dub’s unittest configuration

The prompt asked for “unit tests for the library, in files separate from the library source”. That is less common in D, where unit tests usually live alongside the code they test, but dub handles it fine, and the Go prompt asked for the same. What happened in just over half of the D sessions is that the agent built a unittest dub configuration that didn’t have a main, and then dub test failed with an undefined symbol error at link time for _main.

The fix is to use targetType "library", but it often took agents a few attempts to find it.

Undefined symbols for architecture arm64:
  "_main", referenced from:
      <initial-undefines>
Executable configuration "unittest" of package
sqlitefile defines no main source file

NaN-initialised floating point

A justified, but unusual, aspect of D is that floating point values are default initialised to NaN, e.g. with new double[n]. The idea is that if you forget to initialise then NaN is the most likely thing to cause your program to fail so that you learn quickly that you forgot and go back and properly initialise. Other languages either force explicit initialisation (which requires more sophisticated static checking), or default initialise to 0.0, which is often correct/expected, but sometimes just means that you forgot to specify the initial value and things just surprisingly worked with the default. It’s a deliberate but contentious decision that D makes.

Unfortunately, it is one that tripped up the agent in one task in particular. In all three SAT solver runs, the agent didn’t realise this, and the solver never finished, running until it hit the test suite’s 10-minute limit. Each time, it took quite a while to debug. It also caught out the SVG rasteriser, more briefly. Perhaps a niche case, but anecdotally it is something I’ve seen Claude hit in other contexts as well.

Attributes the agent adds itself

Claude likes to add @safe, pure, nothrow and @nogc in its D code. This is arguably a good thing, but it does then spend quite a few turns fixing the inevitable type checker failures. Self-inflicted, but this appears to be what Claude does with D.

The actual failures themselves were varied:

Smaller language frictions

There were also a number of smaller frictions:

  • const / immutable: const(Value) can’t become Value for concatenation, so the agent strips const across the codebase
  • rvalues to ref: resolve(Registry.parse(text)) where resolve takes ref const Registry. D compilers already accept this with -preview=rvaluerefparam.
  • Names that clash with the implicit object module or a package: capacity; a class named ModuleInfo, dozens of conflicting-symbol errors; function inflate in package inflate.
  • Keywords as names: macro is reserved but unused, and natural in a template engine; real() gives a parse error pointing elsewhere.
  • Runtime semantics: appending after shrinking .length reallocates every time, making a JSON parser quadratic, 6.22 s → 0.06 s once fixed; readText of an empty file returns null; deep-recursion tests overflow the unittest runner’s 8 MB stack with only exit code −11.
  • Phobos: to!double misrounds on arm64, so a shortest round-trip printer showed 0.3 for 0.1 + 0.2; relativePath returns relative paths unchanged.

A compiler bug, again

We also managed to hit an LDC-specific compiler bug that cost some turns. In certain scenarios, an exception can escape a catch incorrectly (only tested on arm64 macOS):

class E1 : Exception { this() { super("e1"); } }

void gen(int op)
{
    if (op == 0)
        throw new E1;
    try
        gen(0);
    catch (E1 e)
    {
    }
}

void main()
{
    import core.stdc.stdio : puts;
    try { gen(1); puts("ok: swallowed"); }
    catch (E1 e) puts("BUG: escaped");
}
$ ldc2 bug.d && ./bug
BUG: escaped

By problem

Compared to Go, there was notable variation across different problems in terms of number of turns. SAT was an outlier due to the NaN issue, and a handful required fewer turns than Go.

Tool calls summed over each problem's runs, D minus Go, for all 40 problems

What would make D cheaper for agents

Based on these results, we can make a few suggestions to the D ecosystem for how to make Claude more effective at writing D:

Change Where Calls Sessions Discussion
-checkaction=context under dub test; report every failure dub, druntime 71 29 dub#2296, forum, dmd#19992, dmd#19576
Executable unittest configuration without a main dub 42 31 dub#1270, dub#2060
Catch reads of NaN-initialised floats compiler, D‑Scanner 38 4 forum (2012), forum (2026)
Infer attributes for non-template functions compiler 37 23 DIP idea
Rvalues to ref by default (-preview=rvaluerefparam) language 12 9 DIP 1016, dmd#22078
Let explicitly imported names take precedence over the implicit object module (capacity, ModuleInfo); un-reserve macro language, druntime ~20 ~15 forum (macro)
LDC exception-escape bug; arm64 to!double LDC, Phobos 16 2 ldc#4433 (to!double)

In the short term, as a mitigation, it is likely that some simple Skills that remind agents of these papercuts could meaningfully help their effectiveness.

How this was measured

  • Run: D (LDC 1.41.0, dub 1.40.0) and Go 1.27.1. All 40 langbench problems, with ten of them (bigint, buildsys, chip8, datalog, depresolve, jq, regex, sat, toml and webhooks) run three times: 120 sessions total, 60 per language. After the run, three problems (mime, fulltext, svgraster) had test data replaced for licensing reasons, and their six sessions were rerun; the effort numbers above use the reruns.
  • Agent: Claude Sonnet 5.5 at high effort in Claude Code; one headless session per task in a private sandboxed workspace; no personal settings, memory or MCP servers; the same prompt apart from the language’s name and toolchain.
  • Correctness: each project rebuilt from a clean copy of its sources and run against the public tests, plus held-out tests (never shown to agents) for the five large problems.
  • Classification: all 2,812 tool calls of the original 120 sessions (before the six reruns) labelled by eight reviewers with shared instructions, reading D and Go sessions of the same problem side by side; large incidents rechecked against transcripts; the compiler bug reproduced.
  • Limits: one model, one harness, one platform (Apple Silicon/macOS). The productive/overhead boundary is uncertain by about one call per session; categories under five calls are anecdotes. Most problems ran once, so per-problem numbers are noisy.
  • Earlier run (different prompt, agents in a personal environment): D-specific costs were 62% of the gap then and 59% now, with the same causes recurring. Both prompts asked every language for unit tests in separate files.

Conclusion

I’m excited that we’re entering an era of software development where we can answer empirical questions more accurately and easily, where we once had to rely on anecdotes or small, often flawed, studies. I’d certainly not consider this run at the calibre of empirical quality that I’d like, but it’s a good starting point.

Some things I may pursue next:

  • Try different models, effort levels, harnesses.
  • Even larger problems, perhaps on established code bases with >100kLOC.
  • Timed measurements that also account for build time, run times.
  • Test out the impact of Skills that mitigate known issues.
  • Try different prompting strategies to guide agents in different directions, e.g. “keep things simple”, or “efficiency is important”.
  • Use a harness like this for evaluating new language features: wouldn’t it be nice if a language proposal came with an “if implemented, agents use X% fewer tokens in benchmark B”?

Posted: • 7 min read

I’ve never really gone very deep into graphics or rendering, but I’ve long been quite fascinated by the use of signed distance fields for defining geometry and implicit surfaces more generally. Like many people, I found out about this through Inigo Quilez’s work and Shadertoy in particular. A long time ago I did spend some time playing with it. I’d like to create more if I find the time / inspiration.

What I like most about SDFs for defining geometry is that simple geometry can be expressed simply. A sphere is defined with just a couple of mathematical operations:

float sphere(vec3 p, float r)
{
  return length(p) - r;
}

Compare this to regular triangle meshes where you can’t even define an exact sphere and trying to approximate it requires a huge number of vertices. This is not too difficult to define programmatically, but you are essentially forced to use authoring tools to deal with the complexity of defining geometry, no matter how simple. Many times when I’ve been prototyping game ideas, I’ve been frustrated that I’m forced to open up Blender just to create the most trivial piece of geometry.

What I’d really like is just some textual format for defining geometry similar to how you do it with SDF. OpenSCAD is there for CSG, but as far as I can tell you can’t do anything like smooth unions or domain transformations, which are needed for more organic-looking geometry. Using pure shader code is ok for this, but with shapes being represented as functions, transformations really need a language that supports first class functions (an SDF is a function of ℝ³→ℝ, then a union transformation is (ℝ³→ℝ)→(ℝ³→ℝ)→(ℝ³→ℝ)).

Finally, with LLMs in the picture, as cool as Blender MCP is, it would be nice if the LLM could just spit out a textual scene format. I tend to agree with Carmack here:

To that end, I started playing around with an idea of an SDF-based file format. As far as I can find, there isn’t really anything like this (which was quite surprising!). Would be interested to know if something like this exists.

The basic idea is that you can define shapes textually via simple primitives, transformations, and operations:

# Define sphere at origin, and rotated cube.
shape s = sphere(radius: 2);
shape b = box(width: 3, height: 3, depth: 3) |> rotate(x: 100, y: 200, z: 100);

# Translate and then combine via smooth union.
# As the last expression in the file, this is the output shape.
smooth_union(
  s |> translate(x: -1, y: 0, z: 0),
  b |> translate(x: 1, y: 0, z: 0),
  k: 1.0
);

This file defines the shape. I haven’t put a lot of thought into syntax yet, but the shape type here actually is an SDF, and we have primitives such as sphere and box as leaf nodes and higher-order SDFs such as translate and smooth_union for transformations.

With geometry defined this way, you can then render via a simple tool that understands how to parse the format. The tool does basic parsing and semantic analysis producing a runtime data model that supports querying the field at any coordinate. The tool also supports rendering using basic raymarching (its only function right now).

sdftool demo.sdf -o demo.png

This produces:

Example output of SDF tool defined earlier

Since we have an analytical shape, we can also extract normals via tetrahedron technique.

sdftool demo.sdf -o normals.png --render-mode normal

Example of rendering normals

If we need a mesh, we can generate one (currently using marching cubes with basic QEM simplification):

sdftool demo.sdf -o mesh.obj --mesh

Example of generated mesh

With this language in place, I can quite easily generate non-trivial meshes purely with text such as a chess pawn with smooth curves:

shape base = cylinder(radius: 1.4, height: 0.35) |> color(0.4, 0.35, 0.33);
shape base_rim = cylinder(radius: 1.2, height: 0.15) |> translate(y: 0.45);
shape stem = cylinder(radius: 0.36, height: 2.4) |> translate(y: 1.5);
shape mid_stem = cylinder(radius: 0.7, height: 1.4) |> translate(y: 0.5);
shape collar = cylinder(radius: 0.78, height: 0.01) |> translate(y: 2.7) |> expand(0.04);
shape head = sphere(radius: 0.7) |> translate(y: 3.5);

shape base_assembly = smooth_union(base, base_rim, k: 0.1);
shape mid = smooth_union(base_assembly, mid_stem, k: 0.4);
shape lower_pawn = smooth_union(mid, stem, k: 0.5);
shape pawn_collar = smooth_union(lower_pawn, collar, k: 0.1);
shape pawn = smooth_union(pawn_collar, head, k: 0.30);

translate(pawn, y: -2.5);
Chess pawn normals Chess pawn mesh

Another benefit of this format is size: the raw text for this pawn mesh is just 757 bytes, and this is without any attempt at compression. Could easily be under 100 bytes with minimal effort. The mesh by comparison is a few hundred kilobytes.

Of course, there are downsides to SDFs: rendering SDFs directly is still very expensive and complex geometry may end up being extremely difficult to model this way.

There are some future directions I’d like to take this:

  • A runtime library could load the file and support an API for evaluating the field at points, or raymarching.
  • The library could also allow creating the data format in memory at runtime.
  • Build more scaffolding to allow LLMs to create geometry from a prompt.
  • Support approaches to texturing.
  • And of course, improved algorithms for mesh generation and rendering.

Will continue tinkering on this and if it proves useful will maybe open source.


Posted: • 4 min read

I’ve been getting back into a bit of D programming recently and catching up on what’s been happening in the community. Some things have changed (Andrei isn’t around any more), but most things have stayed the same. Walter is still around and the language has a small and loyal following. DConf is still happening every year and people are still shipping things.

It is significantly smaller than both Rust and Go though. I think this is a real shame since I do feel that D fits a niche of languages that give you systems level access with quite powerful features around expressiveness and generally pleasant (and familiar) syntax. By comparison, I find Rust a bit too opinionated on how it should be written (yes, the borrow checker) and while Go is pleasant, it’s not very powerful when it comes to setting up scalable abstractions. D fits the gap quite nicely (while having problems of its own).

A perennial problem that D seems to have is that it can’t quite decide what it wants to be, and as a result tries to please everyone. As an example, D has always had a GC, but you get a subset of devs that really want to avoid the GC (for legitimate reasons). As a result, a long time ago, D added the @nogc attribute to functions, which statically checks if the function may allocate memory. The idea is that you can wrap your latency sensitive code (e.g. a game’s update loop) in @nogc and then rest assured that you will avoid GC pauses.

// Simple @nogc function - this compiles fine
@nogc int add(int a, int b) {
    return a + b;
}

// This won't compile - array literals allocate
@nogc int[] getNumbers() {
    return [1, 2, 3];
}

On the surface, @nogc is fine, but now you have a problem with what to do with common libraries: do they use the GC, avoid the GC, or provide the option of GC/no-GC? Whatever you choose, you are not going to win: either deal with endless complexity of supporting both, or make one camp unhappy.

I was not a fan when @nogc was originally proposed and I still think it was a mistake. I should mention that as a game developer, I do often want to avoid the GC!

I’d much prefer that D did not implement @nogc and instead:

  • Provide runtime tooling that helps you find GC allocations.
  • Try to avoid GC as much as possible in the standard library.
  • Continue to provide language features / utilities to avoid GC when needed.
  • Continue to invest in improving the GC pause times.

If you are a fan of @nogc, I’d further offer some other observations:

  1. If you care about minimising pauses in your application, GC is only one source of those. You have to also avoid any sort of non-wait-free locks or concurrent data structures / blocking syscalls / pauses from reference counted deallocation cascades.
  2. Modern GC implementations can get those pauses down <1ms (see Go as proof of life). This tech continues to improve, so it is a good bet.
  3. You can achieve 99% of the desired outcome via runtime tooling (logging on each GC allocation). I’ve done a lot of work in Unity3D to avoid GC in C#. Yes, it does take some effort to track down those allocations and fix them, but really the work is not that different from fixing compilation errors from @nogc, and you have the option of just ignoring the GCs that happen on the rare paths that you don’t really care about.

All this being said, @nogc has shipped and I would never advocate for breaking existing code. However, there are active discussions going on about a “Phobos 3” and I would suggest that the approach is to go all-in and assume GC. Standard containers should just use GC unapologetically (but responsibly). There’s nothing stopping someone from creating an allocator-based container library and putting it on dub, but it doesn’t need to be in the standard library.

I continue to believe that D has its place in the set of useful and unique programming languages and really ought to have more usage. Complicating the language / compiler / libraries in order to try and make everyone happy I think goes against the pragmatism embodied by D and just stretches already thin contributor resources. Stay simple, choose a 90% solution over 100%, and focus on quality of what’s already there.


Posted: • 3 min read

Often when building APIs we’d like the caller to provide some information in a format, or using a protocol defined by the library. For example, we might want access to a contiguous buffer specified by a (void*, size_t) pair.

void cool_feature(void* buffer, size_t len);

An API like this is fine if the buffer will be read synchonously, but sometimes we need asynchronous access. This introduces a lifetime problem: how does the caller know when it can free the buffer?

A common way to solve this is to provide a callback that will be invoked once the processing has completed:

void cool_feature(void* buffer, size_t len, void (*done)(void*));

Here, done will be called with buffer once cool_feature has asynchonously finished its work.

This works most of the time, but has a couple of problems:

  1. You need to remember to call done in all cases.
  2. It only works if done only needs to be called with the buffer pointer. What if my buffer is part of a larger object?
struct Object {
  // ...
  std::string buffer;
  // ...
};

void do_stuff(std::unique_ptr<Object> obj) {
  cool_feature(obj.buffer.data(), obj.buffer.size(), ???);
}

There’s no obvious choice of function to pass in. We want to free obj once buffer processing is done, but we’ll be given a pointer to the string data, not obj.

A possible solution is to pass in an std::function<void()> as the callback, so that obj can be captured and destroyed as necessary. This works, but std::function has difficulty with move-only types, and there’s no guarantees that a capturing std::function won’t allocate memory.

We could, of course, define another interface (maybe call it ContiguousBufferProvider), but this also requires an extra memory allocation.

A little trick we can do to handle most of the common cases is to pass in a type-erased context to the function.

using Context = std::unique_ptr<void*, void(*)(void*)>;

void cool_feature(void* buffer, size_t len, Context context);

The context is essentially an extra piece of baggage that must be carried around until the buffer has been processed. Once processed, we just drop the context and it cleans up after itself.

We can also benefit from an extra function that converts any object into a Context through type erasure:

template <typename T>
void deleter(void* p) {
  delete static_cast<T*>(p);
}

template <typename T>
Context erase_type(std::unique_ptr<T> p) {
  return Context(static_cast<void*>(p.release()), &deleter<T>);
}

With this, the Object example is easy to solve:

void do_stuff(std::unique_ptr<Object> obj) {
  cool_feature(
      obj.buffer.data(),
      obj.buffer.size(),
      erase_type(std::move(obj));
}

This guarantees no extra memory allocations, and cool_feature just needs to drop the context once finished.

There are probably more general ways to solve this, but I like this approach as it is clean, simple, and solves all the problems I’ve encountered so far.


Posted: • 5 min read

Last night I rewatched Joe Armstrong’s excellent keynote from this year’s Strange Loop conference. He begins the talk with a fairly standard, but humorous lament about the sorry state of software, then goes on to relate this to some physical quantities and limits. He finishes by suggesting some possible directions we can look to improve things.

One of his suggestions is to build something called “The Condenser”, which is a program that takes all programs in the world, and condenses them down in such a way that all redundancy is removed. He breaks this task down into two parts:

Diagram showing The Condenser concept with two parts: Part 1 removes all duplicate files, Part 2 merges similar files using least compression difference

He also explains the obvious, and easy way to do Part 1.

Slide showing Part 1 implementation: Calculate SHA1 of all files and remove files with duplicate SHA1 signatures

At this point I remembered that I hadn’t written a lot of D recently, and although Joe is talking about condensing all files in the world, I was kind of curious how much duplication there was just on my laptop. How many precious bytes am I wasting?

It turns out that The Condenser Part 1 is very easy to write in D, and even quite elegant.

import std.algorithm;
import std.digest.sha;
import std.file;
import std.parallelism;
import std.stdio;
import std.typecons;

auto process(DirEntry e) {
  ubyte[4 << 10] buf = void;  // 4kb buffer, uninitialized
  auto sha1 = File(e.name).byChunk(buf[]).digest!SHA1;
  return tuple(e.name, sha1, e.size);
}

void main(string[] args) {
  string[ubyte[20]] hash2file;  // hash table of SHA1 -> filename
  auto files = dirEntries(args[1], SpanMode.depth, false)
               .filter!(e => e.isFile);
  foreach (t; taskPool.map!(process)(files)) {
    if (string* original = t[1] in hash2file) {
      writefln("%s %s duplicates %s", t[2], t[0], *original);
    } else {
      hash2file[t[1]] = t[0];
    }
  }
}

I’ll explain the code a little, starting from main. The files variable uses dirEntries and filter to produce a list of files (the filter is necessary since dirEntries also returns directories). dirEntries is a lazy range, so the directories are only iterated as necessary. Similarly, filter creates another lazy range, which removes non-file entries.

The foreach loop corresponds to Joe’s do loop. While I could have just iterated over files directly, I instead iterate over a taskPool.map, a semi-lazy map from David Simcha’s std.parallelism that does the processing in parallel on multiple threads. It is semi-lazy because it eagerly takes a chunk of elements at a time from the front of the range, processes them in parallel, then returns them back to the caller to consume. The task pool size defaults to the number of CPU cores on your machine, but this can be configured.

The process function is where the hashing is done. It takes a DirEntry, allocates an uninitialized (= void) 4kb buffer on the stack and uses that buffer to read the file 4kb at a time and build up the SHA1. Again, byChunk returns a lazy range, and digest!SHA1 consumes it. Lazy ranges are very common in idiomatic D, especially in these kinds of stream processing type applications. It’s worth getting familiar with the common ranges in std.algorithm and std.range.

I ran the program over my laptop’s home directory. Since the duplicate file sizes are all in the first column, I can just use awk to sum them up and find the total waste:

$ sudo ./the-condenser . | awk '{s+=$1} END {print s}'
2347353793

That’s 2.18Gb of duplicate files, which is around 10% of my home dir.

In some cases the duplicates are just files that I have absentmindedly downloaded twice. In other cases it’s duplicate music that I have both in iTunes and Google Play. For some reason, the game Left 4 Dead 2 seems to have a lot of duplicate audio files.

So, in theory, I could save 10% by deduplicating all these files, but it’s not a massive amount of waste. It doesn’t seem worth the effort trying to fix this. Perhaps it would be nice for the OS or file system to solve this automatically, but the implementation would be tricky, and still maybe not worth it.

Part 2 of the Condenser is to merge all similar files using least compression difference. This is decidedly more difficult, since it relies on near-perfect compression, which is apparently an AI-hard problem. Unfortunately, solving AI is out of scope for this post, so I’ll leave that for another time :-)