Dominic's Website

Applying Zettelkasten to programming

2025-06-08T12:00:00Z

As any programmer will tell you, code organization is paramount for any non-trivial project. When our code grows too big for a single file, we split it up into multiple ones. When we have too many files, we put them into folders. When we have too many folders, we put them in more folders ad infinitum. This hierarchy strongly shapes a project's structure and maintainability.

Tree hierarchies feel quite natural. They conceptually fit with abstraction, where parent modules abstract the capabilities of their children. It's also easy to implement, leveraging the filesystem for most of the work. But how does it work in practice?

Siblings and permissions

Having private utilities is a core piece of encapsulation; we carefully control the surface area exposed to the public. If a single module needs a private utility this is fairly straightforward, but often we need multiple modules to have access to the same internal operations. In basically every language, we end up doing this by creating a semi-public sibling module.

This causes two problems:

util is now public to the rest of our application
The folder structure no longer represents our architecture

While this simple example is still understandable, over time this conflict leads to gradual erosion of any correspondence between where code is and how it's being used. Look in nearly any codebase, and you will see imports traversing up & down to distant cousins in the folder tree.

While there are some clunky attempts to implement more granular permissions (e.g. Java's protected), generally you have to choose between being used by one module, or being public.

Representing a dependency structure like this is not generally possible

The first casualty: tests

This limitation has caused a long-standing debate in the unit test community: how do you run tests, which are in a separate module, against private functions? Currently, your options usually are:

Don't test private functions; only test public interfaces
Publicize all functions, but with naming conventions that indicate they are actually private
Write tests in the same module as code

To be straightforward, option 1 is flat-out incorrect. The point of unit tests is to make sure that each functional unit in your application behaves as expected under a wide variety of scenarios. If you are only testing a module's public interface, you are writing integration tests. Being able to test private edge cases and have a well-defined purpose for private functions leads to higher confidence that your public interface is actually robust. This is an "actually it's a skill issue" justification for people struggling against the very real limitations of our programming paradigms.

I hope it's also clear that patterns like public _myPrivateFunction and if (<inTestRunner>) { ... } are hacks around design limitations, not actual solutions.

What does Zettelkasten have to do with anything?

Zettelkasten is a knowledge management system that organizes things through links, rather than locations. Instead of trying to constantly move information around in a hierarchical system, you divide notes into small, atomic nodes and use their links to other nodes for discovery.

This is surprisingly similar to our requirements for programming! We have a series of atomic nodes (functions) that can arbitrarily link to each other. Instead of code living in a single location, its purpose is organically derived from how it's used and documented.

We actually somewhat follow this pattern today! Monorepos encapsulate code in packages with flexible dependencies, hinting at a node-based approach. This idea is an extension of that to all code, further reducing the overhead of manual organization.

The key for this to work is a good user interface that lets you have dynamic "views" of code, such as:

Show me this function and all functions it uses within the current module
Show me all the code relevant to this stack trace
Show me all the consumers of this function

Parting thoughts

This is the first in an upcoming series of posts describing some of the design principles of an in-progress programming environment. The core to everything here is that we need to start treating code in a way conducive to large-scale management and understanding, not in a way that fits into our decades-old general computing techniques.

Technology sucks. Why?

2024-03-04T14:27:00Z

I live and breathe technology. I program for work and for fun. I play games, watch shows, and spend time with friends all on my computer. Technology is an incredible innovation that has unlocked a range of new activities and possibilities, quickly becoming something we rely on every day.

It's also, in many ways, a giant piece of crap.

Part one: I'm not being melodramatic, I swear

I am admittedly a bit biased due to my intimate connection to technology, but I think the tech industry is failing in ways I don't see in other industries. I encounter significant bugs in major pieces of software like Google Maps, Android, or Windows on a daily basis, and many large tech companies debug and patch multiple user-facing issues per day. Software is regularly abandoned or completely rewritten due to tech debt, and upgrades often make things slower and less usable rather than the opposite.

I'm not talking about the typical small-business website made on a tight budget. Those are admittedly bad, and there are ways we can improve them, but low quality in projects like that is common across industries. I'm talking about companies with thousands of crazy smart, talented, full-time engineers who are struggling to make something good.

Why are we stuck with a few pieces of good software and a frustratingly bad majority? The industry is certainly paying enough to attract necessary talent. It's not like software companies are uniquely bad at making things either; the worst part of pretty much any modern car is generally the infotainment system! It seems to just be software itself that's the issue. Why?

Part two: What went wrong?

Software is abstract

Unlike in mechanical engineering where the product is a physical object, software is abstract. I believe this is actually a major barrier to creating software at scale.

Humans have evolved to be masters of visual pattern recognition, and the physicality of mechanical engineering provides a source of truth for what you're creating. Where is each part? What parts are interacting? Where are likely failure points? How does the system work at a high level? It's orders of magnitude quicker to get this information from a 3D model of a part versus carefully reading a codebase.

This abstractness makes code much more opaque. We use abstractions to solidify this opacity in logical places, but in a company setting it's very easy to create poor, undocumented, or incorrect abstractions. Fundamentally, software engineers are operating with less information and a poorer understanding of their work.

Complex, abstract work isn't limited to software engineering. I think a lot of similarity can be drawn between a mathematical paper and a piece of software. Both have "hard" bugs, where an oversight leads to a result that's just plain wrong. Both are abstract, and take a significant amount of time to audit and understand. Where software engineering becomes unique is in its scale, both in the amount of code and the number of people working on it at once.

Software is relatively unconstrained

There's a ton of constraints when building physical objects. Every new part costs money up front and per-unit, so there's a very strong incentive to minimize complexity and use pre-existing parts. And of course, the design itself is limited by reality; you can't just build a stove that floats or have your fridge magically cool your bathroom.

By comparison, software has much fewer constraints, both from a technical and economical perspective. The cost of a programmer is so enormous compared to any running costs of the software itself that extra complexity and inefficiencies are easy to justify. Technically, any part of a piece of software can interact with any other piece.

This freedom is one of the strengths of software! It's how we can iterate quickly and have such creativity. But it also means that, left to grow organically, software becomes overly complex and interconnected. In tandem with its abstract nature, large pieces of software are fragile and difficult to understand.

Best practices aren't enough

In a perfect world, we have best practices and architectures to deal with the above issues. With enough work and thought, we can create clean and maintainable software.

The world is, surprise, rarely perfect, and discipline rarely beats out structural incentives. Engineers with the best of intentions will make mistakes, and over time as best practices become more difficult to maintain, discipline will erode. Even if each individual change is well-designed, their sum will eventually corrode the quality of the code. As the system becomes more complex, the understanding needed to develop an intelligent architecture also becomes harder to obtain.

Putting effort into developing a better piece of software is important, but our systems are not supporting it at scale.

Mistakes are cheap (until they aren't)

Mistakes in software are among the cheapest to fix. You don't have to do a product recall or change manufacturing; you just push out an update. As a result, you don't generally need to take as much care when developing software. A fix is always a rollback away.

That is, if the mistake is a bug.

A bug is relatively obvious; the program is not doing what it's supposed to. But we make other mistakes too. We may architect a program incorrectly, not anticipate a future use case, or simply make tradeoffs that were reasonable at the time but cause issues later down the road.

These mistakes usually don't rear their head for years or decades, and once they do, they are extremely hard to remove. It's why Windows is still compatible with DOS, why memory safety vulnerabilities are so prevalent, and why the mainstream way to create a cross-platform application is to embed it in its own mini-operating system (read: a web browser).

Software builds on software in a way that's extremely costly to change, possibly even more than in manufacturing. This is a dire enough problem on its own, but combined with the apparent cheapness of fixing software, we are lulled into a false sense of complacency with lackadaisical software development. Costly mistakes are introduced at a rate that's not feasible to deal with continuously, and so the spectre of tech debt begins to haunt the files until someone takes a flamethrower to the whole thing.

We live in a society

And finally, sometimes building good software just isn't the goal. Almost all software we use is commercial, and the end goal is to extract value. If a crappier product makes more money, then the crappier product will win in a commercial setting. Software also has powerful monopolistic properties that mean it can be cheaper to just force people to keep using your product rather than fixing it.

This topic deserves its own blog post, but needless to say, if making really good software is hard (and it is), companies will find ways to make money from software without focusing on quality.

Part three: So what do we do?

¯\_(ツ)_/¯

Underneath the technical details, this is mostly one of those annoyingly intractable "human" problems (ew). It’s very difficult to move the mountains that are large tech corporations in a meaningful way.

If you work somewhere that you feel is falling into this trap of bad software, the best thing you can do is act like it's an important issue every step of the way. Be curious, talk to people, and figure out how to break down silos. Dogfood the product, give feedback, and make sure things actually get fixed. Be loud about why quality is important in ways that maybe aren't immediately obvious on an A/B test. Make it a product problem, not an engineering wishlist.

Going beyond the REPL

2023-11-11T12:05:00Z

REPL-driven development has long been lauded by Lisp programmers, but has failed to go mainstream. Can we design a better approach to interactive development?

The problems with REPLs

Separation of the REPL and code

REPL-driven development generally involves testing expressions in a REPL, and then transferring them to code when completed. The state of your REPL and code are completely disconnected. Some common issues here are:

Changing a function in code and forgetting to change it in the REPL
Using a variable with different values in the code vs the REPL
Using a function or variable that no longer exists in the code
Needing to emulate the state of the program at the point an expression is run

Ephemeral knowledge

Exploratory development via a REPL can be incredibly empowering. Rather than guessing-and-checking via print statements, you can interactively test any expression.

But once you've finished your coding session, what happens? You've built knowledge on how all the expressions in your function operate, and after closing your REPL window…it all disappears. We just throw it away.

By giving the developer powerful introspection abilities but not the reader, REPLs create a rift in understanding. Questions that are trivially answered in the developer's state of mind are lost to the reader because they don't have the same context.

Code written without a REPL is clearer to the reader by necessity, because the information the developer and reader are working with is more similar.

What about debuggers?

Debuggers are great, but they don't replace REPLs. Testing an arbitrary expression requires writing it and re-running the entire program to a breakpoint. You then inspect the value. This is too slow for REPL-driven development; we need to incrementally update the program state.

What about computational notebooks?

Computational notebooks have gained mainstream support, primarily due to their REPL-like structure. Unfortunately, they are generally single-file programs meant to run interactively, and are too narrow in scope to serve as a general-purpose programming technique. Computational notebooks do, however, hint at the direction we need to go.

The idea

Regardless of what language you're in, there's always some way to inspect an expression. The problem is not can we test an expression, but how fast we can do so. REPLs test small expressions quickly, but introduce a friction boundary between the REPL and code. Can we test small changes directly in the code, rather than in a REPL?

f x = out
  where
    a = op1 x
    b = op2 a
    out = some_complex_function b

-- We need to know what b is first!
-- Copying all the previous expressions is tedious
>> a = op1 "my input"
>> b = op2 a
>> some_complex_function d
"my answer"

Technologically, this is not an interesting idea; debuggers already do this. The point is developing a UX that encourages a healthier developer workflow.

How many times have you written an entire function, file, or more before ever running any code? How often were you unsure if an expression was correct, but delayed testing it until the function was complete? The harder it is to inspect and test code, the longer we will wait until testing it.

We need to put the value of an expression right in your face, as soon as you write it. Let's evaluate our code continuously, and show the values of expressions as soon as you type them.

As a developer, this makes the feedback between writing and testing code instant. There's no explicit action to take to start a test, and so no incentive to wait to test. If an expression is incorrect on the given test case, the developer will see that immediately after writing it.

As a reader, this is a powerful tool for exploring an unfamiliar codebase. Learning by example is a powerful tool, and seeing concrete instances of any value lets readers think less abstractly about code.

Leveraging unit tests

We can use unit tests to run our desired function. Not only does this give the reader documented examples to introspect, but it also encourages developing unit tests in parallel with function development.

Loops & recursion

If our function has recursion (or loops which can be expressed as recursion), an expression can be run multiple times in a single call. How do we let the user explore all different usages?

As long as our arguments are immutable, storing the arguments to all calls of a function is generally not significantly expensive. Then the user can selectively introspect a specific call, or see the value of an expression/return value across a series of calls.

factorial 0 = 2 -- Uh oh, this should be 1!
factorial x = x * factorial (x - 1)

-- Seeing the return values across the call stack
-- makes finding the bug easy
>> factorial 3
factorial 3 -- 12
factorial 2 -- 4
factorial 1 -- 2
factorial 0 -- 2

In pathological cases where complete storage is too expensive, we can still:

Explore a single branch of recursion rather than the entire tree, which is guaranteed to be as cheap as the function call
If the input range of the expression is small, wrap it in an anonymous function and observe all the times it is called

Retrofitting existing languages

We can inefficiently implement this in any functional language with a REPL via code transformation. Adding wrappers around the tested function and expressions makes it easy to track their values and arguments.

factorial x = x * factorial (x - 1)
=> factorial x = memoize
    	(Memo {args=[x], value=x * factorial (x - 1)})

But this requires re-running the entire function on change! A key feature of REPLs and computational notebooks is running an expensive expression once, and then re-using it. Can we do this automatically?

Taking advantage of purely functional code

If we know parts of our code are purely functional, then yes! Let's look at a simplified model of function design to see why.

Say a function is a series of assignments, that eventually returns an expression:

format_matrix matrix = -- some expensive computation

make_csv headers matrix = csv_output
  where
    formatted_matrix = format_matrix matrix
    csv_lines = map (join ",") (headers : formatted_matrix)
    csv_output = join "\n" csv_lines

Now we want to change the file to be tab delimited:

format_matrix matrix = -- some expensive computation

make_csv headers matrix = csv_output
  where
    formatted_matrix = format_matrix matrix
    csv_lines = map (join "\t") (headers : formatted_matrix)
    csv_output = join "\n" csv_lines

We know that format_matrix has no side effects, so we can skip it and just re-run the csv_lines and csv_output expressions. Or, inversely, we know we only have to re-run csv_lines and the things that depend on it.

If we develop a function top-down, we'll generally only have to re-run the last expression. This is potentially much more efficient than running a test case from scratch.

Parting thoughts

For decades, our solution to fundamental challenges with programming has been "make a new language". This is important, but I think we have neglected innovating how we interface with code. Going beyond the REPL is just one step in improving our programming methodologies.

Convenience is a bad goal

2023-10-13T12:07:00Z

Everyone likes convenience. We like it so much that many of our day-to-day decisions revolve around it. But just because something is convenient doesn't mean it's good—or even enjoyable!

Convenience can be a useful tool: when we make good things convenient (e.g. public transport), we encourage people to use them. However, modern society often enables unhealthy forms of convenience.

Abstracting cost

Convenience, and particularly technological convenience, is used to hide real costs. When you order delivery you no longer interact with the restaurant or delivery driver. When you buy on Amazon you no longer interact with a store and cashier. Businesses intentionally hide these externalities so they don't factor into your mental calculus. Abstracting away the costs involved in business and reducing everything to a monetary exchange robs individuals of agency and importance.

Consumerism

The advent of an economy where people simply continue buying more and more new things is relatively recent. We used to be able to fix products, or continue to reuse them for a long time. However, given a design trade-off between convenience and sustainability we almost universally choose convenience.

Think of every wrapper from the grocery store and every box from an online purchase. Convenience has increased both the waste of each purchase and how much we buy.

Discomfort is important

Convenience often caters to our avoidant tendencies: we use it to dodge discomfort, even when that discomfort is normal and healthy.

If you are forced to confront the reality of where your food and products come from, to interact with someone, or to see what other people are going through, that is a good thing. Avoiding discomfort is never worth avoiding the truth that comes with it.

Visual clarity of code

2022-12-13T12:00:00Z

When discussing how easy a programming language is to read or write, we primarily think about functionality; what abstractions does the language provide to make programming easier? What's not often mentioned explicitly—even if we recognize it instinctually—is the visual clarity of the language.

Take the following snippets:

def mult_by_2(arr):
  return map(lambda x: x * 2, arr)

def mult_by_2(arr):
  return [x * 2 for x in arr]

You probably found the second snippet more readable. We're operating at the same level of abstraction: these two snippets map one-to-one. However, the list comprehension syntax makes it visually clear that we're mapping elements of a list, and presents the information in a more natural reading order. We've gained type information and readability through a bit of syntax sugar.

Let's look at a more extreme example:

(integrate x a b
  (/
    (exp (- (/ (^ x 2) 2)))
    (sqrt (* 2 pi))))

$\int_a^b \frac{e^{-x^2/2}}{\sqrt{2\pi}}$

Fundamentally, these two snippets represent the same operation at the same level of abstraction. Yet the mathematical notation visually communicates that:

We're doing math
We're performing an integral from a to b
We have a fraction with an exponential term over a constant
This is an integral over the normal distribution

Teasing this information out of the Lisp program takes a lot more effort.

Except, well, you actually might not have understood the mathematical notation at all! Sure, you can technically grasp all that information quicker from the mathematical notation, but you could also be unfamiliar with something like integration and be left without even a function name to look up.

Climbing the curve

The Lisp snippet has a much lower barrier to entry. As long as you know Lisp syntax (which is famously simple enough to fit on a business card), you can see what function is applied to what variables/numbers. And if you're lost as to what integrate or exp do, you can look it up pretty easily.

On the other hand, getting a complete picture of the equation in Lisp is difficult. If you can read the mathematical equation, you probably find that syntax much easier to comprehend.

Every piece of syntax represents a chance to add a visually distinct operation, but also increases the learning curve—especially for reading code.

This is (in my opinion) one reason why many find Lisp and APL difficult to use despite them being such powerful languages. In Lisp, everything has exactly one visual form, rendering visual processing of the code very difficult. APL's symbols make it easier to scan for operations, but because it is almost entirely non-textual and doesn't use other visual demarcations like whitespace, a massive amount of memorization is necessary.

life ← {⊃1 ⍵ ∨.∧ 3 4 = +/ +⌿ ¯1 0 1 ∘.⊖ ¯1 0 1 ⌽¨ ⊂⍵}

An APL implementation of Conway's Game of Life. Impressive stuff—for the people who understand it.

Figuring out how to create programs that are both visual and readable is important. Math and engineering rely heavily on symbols and visualizations for a reason; the human brain is an incredibly efficient visual processor, which we neglect when we adopt a single syntax for every problem domain. In general-purpose languages, all code devolves into a series of functions and control statements, regardless of the code's meaning.

Imagine if there was an easily recognizable syntax for accessing a database, or logging a block of code. Every hint we give to our pattern recognition system makes understanding code easier, and we should take advantage of it.

So let's take a look at the practical challenges of visual programming representation and how we might approach them.

Dealing with the domain

It's very difficult—perhaps impossible—to have the necessary visual patterns in a general-purpose language out of the box. Common patterns vary between domains, specific applications, and even sub-applications. Users need to be able to create their own syntax to express their architecture.

Lisp does the functionality part of this pretty well, but it misses out on visual clarity due to its limited syntax. Other languages do some form of this, but I've not found a language that's truly flexible in terms of program syntax yet.

But there's good reason even macro-focused languages don't venture here. Even ignoring technical challenges like parsing, how would we make such a language readable to people outside the domain? Due to the wide applicability of programming, programmers shift between domains more often than in many other knowledge fields; if it takes a week to learn the application syntax, we can basically kiss open-source contributions goodbye. We need another solution.

Multiple representations

Putting a layer of indirection between the canonical program representation and what the user sees is not unheard of; it's what every WYSIWYG editor does. In programming, projects like Enso, Smalltalk, or even syntax highlighting act as middlemen between the literal program file and what shows up on the screen. Using indirection, we can modify what people see to align with their experience level. A symbolic language like APL could show textual versions of its operations, or a textual data science language could show a boxes-and-lines-style visual representation.

This is similar in spirit to the gradual programming model of Hedy. The needs of beginners and experts are different, and switching program representations should accommodate that.

Language support

The previous section hints at how this could actually work. Even with custom syntax, there's a canonical AST that can be used by tooling. There's even prior work here: Tree-sitter provides language-independent syntax higlighting, and Tree-edit provides structural editing on top of that AST.

So what?

OK, say you're not convinced that we need better visualizations of code. What does all this work get us?

As is often the case when indirection is introduced, a whole lot.

Bidirectional representations for editing source code are fairly limited, since you have to maintain complete information over the transformation. But what if you allowed one-way transformations? You probably don't care what AST your programming language uses under the hood, so what if they just used the same one?

Every time we start a new language, we start at almost a blank slate. Debugging, editing, profiling, introspection, toolkits, package management; every language rebuilds these ideas all over again. Hell, nowadays these tend to be the killer features of new languages. We should be able to focus on the language syntax and features—the representation of the language—separately from the tooling.

Having this separation makes it much easier to develop small, easy-to-reason-about DSLs instead of hammering everything into general-purpose languages.

We already have this functionality in highly programmable languages like Lisps. It's how we can get powerful features like pattern matching without compiler support. All we're missing is the ability to visualize our new constructs in readable ways.

Wrapping up

Hopefully this post gave you some new insights. This space is massive, and there's probably a lot of research material out there I'm not aware of. I've included some further reading below; if there's something cool I'm missing let me know and I'll add it in!

Dominic's Website

Applying Zettelkasten to programming

Siblings and permissions

The first casualty: tests

What does Zettelkasten have to do with anything?

Parting thoughts

Technology sucks. Why?

Part one: I'm not being melodramatic, I swear

Part two: What went wrong?

Software is abstract

Software is relatively unconstrained

Best practices aren't enough

Mistakes are cheap (until they aren't)

We live in a society

Part three: So what do we do?

Going beyond the REPL

The problems with REPLs

Separation of the REPL and code

Ephemeral knowledge

What about debuggers?

What about computational notebooks?

The idea

Leveraging unit tests

Loops & recursion

Retrofitting existing languages

Taking advantage of purely functional code

Parting thoughts

Convenience is a bad goal

Abstracting cost

Consumerism

Discomfort is important

Visual clarity of code

Climbing the curve

Dealing with the domain

Multiple representations

Language support

So what?

Wrapping up

Further reading