Thoughts About the Shapes of Software
How we can focus on describing the problem we’re trying to solve and not on describing how the computer should solve it
I was in a conversation at a birthday party that forced me to think about how I have grown in the ways I think about software. Here is me trying to put it in words.
I started programming as most of us did, writing code to solve problems using the only tools I knew at the time (the programming language) to make the computer do what I wanted it to. But after more than a decade of building software, it became clear that we are building the same things repeatedly.
So, on a macro level, I could break down the kinds of software I've been writing into several shapes:
- Web applications
- CLI applications
- Daemons that react to events (System events or events from job queues)
- Embedded Software
- Software for data exploration (Usually in Jupyter Notebook or just random SQL queries against a database while investigating issues)
There are likely more shapes than these, but these are the ones I find myself writing. And I was convinced that within these shapes, we write the same things repeatedly. Let’s zoom into one of these kinds of software.
The Shape of Web Applications
I built a lot of web applications in my career, especially early on when I ran a software studio with a friend. We were building a lot of software with a pattern we tend to call CRUD.
- CREATE
- READ
- UPDATE
- DELETE
Whether it was a school management system, a food delivery application, etc., there was always 95+% CRUD, and then less than 5% would be some business logic that probably made that software unique.
For example, for a Yellow Pages application I built, we needed an administration dashboard where we could create listings, update those listings, delete them, etc. And this was true for almost all the other web applications I built. Then, of course, there would also be unique business logic, like the logic that businesses have to pay a certain amount monthly for some privileges, etc.
But this was a basic shape for these applications. I've been thinking a lot about how these basic shapes could be abstracted out so we don't solve the same problems over and over. I will write more on this topic, but today's article is about something other than this macro shape, but even more low level.
Plugin CRUD dashboards are an example of people abstracting out these basic CRUD functionalities which we always need, so we can focus on that 5% which makes our application unique. An example I can think of would be the Python Django dashboard system, the PHP Laravel backpack dashboard system (https://backpackforlaravel.com/), etc.
Although, I think we still need more solutions to abstract out the 95% work we do all the time in our applications. But it’s lovely to see some language ecosystems making huge strides here.
The Shapes of Software Logic and Algorithms
Even beneath these CRUD apps or any other kinds of apps we build, it became clear that the code we write is a repetition of a finite amount of patterns.
For example, we tend to do the following:
- Filter things a lot (Reduction?)
- Transform things a lot (Functors?)
- Summarize things a lot (Fold?)
- Do things that could fail and would want to perform other computations when those things fail or succeed (Alternatives and Monads?)
Around this time, I started to appreciate JavaScript libraries like lodash because it almost felt like a lot of people had noticed these patterns as well, and created a library that gave names to some of these patterns.
To give practical examples, there were functions like:
map: apply a part against every item in a listreduce: use a function and a starting value to summarize a list into a single valuefilter: reduce a list to a subset of the list based on a predicate function, etc.
There are many of those functions. Feel free to check out lodash.
I moved on to using many other languages like Go which didn't have similar libraries at the time and whose type system didn't make it easy to implement, but that didn't stop me from noticing we kept writing the same code over and over.
Recently, I've been coding in Rust and Haskell for work and fun, and this idea of patterns is even more appreciated in those ecosystems. But Haskell introduced me to the topic called Category Theory. I won't go into what category theory is because I'm barely scratching the surface of the topic.
But what interests me is that this topic is an entire branch of mathematics and computer science (?). Basically, a lot of smart people have also been noticing these patterns and have given names to these patterns. [I find that giving names to patterns is very important].
And the interesting thing is that these patterns extend even outside of software. For example, we filter and sort things in the real world too. But within the lens of software, I have found this amazing. Giving names to patterns means those patterns become a new abstraction, which can serve as a language we can use in describing solutions and software to each other.
For example, if I had a database of listings for my Yellow Pages application, and my client realized he needed a very important number, how he would describe what needed to be done?
Anthony, could you find out the total number of employees of all our listings in Nigeria?
And I would say sure and then try to write some code that might look like this:
func (listings []Listing) Int {
totalEmployees := 0
for _,listing := range listings {
if listing.Country == "NG" {
totalEmployees += listing.Employees
}
}
return totalEmployees
}This is a very simple example, but in this code, there are a lot of patterns that we are using explicitly, as opposed to calling the patterns by name.
What I did could be described like this:
Filter out the listings in Nigeria, then return the sum of their employees.
It's very simple when I explain what I did in English, but the code needs to make it easier to understand.
If we were using a language/library that has given names to these patterns, I would describe what we did as follows:
totalEmployees = filter (l -> l.country == "NG")
& map (.Employees)
& sumSo, now I have code that reflects what I'm trying to do by using functions that represent common patterns. We filter the listings where the country is "NG," then we map them from the listing to the employees, so now we have a list of employee counts, and then we sum up all those counts.
I appreciate this way of programming because we should be trying to do less and less as engineers. And even though these patterns in my code example are lower-level patterns, we could benefit from even more high-level patterns. For example, if we want to transform things, filter them, and sum them very often, we could give that pattern a name as well and give it its own function, which every programmer in the ecosystem can reuse instead of writing it themselves each time.
I remember, as an undergrad, we were taught about machine language and how we moved to writing assembly and from assembly to writing in languages like C. So instead of pushing around registers, we initialize a variable at a higher level and allow a compiler to interpret and translate what we mean. We were taught that the goal was to get higher and higher in abstractions. And that the future high-level language would be languages that optimized for code easier for humans to understand, as opposed to optimizing for computers to understand our code.
The idea of putting things in memory addresses and managing registers was a pattern, and we gave a name to that pattern which is now in every language. In Go, we simply do := which would initialize the variable and set the value to some memory location and then give it a name we can refer to.
We need to, as an industry, keep chasing this ideal by giving names to higher abstractions and then using those names.
Instead of a for loop that does a bunch of stuff, we can filter, sort, map, translate, or transform. And the good thing is that the field of maths and category theory has already discovered a lot of these patterns and given names to a lot of them. We simply need the industry to adapt the behaviour of introducing functions or symbols for these patterns that everyone understands. And hopefully, as an industry, we will discover and give names to more patterns that are unique to our industry.
For example, in maths, we would say 3 ∈ [1,2,3], but in programming, we might do:
func elem(num int) bool {
list := [1,2,3]
isElem := false
for _,v := range list {
if v == num {
return true
}
}
return false
}
elem(3)We should have more toolkits like lodash for most languages so no one needs to reimplement that elem and can simply reuse it; it’s a common pattern we find ourselves writing very often.
APL (A Programming Language) / BQN
I respect the APl family of languages because they take the ideas in this article even further. They created symbols to represent these common algorithms and patterns.
- Find:
⍷ - Fold and Insert:
´˝ - Group:
⊔ - Identity functions:
⊢⊣ - Join and Join To:
∾ - Mapping:
¨⌜ - Match:
≡≢ - Ordering functions:
∧∨⍋⍒ - Pair:
⋈ - Pick:
⊑ - Range:
↕ - Repeat:
⍟ - Reverse and Rotate:
⌽ - Search functions:
⊐⊒∊
This means that APL programs are usually very succinct because the author only has to describe what they want to do, using these symbols and algorithms. The developer doesn't focus on describing how the computer should do something but rather describes what the computer should do. This is the idea behind declarative programming and, in a way, also the idea behind Denotational Semantics.
There are lots of videos where you can see APL being used. Feel free to check them out:
Conclusion
Let's give names to larger and larger patterns so that, as engineers, we can focus on describing the problem we're trying to solve and not on describing how the computer should solve it. We should be more declarative.
Create more libraries to abstract out common patterns that the entire industry keeps reimplementing. And let those libraries help shape a higher level we can use to reason about our programs.
Want to Connect?
I'm building a tool that tries to learn and deeply understand the APIs
and backends we build, so it can detect issues, bugs, generate reports,
documentation, etc., automatically.
If you're building APIs, I would really appreciate if you try it out and
give me feedback.
This is the website: apitoolkit.io
