10 years ago I wrote a public-facing website with a separate private web app for a small business. The technology I used was horrible, the processes I used to deploy it were completely manual, and the design was very much the UI version of hello world. The solution, from a technical perspective, is objectively bad, so why am I proud of it? Because it’s been helping run a small business for 10 years and has solved real-world problems for them.
The business
My parents run a small photography business. Their work is mostly event photography, where they photograph sports and music events. Their business model for many years was one of sale or return. For example, they would photograph a kids’ rugby tournament, getting the team and individual shots for all the teams playing. They would then get the address of the coach. After the event, they would print all the photos and put them into cardboard frames. They sent all the photos out to the coach. The parents of the kids would either buy the photos or send them back via the post. In terms of getting paid, my parents would have to wait for physical money or cheques to be sent back via the post. Yep, physical money in an envelope!
If you have experience with family businesses, you can never really escape them. As a family member, you often get brought in to help them run aspects of the business. When I was starting my career as a software engineer, my tasks evolved to building out accounting software or spreadsheets, or designing and wiring up their small office LAN network. After a few years working in tech, I persuaded my parents we could digitise aspects of their business: sending actual coins in the post is probably not the best idea in 2014. Little did I know that would result in some shonky code still running in production that my parents (in their 70s) still use to run their business.
The tech
What’s the worst thing you have ever done for money?
Written PHP
Below is some technical detail; if any of it doesn’t make sense, don’t worry; just skip to the end of the road section.
This was before every single podcast was advertising Squarespace and all the cool kids were using WordPress! WordPress is a content management system (CMS) that is written in PHP. An SQL database backs it. It was a good solution for building a website where you needed your non-technical users to control the content or run something standard like an e-commerce shop. This could all be achieved by adding open-source or paid plugins. The issue I had was the use case was not something that traditional plugins were going to solve. My parents needed a way to track sale and return items and allow customers to pay for them online. This required a custom solution to match their workflow. As a result, I had to write some custom PHP to enhance the WordPress website. You might think I should have written a full plugin for WordPress to do this job - or realised it would require writing PHP, so found another solution. I opted for neither, I decided to find a plugin that allowed you to add snippets of PHP directly into pages on your website:
These snippets were written directly in the browser, with no source control, no IDE, and no way of testing it. If you made a mistake in your PHP (which I often did), the website would simply crash and display a white screen. The code was saved in the CMS and put into the SQL database. That’s right, my code was saved as a text column in an SQL table. The scripts I wrote were not just snippets they were full webpages with lots of complexity dealing with payments and PayPal integrations. Not what that plugin was ever intended for.
In addition to a WordPress website with Edd’s special PHP snippets. I wrote a UI in React and a backend that has evolved from Java to Javascript. The UI I wrote using material design components. At the time, I thought I was a proper UI developer now, but looking back, it’s quite clear I was just stringing together a whole bunch of components with no real understanding of how they fit together.
The deployment is a single EC2 in AWS: not a cost-effective way to run a website. Also, there’s no form of dev environment, just a single prod box. Sometimes the IP of this box changes, and I have to update the DNS records manually. We hold photographs on the EC2, so I’ve got an EFS mount that gets lost if the box ever gets restarted, and I have to remount it manually. To get the code onto the box, I build it on my laptop and push the full build artefacts into git. I then simply do a pull on the EC2 to get the latest code. I run the node web application using the npm package forever, which just runs a process on the box:
https://www.npmjs.com/package/forever
Logging isn’t a thing, no containerisation, just raw javascript files running on a VM. The code used to be java; I then changed it to javascript, and later updated some of it to use typescript. So the code base is a mix of conventions that have organically evolved over the years. The only person with any chance of editing or maintaining it is me.
Support headaches
To my horror, something went wrong last year and the UI couldn’t talk to the backend. One of the few good things I added was protecting the API routes using auth0:
However, the authentication stopped working. This did not sound like a straightforward fix, and I had actual dread thinking about it. I logged onto the box crossing my fingers it would still let me SSH on. After some looking around, I saw the backend was getting SSL issues. After a lot of investigation, it turned out the box was running node 8… yes, node 8, end of life December 2019. The actual CAs bundled with node 8 didn’t have the new CA auth0 was using, so it couldn’t validate the JWT. I tried to install the latest version of Node. It complained it wasn’t supported, so I had to use node 14. (Still being supported for a few months at least.) When I installed node 14 the box ran out of space halfway through. I had to clear down some space just hoping I wasn’t deleting anything needed on my snowflake EC2. Hurrah! It worked, and the auth sprang back to life. Panic over! Actual panic because if that didn’t work, I wouldn’t have known what to do.
End of the road
I’m happy to say my parents are doing their last couple of event photography jobs this year. They are a bit mad, still running 2 businesses when they are both 74.This means I will be able to switch off this EC2 workhorse. This will provide me a lot of joy because I’m confident there are glaring security issues with it, and it will remove the fear I get when I see an email from my Mum with the subject ‘website help’. Knowing I will have to log onto this relic and somehow nurse it back to life is something I don’t enjoy. One wrong move, and I know I won’t be able to recover it.
So why am I proud of it?
I look at this solution and cringe at all the terrible decisions I made. I’m slightly disappointed with the previous me when I must delve back into it to fix things. However, the reason I’m proud of it is that it’s helped solve problems for a real-life business. My parents’ cash flow improved by weeks and sometimes months. They had actual data they could use to make decisions about jobs. It also allowed them to pivot during COVID to sell digital products online.
The whole solution is hard to support and doesn’t scale, but it doesn’t need to. It is only me supporting this thing. So I could spend lots of time improving it and improving it, but in reality, all of that work would only save me an odd couple of afternoons during the year I need to fix things. I learnt a lot by building this solution from start to finish, gathering requirements, designing the system, building it, testing it with users. I would do it all differently today, but that’s only with the benefit of hindsight and with all of the learning I got from this bad technical solution.
Technology is just a tool for generating value
The important lesson for me is my parents benefited from what I built because it helped their business. Building it in another language, deploying it via serverless technologies or adding artificial intelligence or quantum computing might be fun and exciting from a technical perspective. Still, the driver should always be - what are the business benefits!
Now I appreciate there could also be business benefits to different designs - for example, it might be cheaper to run using a different design. In that case, that is the business benefit that drives the technical decision.
Technology should never be the primary driver for your decision-making. It should always follow the order:
What impact will this have on our users
What technology can we use to deliver this impact
I know most engineers would love to rewrite stuff they have written. There are sometimes good business benefits; for example, a service might have lots of technical debt, which means it's very costly to make changes. However, if you aren’t changing that service very often, there could be minimal business benefit to doing the rewrite.
At Gemba we are paid to improve the working lives of the people that make a difference. We do this by delivering that value through technology. For example:
We don’t deliver an ElasticSearch database - we allow our users to find their answers quicker.
I don’t deliver a crappy WordPress website - I help businesses get their money quicker
To be honest, I will never work on a WordPress site again, but I wouldn’t turn down helping a business to get its money quicker. Product books are full of this message, for example:
Fall in love with the problem, not the solution
As a techie, I think it’s sometimes hard to think in these terms. One of the reasons we work in tech is that we enjoy technology and learning and trying new things. There is nothing wrong with that. However, it is important to remember why we are doing it. At Gemba, we work to help important people, and if we do our jobs right, we can help amplify the difference they can make in the world. So my one takeaway from this blog I have illustrated below using stick figures in top hats:
We get paid to make this happen:
This is what we should measure, and this is what we should care about. Technology and the solutions we build are only the enablers. I have focused this blog on technology; but our solutions don’t always have to be technical. The aim is to amplify the difference our users make in the world - how we do that is down to you.
We have been giving a lot of thought and debating at a board level how we look to measure these impacts at a company wide level. How we can track the cumulative impact Gemba has. What I find exciting about this is we can apply the same to teams and individuals so we can celebrate the impacts we are having each day.