What You Can Learn from Building Things Yourself

We are going to discuss a quick story, then we are going to talk about the best way to learn programming and get better at anything

Storytime

There was a place I worked that had some legacy code. It was pretty old, and not written in the best way for reasons we get into, but the worst part was that it worked completely fine most of the time. There is a great danger between something being good enough and great, and good enough code generally has been the most annoying in my career to deal with.

So the good. This was a job that was required everyday to run due to upstream dependencies, and it was a streaming job, meaning it didn’t have everything available when it started, so it had to wait as data came in. Most every day it would run, read some data, and be off to the races.

However, the issue in this case is what happens when this job doesn’t go as planned. Since this was a rare occurence, but not unknown, there was protocol to handle the failure. When this job failed, you simply restarted it. Easy right? Well not quite. First you needed to kill some processes. Okay fine, kill the processes and… wait there are more processes to kill? To shorten the story, we will just say that the known way to restart this process was a little confusing and hard to guess, and definitely not documented anywhere. The answer was an interesting pkill command that used a process group and killed everything in the group.

If it isn’t obvious, the issues here is that this job streamed hours of data. So restarting it meant it needed to stream al the data from the beginning of the job again. And let’s say it crashed not to a one off issue, but because of an actual code bug. Well to know that you had to traverse many different log files, streamer.log, parser.log, splitter.log, tracker.log, etc. It was a nightmare.

Good enough is the enemy of great. But of course, no one wanted to touch it or fix it. Until one particular couple of weeks where this job started failing over and over again due to some DIFFERENT issues in another system. Basically there was something causing this job to fail in the streaming process that it wanted to write to, let’s just for simplicty call it a hardware failure that was intermittent. Sometimes the job could finish, but the chance was a lot higher that it would be interrupted and need to be restarted. So of course I was the person managing this job at that point in time. No one else who had worked on this script existed at the company, and we could not figure out how to fix the issues that were piling up.

This particular job required some overnight work in the event of failure, so I would remote in, restart the job, hope for the best. It would often fail. Restart it goes. Until one night in particular I had been up a long time at this point babysitting the job, and I am cussing to myself and just overall cranky as one would be dealing with work in the late hours of the night instead of playing video games (good screen, bad screen), and I just got fed up and decided. How hard can this be to re-write?

By the next morning I had a simple prototype, and I can tell you that for me working on that problem was incredibly cathartic. By the end of the week I had code ready to test out in parallel to see if I could replace this process. It saved myself a ton of time and headache, as restarting now was more of a “resume”, processing each of the smaller chunks in sequence rather than one large stream. Of course, there were complications, we had lots of smaller hiccups along the way to try to replicate some of the exact behavior we needed in python rather than in the bash script, and even some cool pybinding to c++, but the result after a few weeks of work was a system that could tolerate the failures we expected. I never had to stay up to deal with that script again. Not sure if that has held true to this day, as I left the company, but I learned more about the system fixing something that was annoying than I had been learning for months on a part of the system I was familiar with.

What can you take away from this story? You can do the same thing in your life. Sometimes things are harder than we expect, sometimes they are easier, but rarely do we walk away with nothing when we confront a difficult task, at work or in your day to day life, it’s the same thing.

You can learn anything if you try

The best way to learn any sort of skill, from home rennovation to business, is to practice that skill. The saying an expert comes at 10,000 hours has a sense of truth to it. You might get decent at something, maybe even good, from a small amount of effort, but truly practicing a skill is the only way to become a master.

Programming is the same way. Do you want to make websites? Make a website. It won’t be easy the first time, but the more times you do it, the easier it gets. Along the way you might even run into some people who can teach you thinks not related to websites. Such as building and releasing changes to your website. That’s called devops. How do you learn that? You guessed it. You practice devops. Go try to make a build pipeline. It will probably not be very good. But the next time you do it, it will be a little better.

We are in an age where you can find FREE open source software and read it. You don’t even need a library card. You can freely look at the code, and it even comes with instructions on how to run it. Does it not make sense looking at the code? Okay then learn how to read code. There is a reason you can’t duplicate experience. It’s because intuition is built up slowly over time as you learn and fail.

Babies have to learn physics before they can learn to talk. They have to crawl and navigate the world around them. How do they learn it? By being terrible at it for a while. But our brains have this amazing ability to slowly adjust over time to small amounts of information. Fitness, coding, reading, comedy, anything in the world you want to be good at, you have to DO it.

Being afraid to try something prevents us from greatness. What is that thing you are ignoring right now that you need to confront? What are you afriad of starting because you might be bad at it? Guess what you will be bad at first. But that’s the fun part.

If you enjoyed this post and want more like it, consider subscribing to the email list via the buttons below. If you have ideas for future posts, shoot me an email at jared@jaredbwelch.com . Also I am learning by doing this blog, so feedback is welcome if you have ideas on how to make this blog better.

• Jared