A few jobs ago, I worked at company that collected data from disparate sources, then processed and deduplicated it into spreadsheets for ingestion by the data science and customer support teams. Some common questions the engineering team got were:

• Why is the data in some input CSV missing in the output?
• Why is data in the output CSV not matching what we expect?

To debug these problems, the process was to try to reverse engineer where the data came from, then try to guess which path that data took through the monolithic data processor.

This is the story of how we stopped doing that, and started storing references to all source data for every piece of output data.

Read more

When writing code to interact with a third party like a SaaS provider, you typically need to write both code to handle the API the third party uses, and translate their data model into your data model. One thing I’ve found that makes this easier is to do each step separately: First write a library to interact with their API using their own data model, then separately write code to translate between your data model and theirs.

Read more

A friend of mine recently convinced me to take a class on quantum computing. The class just started, but I’m already learning interesting things.

For example, one student linked to a lecture on quantum physics which explains it as a generalization of probability theory which allows for complex …

Read more

Car side and rear-view mirrors are designed to give you a view around your car, but they typically don’t show you the two spots directly to the side of your car (the “blind spots”). Luckily, you can increase the field of view of your side mirrors by over 200% and virtually eliminate blindspots by adding aftermarket blind spot mirrors.

Read more

I’m part of a small Discord server and thought it would be funny to make a Geoguessr-style game where you get presented with a random interesting message from the server and have to guess when, where and by who it was posted.

Read more