Link roll: 2020-12-15

  • Strava, The Boring Option: A story about a schema design decision (width of an id field in one of their tables) that worked great from 2009 – 2020 but then needed to change. A 32bit unsigned, monotonically increasing id field is good for 4b unique values before it wraps around. Depending on how many of these you’re using, it could last a long time. It did for Strava. The covid19 pandemic meant all their users were using their service way more than normal normal which accelerated the need for re-work here. They were pragmattic about what they did. They considered different datastores to store this data (huge table, lots of read/write activity on it) but in the end decided they knew mysql and were comfortable with it. They found a way using their current datastore (and reserved the right to consider different ones in the future but they had a problem to solve today). Great story!


Bryan Cantrill on automation, complexity, microservices, human-machine systems

Such a great talk. “Don’t think because you have an alert for something that you’re protected.” “The more alerts you have, the more information overload you may have for the operator.”

  • Ideas
    • Respect distributed systems
    • Debuggability in production
    • Debugging == new knowledge about the way a system works

Rewatched James Hamilton’s Keynote from Reinvent 2016

This may very well be my favourite conference talk ever. So many big ideas presented in 1.5 hours and articulated well and with enthusiasm!


Advent of Code 2020

Day 1

Watching this stream:

Watching LizTheGrey’s stream. She’s fantastic. She’s going through computer science principles that factor into the choices she’s making

Today I learned

  • Thinking about how much work we’re doing is an important exercise. Avoiding repeatedly computing the same thing makes a lot of sense (relates to algorithmic complexity)
  • If you are making an assumption about input (eg no negative values, no zeros) you can make this explicit by adding an assertion so that your program fails fast (with hopefully a helpful error) when that assumption is invalidated

Day 5

  • This one is a bit of a bugger. My solution uses a binary search strategy with low, high pointers that shift as you get closer to your goal. Not hard to write but was a bit fiddly
  • Liz (and others I can see) did something much simpler. Set ‘B’ -> 1 and ‘R’ -> 1 in the input, ignored ‘F’ and ‘L’ (or set these to zero) and somehow with only a little more energy got the answer. What the hell is going on here?

Have to think about this one a bit more

Day 7

Had to remind myself today to not try to overly anticipate what part b will ask me to do. More often than not the complexity of part a goes up and what I think is coming next doesn’t.

Keeping it simple is an important principle in systems design!

Day 13

Alright so part 2 I don’t understand the answer for. I have to think about this one a bit more. It’s only a few lines long …


  • Revisit day 13, part b



Linkroll: 2020-11-30

  • DNS load balancing: This company is using DNS load balancing to good effect for some of it’s traffic. Not machine-to-machine type api traffic it sounds like. (Works ok for human clients that honour ttl). 2 big problems with dns load balancing are 1) uneven distribution of load (a problem for load balancers too but you at least have some say in how requests are forwarded), and 2) how are failed servers removed from the pool?
  • Cloudflare postmortem (Byzantine failure in etcd cluster): Interesting. A few distributed systems bolted together to create a bigger one. Each individual component is “fault tolerant” on it’s own but there emerge new kinds of failures when they are connected to eachother. Keep it boring for as long as you possibly can! This is usually a lot longer than you think

Let’s Get Lazy

Lots to think about here. The way think about organizing logic, process management, deferred code execution, …

One of my favourite tech talks I found in 2020


Getting to Activites and KPIs

Jeff talked about this today and I like the framework! Coming up with OKRs and high level goals is so hard. I’ve only been involved in the process once or twice so I don’t have a lot of experience but I remember it being fairly abstract (a bunch of people get in a room and talk about what they’ve learned in the past year or so – in terms of customers, the business, competition, etc)

Happy that our exec team are running out in front for the rest of us and that they know what they’re doing! 🙂


Scaling Yourself (by Scott Hanselman)

Really great talk about training yourself to focus on things that matter an let go of everything else. This kind of thinking actually helps you be more effective.



Starting a bit of a focused read this week into ETL pipelines – what they are, the abstractions, common practice around organizing logic, etc – as the topic is starting to come up at work. (We have recently hired a business analysts / revenue management type person who’s keen on being able to ask adhoc, deep questions about our users and how they’re using the system.)

Here goes!

ETL stands for extract, transform, load:

  • Extract: the first step in a pipeline is always sensory, or collection. It can work at different levels (individual events, or batch) and involves ingesting data up front (and possibly filtering) for use by later stages
  • Transform: Sometimes data needs to be cleaned up, enriched, de-duped, etc before it is delivered to a final destination (which may itself be only the beginning of another pipeline!). An example could be taking a record that is origin system formatted, and converting it into something more destination system compatible
  • Load: Concerned with delivery to systems where data will rest either for analysis or to become input into downstream pipelines

Word Cloud

extract // transform // load // data partitioning // ingest // enrich // airflow // batch // event driven // event stream // idempotent // workflow engine // real-time //


  • Airflow’s creator talks about it like it’s a workflow engine
  • ETL type processing
  • Data warehousing. Data data, and clean, well structured to support adhoc querying
  • This is an apache project
  • It’s a python application


Random Thoughts

  • When you’re small, direct integrations to specific data sinks / destinations is fine
  • Real-time can be more complex than batch ETL pipelines (but is it really?)
    • One consideration here is what happens if / when the transactional db and data warehousedb (destination) drift …


SpringOne 2020

We’re heavy users at Spring at work and the more time I spend with it, the more I am appreciating everything it’s doing for me. It’s been around for many years and represents the combined experience building distributed apps of thousands of developers!

SpringOne was a virtual event this year because of times we live in (Covid 😢) and there are a bunch of videos that came out of it that I’m going to start watching …

  • What is Spring? is a great starter talk for the conference. Historical context for the framework which is super important for me because I’m coming back to Java and Spring after a few years away from the ecosystem
    • Spring Framework: Inversion of control, dependency injection, mvc, testing. The framework is the foundation on which everything else builds on. It’s an integration tool that let’s us compose and combine disparate technologies reasonably in code
    • Security, batch, integrations, data are the other bedrock components
    • Boot came a bit later and combines all these pieces tastefully with sensible defaults and world class capabilities
    • Start every project at
  • Batching for the Modern Enterprise
    • Batch computing: “working on finite amount of data without interaction or interruption”
    • Jargon: job, step, tasklet, chunk
    • Jobs are a series of steps. Each looks like : read some data -> process it -> then write something
    • Scaling: He talked about 5 different methods … I can only remember a few 😀 threads, partitioning, chunking, …
  • Spring Boot Observability
    • I really need to look at this more closely. Such observability power with very little energy from a dev
    • Including the spring boot actuator dependency lights up capabilities around standard jvm usage metrics, jmx things if present, and a framework for custom metrics (timers, counters, gauges)
  • Spring Security Patterns
    • Very good primer for using the spring security module
    • We kept coming back to security by default
    • Java based configuration of spring security looks great. You can create a UserDetailsService to identify users and help establish sessions. You can also leverage a built in capability to run a resource server
    • The url space of an application can be secured from a central place in code (SecurityConfig)
      • /location
    • Basic auth (username, password) is intentionally slow and expensive. A password encoder is designed to take a lot of compute and time which isn’t great for an api server under load. Using an auth token (oauth2?) gives you the same security but is faster to verify
  • A Deep Dive into Spring Application Events
    • Spring has a built in way to publish events in business logic with event registration. Super flexible.
    • Events can be produced and consumed in process as needed to help with maintaining bounded contexts and a loosely coupled architecture
      • Events can be sent outside the origin process too to a message queue or some such as well
    • Really neat stuff!
  • Security Patterns for Microservice Architectures
    • List of things we should be doing in our services
    • Dependency scanning, openid connect for authentication / authorization, secrets handling, secure coding practices
    • Book: Secure by Design has a few chapters worth a skim
      • 12 factor, cloud based design techniques
      • DDD shows up and object immutability
      • Failure handling looks nice
    • Light, fun romp through several topics with pointers for going deeper