Scaling Yourself (by Scott Hanselman)

Really great talk about training yourself to focus on things that matter an let go of everything else. This kind of thinking actually helps you be more effective.



Starting a bit of a focused read this week into ETL pipelines – what they are, the abstractions, common practice around organizing logic, etc – as the topic is starting to come up at work. (We have recently hired a business analysts / revenue management type person who’s keen on being able to ask adhoc, deep questions about our users and how they’re using the system.)

Here goes!

ETL stands for extract, transform, load:

  • Extract: the first step in a pipeline is always sensory, or collection. It can work at different levels (individual events, or batch) and involves ingesting data up front (and possibly filtering) for use by later stages
  • Transform: Sometimes data needs to be cleaned up, enriched, de-duped, etc before it is delivered to a final destination (which may itself be only the beginning of another pipeline!). An example could be taking a record that is origin system formatted, and converting it into something more destination system compatible
  • Load: Concerned with delivery to systems where data will rest either for analysis or to become input into downstream pipelines

Word Cloud

extract // transform // load // data partitioning // ingest // enrich // airflow // batch // event driven // event stream // idempotent // workflow engine // real-time //


  • Airflow’s creator talks about it like it’s a workflow engine
  • ETL type processing
  • Data warehousing. Data data, and clean, well structured to support adhoc querying
  • This is an apache project
  • It’s a python application


Random Thoughts

  • When you’re small, direct integrations to specific data sinks / destinations is fine
  • Real-time can be more complex than batch ETL pipelines (but is it really?)
    • One consideration here is what happens if / when the transactional db and data warehousedb (destination) drift …


SpringOne 2020

We’re heavy users at Spring at work and the more time I spend with it, the more I am appreciating everything it’s doing for me. It’s been around for many years and represents the combined experience building distributed apps of thousands of developers!

SpringOne was a virtual event this year because of times we live in (Covid 😢) and there are a bunch of videos that came out of it that I’m going to start watching …

  • What is Spring? is a great starter talk for the conference. Historical context for the framework which is super important for me because I’m coming back to Java and Spring after a few years away from the ecosystem
    • Spring Framework: Inversion of control, dependency injection, mvc, testing. The framework is the foundation on which everything else builds on. It’s an integration tool that let’s us compose and combine disparate technologies reasonably in code
    • Security, batch, integrations, data are the other bedrock components
    • Boot came a bit later and combines all these pieces tastefully with sensible defaults and world class capabilities
    • Start every project at
  • Batching for the Modern Enterprise
    • Batch computing: “working on finite amount of data without interaction or interruption”
    • Jargon: job, step, tasklet, chunk
    • Jobs are a series of steps. Each looks like : read some data -> process it -> then write something
    • Scaling: He talked about 5 different methods … I can only remember a few 😀 threads, partitioning, chunking, …
  • Spring Boot Observability
    • I really need to look at this more closely. Such observability power with very little energy from a dev
    • Including the spring boot actuator dependency lights up capabilities around standard jvm usage metrics, jmx things if present, and a framework for custom metrics (timers, counters, gauges)
  • Spring Security Patterns
    • Very good primer for using the spring security module
    • We kept coming back to security by default
    • Java based configuration of spring security looks great. You can create a UserDetailsService to identify users and help establish sessions. You can also leverage a built in capability to run a resource server
    • The url space of an application can be secured from a central place in code (SecurityConfig)
      • /location
    • Basic auth (username, password) is intentionally slow and expensive. A password encoder is designed to take a lot of compute and time which isn’t great for an api server under load. Using an auth token (oauth2?) gives you the same security but is faster to verify
  • A Deep Dive into Spring Application Events
    • Spring has a built in way to publish events in business logic with event registration. Super flexible.
    • Events can be produced and consumed in process as needed to help with maintaining bounded contexts and a loosely coupled architecture
      • Events can be sent outside the origin process too to a message queue or some such as well
    • Really neat stuff!
  • Security Patterns for Microservice Architectures
    • List of things we should be doing in our services
    • Dependency scanning, openid connect for authentication / authorization, secrets handling, secure coding practices
    • Book: Secure by Design has a few chapters worth a skim
      • 12 factor, cloud based design techniques
      • DDD shows up and object immutability
      • Failure handling looks nice
    • Light, fun romp through several topics with pointers for going deeper


Hello, WordPress!

I’ve tried a couple different static site generators for my blog in the last few years, but I always missed how easy it was to write, and upload my favourite photos to WordPress.

I think I’m done looking around at others. Quite happy where I am 🙂

Here’s a pic of Carmen and I in February 2020, just before the Covid-19 first wave lockdown began. Feels like forever ago. So lucky to have been able to visit London one last time this year before the world went sideways …


How F’cked Up is Your Management

Underground club session #2

Obvious to you is not the same as obvious. I wish I had gentle reminders like this more often. 🙂 How do I get better at this?

  • 2-way conversations vs 1-way monologs
  • Ask more often “Does that make sense?”
  • Share what I see from my PoV
  • Context
  • As it relates to business strategy whenever I have it
  • High frequency

Disfunctional product orgs creep up on you sometimes. eg Eng does lots of work and releases something but nobody else understands the value. Getting this team wrong has consequences. I’ve seen some too. Do we have a strategy or do we always seem to be doing the things the yelliest people are asking for?

Ideas vs Execution – Ideas are cheap, execution is hard. Ideas needed 5-10% of the time … hire for execution!! Vision is important but a small part of role typically. Good product leadership looks like:

  • fewer features
  • directly customer impacting (high value)
  • instrumented so you know how people are using them <- app metrics !

Elevated product conversation – from “Ship this by Friday” -> “How do we modify onboarding to increase engagement for different kinds of users?”

“Guilt based management” :/

Negative news / feedback is hard to deliver. It’s best given asap and should be direct. It should not be sugar coated or surrounded by other messages. It’s also a good idea to talk about ways to do things differently.

There are at least a couple ways to evaluate people : on effort, and on outcomes. Author thinks this depends on how senior you are. I’m not sure it’s this straightforward. Shouldn’t it always be about outcomes? Lots of energy plus bad results is kinda crappy. Experience should factor into the size of work chunk you hand off to a person.

A team to stop looking for the “elegant” or “right” solution and just do the crappy necessary work seems critical. It’s an indicator of a team’s likelihood to be successful in the long run. Be willing to grind. Do stuff that doesn’t scale. Compound interest! 1% better every day. 🙂

Underground club session #1

You and everybody else needs work and life outside work

Lots of talk about diversity in this book. It clicked for me at gcloudnext. “You can’t be what you can’t see.” Most speakers were women, and / or of different cultural backgrounds and beliefs. I didn’t get what I was seeing at first. And who knows if I hadn’t just been reading that chapter …

Understand the business the company you work for is in. Good advice. You should be able to work with product to understand requirements and the why. Helps to make better decisions in the field.


Mars Rover Curiosity: An Inside Account from Curiosity’s Chief Engineer

I remember thinking to myself, “The people at NASA have always seemed superhuman. When you look at successful people, you don’t always see what it took for them to get there. You see outcomes. There was very likelylots of doubt, failure, and terror getting there! 🙂


“7 minutes of terror”: the time between the lander entering the martian atmosphere and when it lands when it’s stops communicating with earth altogether and you hope it makes it but so many things can go wrong

A powerful question: Has life ever appeared anywhere else in the universe? Mission statements can bring people together.

Chapter 2

Constraints can help us get creative: Pathfinder was an earlier mars mission that had some interesting ones:

  • Small
  • Planned basic research
  • New launch / landing platform (Combination of heat shields, parachutes, airbags.)
  • Relatively small budget

A mistake where a program was getting values in units it wasn’t expecting caused total mission failure (When you’re doing something for the first time expect some things will go wrong. Build contingencies and backup plans where the investment in time, money, and effort is justified …)

Chapter 3

A visual aid to quickly capture ideas for landing the rover:

![Mars EDL Possibilities]({{ site.url }}/images/mars-edl-possibilities.png)

There was a lot of time between project kickoff and concensus where people just weren’t ready to make a decision. Lots of people involved

Multiple project teams were thinking and pitching ideas to people who would select the best

Chapter 4

A mini meltdown from the chief engineer 🙂

Chapter 5

Mid way through the schedule, people were very worried about being able to slow the craft down. A major requirement / risk to the project didn’t have a solution or even the beginnings of one.

Many months of brainstorming and exploring proposals deeper. Another caveat for them is that no matter how much investigation you do you can never be 100% sure because you can’t actually test on Mars … 🙂

Chapter 6

Engineers != good communicators often. This limits ability to build concensus and ultimately create change. Important to be able to convince others your idea is workable. Presentations often go down in flames because of style not content …

There is probably a good tension between engineering and the business. How much a business is willing to invest in a solution and how much risk it’s willing to take on due to lack of investment in ideal solution. Note ideal solution is usually untested and thus fiction anyways.

Lots of opposition in the face of doing something new.

Chapter 7

Within the broader NASA org, the team was considered overconfident and cocky. Past success. Humility was required or they’d have trouble having an impact.

Chapter 9

Getting other countries to contribute by way of providing certain instruments allowed the team to scale. Sounds like working groups: a community of passionate individuals formed around a single concern that’s often temporary in nature.

Important to work on community alignment. Listening. Empathy.

How do you get new people involved? Established players often crowd the field and aren’t always enthusiastic about working with untested, unproven parteners.

Chapter 11

Complex systems are like an enormous painting. Hundreds of artists must each get their own little corner of the canvas. It is project management’s and the systems engineer’s job to make sure that each part of the canvas is allocated to the appropriate artist. When adjacent artists have widely different perspectives or palettes, the whole painting might have to be scrapped. It is the chief engineer’s job to stand back, study the entire painting and resolve disasters in advance, which usually means working with the artists to solve the problem between them before the paint has dried.

A point is reached where Rob sits down with a trusted mentor about the impossibility of the task ahead. There just wasn’t enough time to solve all the problems they could see. He was told to find data and build a case. He couldn’t talk to management about wishes and feelings. Having somebody you can talk to and ask for advice from seems crucial.

Be careful about conveying down messages to the team. It’s likely to make them want to give up.

Chapter 13, 14

Bit if an emotional rollercoaster this was for him

Fear of failure. Physical health reprocussions. 2-3 hours a week of physical training was prescribed by Rob’s doctor.

Chapter 16 “Gremlins”

This is all about planned game days. Tiny faults injected into otherwise normal data the team would have to work to investigate and resolve. Super important.

Complicated, highly choreographed sequence of events that was the landing must be rehearsed

I should read this chapter again. Bloody brilliant.


The Manager’s Path

Chapter 4 Managing people

“Entry level job” – Chapter 4 first quote, Marc Hedlund 🙂

Key points

  • 1on1s
  • onboarding
  • feedback on career growth, progression towards goals, areas for improvement, praise
  • working with team memeber to identify areas for learning and helping them grow into them (project work, mentoring, external learning)

Create a 30/60/90 day plan: basic goals (these I like as starter ones … will use for coops) learning code, architecture, how to write a pull request, shipping code (phased deployment, testing, etc)

New hires provide an opportunity to hear about their initial experience with the team. What was great? What could’ve been better?

Jeff is like Sharell and Beth. He communicates intent, motivation and gives room to let me figure out a path through the work. He’s there if / when he’s needed and can hop in when necessary. eg Release 4 needed a cross team effort in devtools to get to a rel4 dev cluster from the GUI.

Chapters 7, 8, 9

“Develop a sense of where you need to dive in deep”

“You know you have a problem when … someone falls back to playing individual contributor”

What are the things we should be doing other than being a technical contributor? <- this isn’t going to be our biggest contribution anymore it seems … listening, empathy, caring, understanding, bridging between different people / ideas, mentorship, architecture, goal setting with team members, road mapping, building culture, health, productivity. This is going to get uncomfortable fast. Introvert!


An Astronaut’s Guide to Life

I’ve been meaning to add notes to my blog about this book for too long. I loved it. Here’s what I took away:

Chapters 1 and 2

    Be ready work hard and enjoy it.
    Getting to space is a bonus.

His worldview allowed him to love all the prep without the guarantee of getting his prize.

    Look at the person you want to be.
    What do you admire about them.
    What can you do to make yourself more like that person?
    Lasting change happens in tiny increments every day.
    Aim to move the needle just a little bit today.

Chapter 3 was about dealing with fear

Acquire knowledge experience competence through practice:

  • Simulations are run constantly <- game days
  • What could possibly go wrong

Avoid emotional, flight response (this is trained out of astronauts)

If a fire starts on the space station they don’t reach for fire extinguisher -> they start descending a complex decision tree that was developed on the ground (they also have a test lab – the pool – where they can work through novel situations in an environment as close to space as they can get here)

Frameworks for responding to the unexpected in space:

  • Ooda: Observe orient decide act
  • NASA: warn gather work Warn == some sort of alert is sounded Gather == everybody gets together to respond to event Work == work the problem

Simulations identify gaps in our knowledge. They give confidence to actors that they have thought through many scenarios and prepared, and help to put the mind at ease.

Chapter 7 On Family

This is so true …

    Family vs career
    Risk is family could be left behind while you're reaching for goals
    Must always be aware of this
    Plan to be present even when you aren't
    Family is a partnership. Other side should be respected or they can be taken for granted.
    Is this thing you want to do really needed for goals or just what you want to do?

On being a new member of a team

    Aim to be zero
    Entering a new situation try to be neutral , observe, learn from people already where you want to be
    Aim for a tiny beneficial contribution
    -1, 0, +1
    Minus- actively make collective experience worse
    0- no big problem or benefit... Stability
    Plus- big benefit

    Plus maybe comes later when you've been doing good reliable work with team

How to prepare for what you don’t know

    Plans vs preparing for unknown
    Plan a,b,c,...
    What would we do if... ? type questions
    Dream up scenarios that are unlikely but not impossible
    Give yourself tools to respond to unforeseen
    Confidence building maybe
    Try to imagine what it will be like to operate in new environments
    Game days ?