Categories
systems

Linkroll

  • DNS load balancing: This company is using DNS load balancing to good effect for some of it’s traffic. Not machine-to-machine type api traffic it sounds like. (Works ok for human clients that honour ttl). 2 big problems with dns load balancing are 1) uneven distribution of load (a problem for load balancers too but you at least have some say in how requests are forwarded), and 2) how are failed servers removed from the pool?
  • Cloudflare postmortem (Byzantine failure in etcd cluster): Interesting. A few distributed systems bolted together to create a bigger one. Each individual component is “fault tolerant” on it’s own but there emerge new kinds of failures when they are connected to eachother. Keep it boring for as long as you possibly can! This is usually a lot longer than you think
Categories
code

Let’s Get Lazy

Lots to think about here. The way think about organizing logic, process management, deferred code execution, …

One of my favourite tech talks I found in 2020

Categories
work

Getting to Activites and KPIs

Jeff talked about this today and I like the framework! Coming up with OKRs and high level goals is so hard. I’ve only been involved in the process once or twice so I don’t have a lot of experience but I remember it being fairly abstract (a bunch of people get in a room and talk about what they’ve learned in the past year or so – in terms of customers, the business, competition, etc)

Happy that our exec team are running out in front for the rest of us and that they know what they’re doing! 🙂

Categories
code

Scaling Yourself (by Scott Hanselman)

Really great talk about training yourself to focus on things that matter an let go of everything else. This kind of thinking actually helps you be more effective.

Categories
code

ETL

Starting a bit of a focused read this week into ETL pipelines – what they are, the abstractions, common practice around organizing logic, etc – as the topic is starting to come up at work. (We have recently hired a business analysts / revenue management type person who’s keen on being able to ask adhoc, deep questions about our users and how they’re using the system.)

Here goes!

ETL stands for extract, transform, load:

  • Extract: the first step in a pipeline is always sensory, or collection. It can work at different levels (individual events, or batch) and involves ingesting data up front (and possibly filtering) for use by later stages
  • Transform: Sometimes data needs to be cleaned up, enriched, de-duped, etc before it is delivered to a final destination (which may itself be only the beginning of another pipeline!). An example could be taking a record that is origin system formatted, and converting it into something more destination system compatible
  • Load: Concerned with delivery to systems where data will rest either for analysis or to become input into downstream pipelines

Word Cloud

extract // transform // load // data partitioning // ingest // enrich // airflow // batch // event driven // event stream // idempotent // workflow engine // real-time //

Airflow

  • Airflow’s creator talks about it like it’s a workflow engine
  • ETL type processing
  • Data warehousing. Data data, and clean, well structured to support adhoc querying
  • This is an apache project
  • It’s a python application

Articles

Random Thoughts

  • When you’re small, direct integrations to specific data sinks / destinations is fine
  • Real-time can be more complex than batch ETL pipelines (but is it really?)
    • One consideration here is what happens if / when the transactional db and data warehousedb (destination) drift …

Categories
code

SpringOne 2020

We’re heavy users at Spring at work and the more time I spend with it, the more I am appreciating everything it’s doing for me. It’s been around for many years and represents the combined experience building distributed apps of thousands of developers!

SpringOne was a virtual event this year because of times we live in (Covid 😢) and there are a bunch of videos that came out of it that I’m going to start watching …

  • https://springone.io/post-event/sessions/what-is-spring
  • What is Spring? is a great starter talk for the conference. Historical context for the framework which is super important for me because I’m coming back to Java and Spring after a few years away from the ecosystem
    • Spring Framework: Inversion of control, dependency injection, mvc, testing. The framework is the foundation on which everything else builds on. It’s an integration tool that let’s us compose and combine disparate technologies reasonably in code
    • Security, batch, integrations, data are the other bedrock components
    • Boot came a bit later and combines all these pieces tastefully with sensible defaults and world class capabilities
    • Start every project at start.spring.io
  • https://springone.io/post-event/sessions/to-batch-or-not-to-batch
  • Batching for the Modern Enterprise
    • Batch computing: “working on finite amount of data without interaction or interruption”
    • Jargon: job, step, tasklet, chunk
    • Jobs are a series of steps. Each looks like : read some data -> process it -> then write something
    • Scaling: He talked about 5 different methods … I can only remember a few 😀 threads, partitioning, chunking, …
  • https://springone.io/post-event/sessions/spring-boot-observability
  • Spring Boot Observability
    • I really need to look at this more closely. Such observability power with very little energy from a dev
    • Including the spring boot actuator dependency lights up capabilities around standard jvm usage metrics, jmx things if present, and a framework for custom metrics (timers, counters, gauges)
  • https://springone.io/post-event/sessions/spring-security-patterns
  • Spring Security Patterns
    • Very good primer for using the spring security module
    • We kept coming back to security by default
    • Java based configuration of spring security looks great. You can create a UserDetailsService to identify users and help establish sessions. You can also leverage a built in capability to run a resource server
    • The url space of an application can be secured from a central place in code (SecurityConfig)
      • /location
    • Basic auth (username, password) is intentionally slow and expensive. A password encoder is designed to take a lot of compute and time which isn’t great for an api server under load. Using an auth token (oauth2?) gives you the same security but is faster to verify
  • https://springone.io/post-event/sessions/a-deep-dive-into-spring-application-events
  • A Deep Dive into Spring Application Events
    • Spring has a built in way to publish events in business logic with event registration. Super flexible.
    • Events can be produced and consumed in process as needed to help with maintaining bounded contexts and a loosely coupled architecture
      • Events can be sent outside the origin process too to a message queue or some such as well
    • Really neat stuff!
  • https://springone.io/post-event/sessions/security-patterns-for-microservice-architectures
  • Security Patterns for Microservice Architectures
    • List of things we should be doing in our services
    • Dependency scanning, openid connect for authentication / authorization, secrets handling, secure coding practices
    • Book: Secure by Design has a few chapters worth a skim
      • 12 factor, cloud based design techniques
      • DDD shows up and object immutability
      • Failure handling looks nice
    • Light, fun romp through several topics with pointers for going deeper

Categories
travel

Hello, WordPress!

I’ve tried a couple different static site generators for my blog in the last few years, but I always missed how easy it was to write, and upload my favourite photos to WordPress.

I think I’m done looking around at others. Quite happy where I am 🙂

Here’s a pic of Carmen and I in February 2020, just before the Covid-19 first wave lockdown began. Feels like forever ago. So lucky to have been able to visit London one last time this year before the world went sideways …

Categories
books

How F’cked Up is Your Management

Underground club session #2

Obvious to you is not the same as obvious. I wish I had gentle reminders like this more often. 🙂 How do I get better at this?

  • 2-way conversations vs 1-way monologs
  • Ask more often “Does that make sense?”
  • Share what I see from my PoV
  • Context
  • As it relates to business strategy whenever I have it
  • High frequency

Disfunctional product orgs creep up on you sometimes. eg Eng does lots of work and releases something but nobody else understands the value. Getting this team wrong has consequences. I’ve seen some too. Do we have a strategy or do we always seem to be doing the things the yelliest people are asking for?

Ideas vs Execution – Ideas are cheap, execution is hard. Ideas needed 5-10% of the time … hire for execution!! Vision is important but a small part of role typically. Good product leadership looks like:

  • fewer features
  • directly customer impacting (high value)
  • instrumented so you know how people are using them <- app metrics !

Elevated product conversation – from “Ship this by Friday” -> “How do we modify onboarding to increase engagement for different kinds of users?”

“Guilt based management” :/

Negative news / feedback is hard to deliver. It’s best given asap and should be direct. It should not be sugar coated or surrounded by other messages. It’s also a good idea to talk about ways to do things differently.

There are at least a couple ways to evaluate people : on effort, and on outcomes. Author thinks this depends on how senior you are. I’m not sure it’s this straightforward. Shouldn’t it always be about outcomes? Lots of energy plus bad results is kinda crappy. Experience should factor into the size of work chunk you hand off to a person.

A team to stop looking for the “elegant” or “right” solution and just do the crappy necessary work seems critical. It’s an indicator of a team’s likelihood to be successful in the long run. Be willing to grind. Do stuff that doesn’t scale. Compound interest! 1% better every day. 🙂

Underground club session #1

You and everybody else needs work and life outside work

Lots of talk about diversity in this book. It clicked for me at gcloudnext. “You can’t be what you can’t see.” Most speakers were women, and / or of different cultural backgrounds and beliefs. I didn’t get what I was seeing at first. And who knows if I hadn’t just been reading that chapter …

Understand the business the company you work for is in. Good advice. You should be able to work with product to understand requirements and the why. Helps to make better decisions in the field.

Categories
books

Mars Rover Curiosity: An Inside Account from Curiosity’s Chief Engineer

I remember thinking to myself, “The people at NASA have always seemed superhuman. When you look at successful people, you don’t always see what it took for them to get there. You see outcomes. There was very likelylots of doubt, failure, and terror getting there! 🙂

Notes

“7 minutes of terror”: the time between the lander entering the martian atmosphere and when it lands when it’s stops communicating with earth altogether and you hope it makes it but so many things can go wrong

A powerful question: Has life ever appeared anywhere else in the universe? Mission statements can bring people together.

Chapter 2

Constraints can help us get creative: Pathfinder was an earlier mars mission that had some interesting ones:

  • Small
  • Planned basic research
  • New launch / landing platform (Combination of heat shields, parachutes, airbags.)
  • Relatively small budget

A mistake where a program was getting values in units it wasn’t expecting caused total mission failure (When you’re doing something for the first time expect some things will go wrong. Build contingencies and backup plans where the investment in time, money, and effort is justified …)

Chapter 3

A visual aid to quickly capture ideas for landing the rover:

![Mars EDL Possibilities]({{ site.url }}/images/mars-edl-possibilities.png)

There was a lot of time between project kickoff and concensus where people just weren’t ready to make a decision. Lots of people involved

Multiple project teams were thinking and pitching ideas to people who would select the best

Chapter 4

A mini meltdown from the chief engineer 🙂

Chapter 5

Mid way through the schedule, people were very worried about being able to slow the craft down. A major requirement / risk to the project didn’t have a solution or even the beginnings of one.

Many months of brainstorming and exploring proposals deeper. Another caveat for them is that no matter how much investigation you do you can never be 100% sure because you can’t actually test on Mars … 🙂

Chapter 6

Engineers != good communicators often. This limits ability to build concensus and ultimately create change. Important to be able to convince others your idea is workable. Presentations often go down in flames because of style not content …

There is probably a good tension between engineering and the business. How much a business is willing to invest in a solution and how much risk it’s willing to take on due to lack of investment in ideal solution. Note ideal solution is usually untested and thus fiction anyways.

Lots of opposition in the face of doing something new.

Chapter 7

Within the broader NASA org, the team was considered overconfident and cocky. Past success. Humility was required or they’d have trouble having an impact.

Chapter 9

Getting other countries to contribute by way of providing certain instruments allowed the team to scale. Sounds like working groups: a community of passionate individuals formed around a single concern that’s often temporary in nature.

Important to work on community alignment. Listening. Empathy.

How do you get new people involved? Established players often crowd the field and aren’t always enthusiastic about working with untested, unproven parteners.

Chapter 11

Complex systems are like an enormous painting. Hundreds of artists must each get their own little corner of the canvas. It is project management’s and the systems engineer’s job to make sure that each part of the canvas is allocated to the appropriate artist. When adjacent artists have widely different perspectives or palettes, the whole painting might have to be scrapped. It is the chief engineer’s job to stand back, study the entire painting and resolve disasters in advance, which usually means working with the artists to solve the problem between them before the paint has dried.

A point is reached where Rob sits down with a trusted mentor about the impossibility of the task ahead. There just wasn’t enough time to solve all the problems they could see. He was told to find data and build a case. He couldn’t talk to management about wishes and feelings. Having somebody you can talk to and ask for advice from seems crucial.

Be careful about conveying down messages to the team. It’s likely to make them want to give up.

Chapter 13, 14

Bit if an emotional rollercoaster this was for him

Fear of failure. Physical health reprocussions. 2-3 hours a week of physical training was prescribed by Rob’s doctor.

Chapter 16 “Gremlins”

This is all about planned game days. Tiny faults injected into otherwise normal data the team would have to work to investigate and resolve. Super important.

Complicated, highly choreographed sequence of events that was the landing must be rehearsed

I should read this chapter again. Bloody brilliant.

Categories
books

The Manager’s Path

Chapter 4 Managing people

“Entry level job” – Chapter 4 first quote, Marc Hedlund 🙂

Key points

  • 1on1s
  • onboarding
  • feedback on career growth, progression towards goals, areas for improvement, praise
  • working with team memeber to identify areas for learning and helping them grow into them (project work, mentoring, external learning)

Create a 30/60/90 day plan: basic goals (these I like as starter ones … will use for coops) learning code, architecture, how to write a pull request, shipping code (phased deployment, testing, etc)

New hires provide an opportunity to hear about their initial experience with the team. What was great? What could’ve been better?

Jeff is like Sharell and Beth. He communicates intent, motivation and gives room to let me figure out a path through the work. He’s there if / when he’s needed and can hop in when necessary. eg Release 4 needed a cross team effort in devtools to get to a rel4 dev cluster from the GUI.

Chapters 7, 8, 9

“Develop a sense of where you need to dive in deep”

“You know you have a problem when … someone falls back to playing individual contributor”

What are the things we should be doing other than being a technical contributor? <- this isn’t going to be our biggest contribution anymore it seems … listening, empathy, caring, understanding, bridging between different people / ideas, mentorship, architecture, goal setting with team members, road mapping, building culture, health, productivity. This is going to get uncomfortable fast. Introvert!