Seems like an important thing to keep in mind about complexity …

The container operator’s manual

The container operator’s manual

  • good for
    • good for reproducing perfectly certain environments
    • good for stateless apps
    • good for test environments because they’re easy to clean up after
    • good for upgrades <-very easy to do
  • bad for stateful apps … don’t containerize your database <-this is hard
    • you’re not google
    • don’t
  • takes longer than you’d expect … containerization is hard
  • give it to ops … LOL! they already do deployment && config management && incident response && monitoring && infrastructure management && security … why not give the container project to them

Srecon 2019: Building a scalable monitoring system

Molly Struve

https://www.usenix.org/conference/srecon19emea/presentation/struve

The monitoring platform that grew organically over time (can be overwhelmed by the number of different tools): New relic, honey badger (exception reporting), pagerduty, cron, dashboards, elastalert

How alerts were delivered to engineers: slack notifications, sms, email, phone

Alerts inconsistent

  • some reported data but didn’t suggest action
  • some needed immediate action

Eventually overhauled. Goals of alerting system:

  • consolidate monitoring to a single place (what does this mean?)
    • kenna used datadog for this. hooks into all other tools.
    • she’s meaning this in the alert manager sense? using different tools for logs, metrics, traces, etc
  • alerts are actionable. (no alert should allowed to be ignored) can put non actionable things away from actionable things.
  • alerts are mutable (turn them off when needed … eg when we’ve already acknowledged a problem)
    • for a set period of time (should come back on)
  • track alert history. does this condition happen regularly?

Behaviours:

  • if an alert goes off you have to acknowledge
  • here’s how you mute, and when, and how long, and how to dump alerts that aren’t helpful
    • CASE: context heavy, actionable, symptom based, evaluated
  • here’s where the monitoring tool is and how to use it
  • developers should help make monitoring better

Michael Smith’s sweet potato vegetarian chili with cinnamon sour cream

INGREDIENTS

  • 2 tablespoons vegetable oil
  • 1 large onion, chopped
  • 1 green bell pepper, seeded and chopped
  • 8 garlic cloves, thinly sliced
  • 1 tablespoon cumin seeds
  • 1 tablespoon chili powder
  • 1 tablespoon dried oregano
  • 2 cups fresh or frozen corn
  • 1 398-millilitre can black beans, rinsed and drained
  • 1 398-millilitre can kidney beans, rinsed and drained
  • 1 796-millilitre can whole tomatoes
  • 1 sweet potato, peeled and finely diced
  • 1 tablespoon canned chipotle chilies in adobo sauce, chopped
  • 1/2 cup sour cream
  • 1 teaspoon cinnamon
  • 1 or 2 sprinkles salt
  • 1 cup tender cilantro sprigs
  • 2 green onions, thinly sliced

METHOD

Heat the oil in a soup pot over medium-high heat. Toss in the onions and green pepper and sauté, stirring frequently, until the vegetables begin to brown (6 to 8 minutes). Stir in the garlic, cumin seeds, chili powder and oregano. Reduce the heat to medium and cook, stirring, until the spices are very fragrant (another 2 minutes or so).

Stir in the corn, black beans and kidney beans. Add the juice from the canned tomatoes, then coarsely chop the tomatoes and add them as well. Add the sweet potatoes and chipotle chilies. Bring to a boil, then reduce the heat so the liquid is just barely simmering. Simmer, stirring frequently, until the sweet potatoes are tender and the chili begins to thicken (20 to 25 minutes).

Meanwhile, stir together the sour cream and cinnamon. Just before serving, season the chili to your taste with salt. Ladle into serving bowls and top with the sour cream and a tangle of cilantro and green onions.