Annoyingly qcon links aren’t embeddable. This was a great talk about an internal tool made at Netflix that is used by developers and production support engineers (sre, operations, customer support) to learn about errors.
Tracing becomes especially important when you have many services involved in processing a single request. Putting together a picture of what happened when logs and metrics are scattered across log categories and dashboards (could be 1 per service in the worst case) is hard.
Edgar has a global view. It was important that all telemetry sources were fed into edgar. It wouldn’t have been a tool people could rely on if there were gaps.
Another important design decision was the sampling rate. Collecting traces is hard. (aka Resource intensive in a system in terms of ram) But less than 100% tracing means when you go to look for one, there’s a chance it won’t be there. The suggestion was to collect 100% for a small, critical subset of traffic. (eg /checkout)
A really great article in the AWS builders library about making service to service messaging more reliable with tradeoffs!
First ask is this request retry-able? The work must be idempotent!
Note to self : Re-read Release It! (Nygaard)
Mail servers will cross-check your SMTP server’s advertised HELO hostname against the PTR record for the connecting IP address, and then check that the returned name has an address record matching the connecting IP address. If any of these checks fail, then your outgoing mail may be rejected or marked as spam.
So, you need to set all three consistently: The server’s hostname and the name in the PTR record must match, and that name must resolve to the same IP address.
Note that these do not have to be the same as the domain names for which you are sending mail, and it’s common that they are not.
Reverse dns records (ptr): A discussion of how they’re used. The first comment is the most helpful (Included here for posterity :))
A tool for analyzing messages in postfix’s various queues. eg What domain’s they’re going to, and how long they’ve been there
# To list messages all messages by domain
# sudo qshape
T 5 10 20 40 80 160 320 640 1280 1280+
TOTAL 1714 4 5 141 4 256 22 56 1218 7 1
gmail.com 1714 4 5 141 4 256 22 56 1218 7 1
# Only shows messages in the deferred queue
# sudo qshape deferred
# .. Looks like above but filteredGet me a list of messages currently in the active queue (and possibly other queues eg hold)
# what version am I running? cat /etc/system-release # is selinux enabled? getenforce # put selinux into permissive mode setenforce 0
# list installed packages yum list installed # list package updates yum check-update # list package updates (security only) yum check-update --security # apply updates yum update # apply security updates yum update --security # get info about a package yum info python3
Backup a specific mongodb. This is generally slower than physical backups but good for grabbing very specific subsets of data from big db *this version doesn’t include db users in the dump file
mongodump --archive=a.gz --gzip --db <dbname>If you want users too, don’t specify the db (the admin db will be included in the dump and it has users)
mongodump --archive=a.gz --gzipRestoring a dumpfile
mongorestore --archive=a.gz --gzip # load averages uptime # kernel messages can be helpful sometimes dmesg | tail # rolling process, memory stats vmstat 1 # rolling cpu states on multicore systems mpstat -P ALL 1 # rolling ps aux (only shows non-idle processes) pidstat 1 # rolling disk performance metrics iostat -xz 1 # available memory, used, swap free -m # network visibility sar -n DEV 1 sar -n TCP,ETCP 1 # running processes in a system top
Resources