Tuesday, May 21, 2019

Managing Humans with Math

I was in a funky corner of the internet, reading about reinforcement learning when I chanced upon this article that compares a number of reinforcement learning algorithms. Since we as humans are glorified neural networks amenable to reinforcement learning and companies are nothing but hierarchical relationships between humans, it was very interesting to go through that article with a business point of view.

Long story short, the article figures out various conditions in which the algorithms are able to break down a task recursively, the optimality convergence conditions and the "knowledge" required of the lowest level agents to learn successfully. From my reading of the article, I'm most drawn to the "Options" style of management: work with high quality people, give them maximum freedom to act and step in only at decision points.

This is the style of management that I'd grown used to at Google (and that I've seen work really well there) and is quite in contrast to other managements that I have experienced. Not to say that the other managements were ineffective, they just took different routes to get to the same goals.

In summary: prefer the "Options" style of management, work with smart people, give them all the freedom they want and deserve, provide a strong learning function and clear feedback on how they're performing. Step in only at "Choice" points and then too defer to people with context of the problem. Most decisions are reversible, so just take one and let's keep going!

Wednesday, February 13, 2019

Hermetic MySQL in the modern world

So you're looking to run a MySQL docker instance without any environmental dependencies? Here's how to do that:

$ docker run --name mysql-test -e MYSQL_ALLOW_EMPTY_PASSWORD=true -e MYSQL_DATABASE=testingdb -e MYSQL_USER=scott -e MYSQL_PASSWORD=tiger -p 3306:3306  mysql/mysql-server:latest

Documentation on the various environment variables are located here:

To connect to your MySQL instance without depending on my.cnf:

$ mysql --no-defaults --host --port 3306 --user scott --password --protocol tcp testingdb

That's it. You have a running instance of mysql from a docker image and you're connecting to it with a generic mysql command line client. Have fun.

Saturday, February 02, 2019

Alerts should be actionable a.k.a.: Do not email on success!

Following the Unix philosophy: Do one thing, do it well and be quiet about it.

In software engineering, if you're writing a system that's useful and suddenly, one day, you think it's nice to notify users via email that their useful thing is being done, you're making a mistake. 

Emails from software systems should be actionable: if a system is sending an email to a user, it should be helpful, provide enough context about who it is, where it's running, who owns / runs it and what the problem is that requires human attention. Ideally, the alert email should clearly specify the next steps and the dashboards that can be used to ensure that the problem is fixed.

The worst offense of the system is to send out success emails. This fails on 2 counts:
1. Success emails are not actionable - if I read a success email, I am informed and I promptly create a Gmail filter to never see another success email from the system again. The system made me do active work to ignore it.
2. Success emails are not trackable - if I want to see what's the ratio of success to failures of the system, Gmail is a terrible way to do it. From first hand experience, measuring alert volume over time in Gmail is a time sink. Please build a dashboard and make the world a better place. Your future self will thank you.

The best alert emails are those that just tell me the commands to fix the problem. Oh happy me, I don't have to read a runbook, talk to people, poke around dashboards to see what the problem is. Run a few commands and presto, it's fixed.

Take some time and make your life better, don't send success emails and send actionable emails on failure.


Thursday, January 31, 2019

How to debug a crashing docker container

If want to run your docker process with some tweaks because it's crashing in your docker container and causing the container itself to stop (without giving you a way to inspect the files on the image), here's the magic command to start it with just bash.

(I found this after quite a bit of hunting on the internet, the magic flag is --entrypoint and don't forget the -s at the end)

Here's a sample command:
docker run -it --entrypoint /bin/bash  $IMAGE -s

Sourced from:

Wednesday, January 23, 2019

How to be an Effective Engineer?

Read this book by Edmond Lau: 

Highly recommended. His experiences parallel my own. The book is written engagingly, quick to read and succinct in its delivery. 

Friday, January 04, 2019

Understanding Divye

I typically share the contents of this document the first time I meet someone I'm going to be working with for a long time.

Style of communication:
Direct and clear. I don't do well reading between the lines. If you would like me to know something, please say something.

What is important to me:
High quality code and a highly productive, functional team. I don't believe in posturing. I believe in commitments and delivery. Delivering more and talking less is a good way to show impact.

What kind of leader I'm trying to become:
One who can grow people. I'm driven by Mission, Vision, Values.

What I value in an organization:
Transparency, Trust and Integrity. If you have a problem, speak up. Don't ever go behind someone's back. Bring problems up together so that I can see both sides represented fairly. Be direct.

What I seek in people I work with:
Ability to execute and potential to grow. I will endeavor to make you a better engineer or leader.

How I believe our industry operates:
Our industry is a factory - our deliverables are software and running services. Computer Science is a predictable industry with unpredictable schedules. The when might be unclear but the deliverable should be rock solid.

My planning philosophy:
Plan your work and deliver on your own benchmarks. Look out and adjacent.

My operating style:
I will work with you and always give you the "why". On the rare occasion where "because I say so" is required, I will let you know so.

My background:
Born and brought up in India. Son of two doctors, have a sister who is a doctor. Small town guy.

My favorite quotes:
"If I do what I say I will do, I guess that means you can trust me. If I don't, I guess that means you can't."

My biggest weakness:
I evolve, I don't let you know that fast enough.
If you talk to me and I tell you I support X and 24 hours later, I'm supporting !X, it means that I've gained new information that has changed my mind. Talk to me and clarify what changed. I have "strong opinions, loosely held".

  • What's working?
  • What's not working?
  • What's needed?

Things I would like you to know:

Tuesday, December 04, 2018

AI Expo 2019: Tim Jurka (LinkedIn, Director Feed AI) - Part 4 of 4

I recently attended the AI Expo 2019 at the Santa Clara Convention Center. Notes are from my understanding of the talk. Any errors are mine and mine alone.

LinkedIn: A look behind the AI that powers the LI feed

Tim Jurka (Dir. Feed AI)

The talk was focused on the objectives of LinkedIn's Feed. The talk was focused to a high level (exec) audience. While I was familiar with the space, the objective function formulation and presentation was interesting:

The recommendation problem for LinkedIn is maximizing Like/Comment/Share CTR + downstream network activation (virals) + encouraging new creators.

Problem Formulation:

P(click) + P(viral) * (alpha_downstream + alpha_creator * e ^ (- decay * E[num_response_to_creator])

alpha_downstream accounts for downstream effects; alpha_creator penalizes popular creators to induce diversity.

General approaches (Toolbox):

Multi Objective Optimization (ads vs organic content).

Logistic Regression: Features, Embeddings and Decision Trees (XGBoost for Feature Importance), occasional pruning 

Auto tuning of weights of the MOO to correct for drifts in accuracy of the component models (to meet a product goal).

Running the Team:

The goal of the team is to maximize: Successful Experiments / Total Experiments Run

Two approaches: maximize successful experiments, minimize unsuccessful experiments.

Maximize successful experiments:

1. Hire the best talent

2. Increase the total number of experiments being run online.

3. Automate deployments, parameter tuning, retraining, rebasing, ramping to maximize developer throughput.

Minimize unsuccessful experiments:

1. Replay models over historical data to figure out whether they would perform better than the current model before moving to online.

2. Compute actual business metrics, Determine precision @ 1, precision @ top3/top5 over a randomized sample of data.

3. Use bandits to figure out how to be intelligent about collecting data and exploiting the current model.

4. AI for Ai: Auto retrain and evaluate models. Identify promising features and ramp online. Find optimizations for existing models automatically. Highlight promising variants to engineers.