Friday, July 21, 2017

The True Worth of a Life

A life is measured not by how much money is earned but how many people are helped while the candle is lit. The people present when the candle blows out represent the true worth of a life. 

Tuesday, May 16, 2017

Thoughts on Software Development: A recent interview

I had a very productive discussion with a few consultants from India recently. The topic of choice was Software Development: specifically, how best to run teams and a good software development process. Here's a quick summary of what we discussed. Hopefully, this will be helpful to a broad audience:

Q1: Team Structure: How best to structure a team, what are the roles and responsibilities?
A: Largely, teams in tech are structured with 2 leaders: a technical lead and a manager. The role of the technical lead is managing the technical aspects of product development (code reviews, commits, design, testing, client team management and tough debugging). The role of the manager is people management (specifically and preferably practicing "'servant leadership"). The manager is responsible for morale, happiness, compensation, budgeting etc. but is not responsible for the technical direction of the team.

In some cases, it makes sense for the technical lead to also be the manager of the team, but largely, it means that one of the two roles becomes secondary to the other. In many cases, teams suffer slightly. 

Within management, there are two schools of thought: manager-leads and TL-leads. In the manager-leads scenario, the manager has the upper hand and decides prioritization of tasks (leads to conflicts with technical direction where business priorities precede technical quality). In the TL-leads scenario, manager is responsible for resourcing of projects but TL leads project execution and budgeting. Such a scenario leads to better incentives around code quality and execution but might have slips on the budget end.

Q2: So which should I choose? Manager-leads or TL-leads?
A: Manager-leads is run by several teams in the industry (Amazon being one of them). While business results are excellent, there is quite a bit of churn on the team. Churn is bad because it makes producing long term, stable infrastructure much harder (because nobody understands the system well enough).

TL-leads is the philosophy followed by Google (and other similar companies). They are OK throwing a little more money and time into problems as long as the results are excellent. Many other companies cannot afford this style of management and they resource a lot more conservatively.

Q3: What's the team composition like? How many people should be put onto a team?
A: A typical team can vary in size but it's important to understand certain ratios. Micromanagement is a big problem in the tech industry - there's always a lot of work and never enough time. In such a situation, an idle manager just slows down projects by constantly asking for updates and communicating them throughout the org.

A healthy ratio of managers to engineers is 1:7 - beyond this point, the manager really doesn't have the time or the ability to micromanage the execution of a project. A ratio of 1:5 or more is better for a TL as well for similar reasons. The absence of micromanagement makes engineers happier and empowers them by giving them more freedom of choice during the execution phase of a project.

Q4: What about Product Vision? What is the role of a Product Manager?
A: Product Management is a glue role. Product Managers do not have engineering reports. They don't have involvement in the execution phase of a project. A typical ratio for Product Managers is 1 PM for 20 Engineers (or around 1 PM for 4 teams). The reason for this large leverage is that execution takes a lot longer than ideation. By having a PM responsible for 4 teams or the work of 20 Engineers, the PM has enough leverage to keep multiple projects running simultaneously. The PM doesn't necessarily have to track the project progress himself (possibly only the key ones might matter), but he does have to answer daily execution questions from the engineers to guide the execution of the project.

Q5: What are success metrics for a team? What are the KPIs of an engineer?
A: You get what you incentivize: At Google, individual engineers are incentivized to product business impact. This means moving customer and product metrics and placing the improved metrics in context. eg. If an engineer improved suggestion quality by 2%, is that a large change or a small change? (At Google, it might be a large change if most improvements to suggestion quality are in the 0.1% range). Producing business impact is rewarded disproportionately at each level. This leads to lots of launches, several failures but a general bias to produce new things. 

Such a system dis-incentivizes routine maintenance work and as a result, product iteration and maintenance isn't as glamorous. Other important metrics are code-craftsmanship and leadership.

LinkedIn follows a similar structure: the key performance scopes are in "Leadership", "'Execution" and "Craftsmanship". In both these systems, individual productivity metrics like "code coverage", "lines of code written", "commits made", "bugs fixed" are de-emphasized because these numbers are easy to game.

Q6: On to general execution: What does the software development lifecycle look like?
A: On a high level, the software execution lifecycle is split into 4 major phases: Design, Build, Run, Maintain. Engineers are expected to design systems, they're supposed to commit infrastructure and code that creates the system, they're supposed to run it and stabilize it in production (with monitoring and alerting) and they're then supposed to maintain their code (This last part is crucial. In a healthy software ecosystem, lots of things have dependencies on each other and things break all the time; having a sense of ownership of the code you've written promotes code maintenance over time and prevents code rot because "it's someone-else's problem").

Q7: Practical matters: What makes it into a software release? How do you decide what to keep and what to cut?
A: The current best practices in software development are continuous integration and continuous deployment. Continuous integration means that every code commit is tested for correctness by running all tests on a machine that is separate and isolated from an engineer's development environment. This ensures that local dependencies don't leak inadvertently into the build / test / run ecosystem. In addition, Build / CI systems like Jenkins and others allow running the tests across a wide variety of environments / devices / versions etc. This ensures that the system is meeting build and code quality across an entire range of project platforms.

A continuous deployment system takes the "continuous integration" concept a step further: once a code commit passes continuous integration, it's automatically deployed to production. Such a style of development requires strict control of code check-ins. New code is always developed behind a feature flag and the flag is always turned off till the time the code is known to be stable. During the stabilization phase of the code, the feature flag may only be enabled for a subset of users but the continuous integration tests should ensure that all tests pass in both scenarios.

In a world of continuous integration and continuous deployment, the concept of a release is made very cheap: if it's built and the tests pass, it's ready for release. If a feature is released and found to be unstable, the feature flag is just turned off and the rest of the binary is expected to run without issues. What makes it into a "cut" is now simply: "is the code committed, are the tests there and do we have enough confidence in the code to turn the feature flag on".

Overall, this was quite a stimulating process. Hopefully, despite the length, this post will provide value to you as well.


Saturday, April 15, 2017

Freelancing for the gainfully employed

I've been professionally employed for a very long time and I was self-employed before then. Life has changed significantly for me, having come to the US from the small college town of Roorkee and my hometown of Jamshedpur.

Over the past few years, I've been knee deep in technology at Google and now recently at LinkedIn. One of the things that I've missed in both these jobs is the freedom of execution that's inherently present when you pick up work of your own volition. 

There's a lot to be said about companies and the infrastructure they provide and there's a lot more to be said about the problems they abstract away. When working for a large firm, you just don't have to deal with the nitty gritty and some skills go to rust. But every single time I've had the opportunity to pick up a project or a hobby outside of work, I've been amazed at the rapid pace of change out in the marketplace: new frameworks, new technologies and new systems keep coming up and dying out. Systems inside companies are much slower to evolve.

So, to just get a quick pulse on the market, I've realized that there's really nothing better than finishing off a quick side project and grounding myself in the current state of the world. When looking at the freelancing marketplace, there were really 3 options of interest: (too competitive, mixed bag of projects, lowest price wins), Upwork (good mix of jobs at various skill levels, top of the market projects are thin though) and Toptal (top of market, top pricing, pickier customers). 

Amongst the 3 options, Toptal seems to suit my needs beautifully. They have a Software Programmers Group and Customers come there expecting to get great work and are willing to pay top dollar for it. That's exactly what I'm looking for. I'm going to be giving Toptal a try over the next few weeks and I'll see how it goes. Wish me luck!

Excellent Book on Advanced Data Structures and Algorithms

This book draft by a competitive programmer has an excellent overview of advanced algorithms and data structures. It's dense and packed with information and covers many tips and techniques not found in any other "standard" algorithms text. 

If you're looking to learn Competitive Programming and solve some of the more advanced challenges, go through it:

Friday, March 17, 2017

A day in the life of a Professional Software Engineer

Sometimes, it's important to gain perspective over what it takes to run a successful career in your industry of choice. Growing up and in college, I didn't know very many people doing Computer Science in industry. I managed to meet a few during my internships at Microsoft and Qualcomm where I got a taste of life in Computer Science and Engineering. These were the early days of blogging, YouTube, Facebook, Android and the iPhone. Things have moved rapidly in this industry and life today looks very different from life just a decade ago. 

Before college, I was taking some monumental decisions in my life - where do I go study? what do I study? what happens after college? How do I know I'm taking the right choice? 

I'd wished then that I knew some people who had an understanding of life as a professional software engineer. I was pleasantly surprised when Jaeho (from SJSU) approached me to give perspective to life as an experienced Software Engineer in Industry. Our discussion brought up some valuable points and the content turned out to be so useful that I felt it worthy to be shared more broadly.

Here's our discussion as it took place (edited, paraphrased and summarized):

... Introduction ... Greetings ... etc.

Q: How / When did you get started with Computers?
A: When I was 6 years old. I had a neighbor who had an old 486 machine. These were the days of floppy disks (3.5 inch and 5.25 inch) and a green screen. One of the first things I learnt was how to draw with a mouse. Then I gradually moved to programming in Logo and GWBASIC. I had fun drawing staircases and circles on the screen with commands like FD 20, RT 90 instructing a virtual turtle (move forward 20 steps, turn right 90 degrees). It was fun and very encouraging. I learnt a bunch of stuff from Computer Magazines (those days they would print some GWBASIC game programs in the magazine itself). In 4th grade, my school got a computer lab and I learnt more there. Eventually, I started learning how to program C++ in 7th grade and then moved on from there. In 11th grade, I got exposed to the USA Computing Olympiad which helped me significantly improve my skills. Then, when I reached college, it was quite easy for the first few years. Things ramped up later (by the 3rd year) but I just took the toughest courses and kept going to classes and studying. I enjoyed programming and I learnt a lot. Competitive coding taught me a lot of stuff...

Q: What were the things that you found most useful to study in college?
A: Graph Theory, Linear Algebra, Computational Geometry, Data Mining / Machine Learning, Compiler Theory, Functional Programming.

Q: If you had to go back, what courses would you have taken that you didn't?
A: Few more Math courses ☺

Q: What does your typical day look like?
A: LinkedIn has flexible timings:
I come in to work at 10 AM, Leave by about 6:30 PM - 7 PM
10:00 - 10:30 : Email
10:30 - 12:00 - Coding and Code Reviews
12:00 - 12:30 - Lunch
12:30 - 6:30 Coding and Code Reviews
On some days, there are a few meetings
eg. Monday mornings - 10:00 - 12:00
Friday evening 4PM - 4:30PM
I generally like to keep the bulk of the week clear
My morning email is usually co workers and organizational emails. Most of the day, communication happens on Slack. Meetings are held usually in conference rooms... Monday mornings are team meetings of 10 - 12 people. Smaller meetings are more ad-hoc. Sometimes we do Video Conferencing over BlueJeans. LinkedIn has a Cisco based Video Conference setup.

Q: What do you work on?
A: I work in Distributed Systems infrastructure.

Q: What are the skills required for this job? What do you need to know to be successful?
A: Typically C++, Java, Distributed Systems are the basics of what is required. In my daily work, I get to use my knowledge about the Linux kernel, TCP/IP, Operating Systems, Systems Architecture, Microprocessors and von Neumann Architecture (in practice, not just theory). It's a fun job that helps me use several of my skills. 

Q: Amongst all the things that you know, what do you like best?
A: "Mechanical Sympathy" - how to work as a friend of the machine rather than an enemy. How to understand what a machine does and why it does it and how to leverage the best out of it.  A lot of this has come out of my experience with C++ and Pointers. I've learnt a lot from getting a machine to do the things that I want quickly :)

Q: How many languages should one know? Many or a few?
A: You should definitely know the Industry standard languages:
Java, Python, C++, C, Bash. I would also strongly recommend that you take the time to learn and explore Lisp and a few functional languages. eg. Haskell, ML, Scala, F#. They teach a very different way of thinking about computers.

Thank you and it was a pleasure meeting you...

Overall, this was a an amazing experience. One hour of my life might help Jaeho take several critical decisions in his life. My underlying advice has always been the same: study hard, work hard and the results will follow. It's been a relatively straightforward life for me. I've come to the US because my skills were in great demand here. I owe my life to my learning. A lot of things in life just happen, some by luck, some by circumstance. Opportunities come and go, and one just needs to be prepared to seize the day. I'm humbled and grateful to be where I am. Hopefully, others will follow a similar path in life guided by the signposts that we leave for them. 

Wishing you the best of luck: may your life be better and more enlightened as a result of the efforts of those that came before you.


Wednesday, December 07, 2016

My view on Security

People in the Software Industry seem to be quite a bit bothered by the notion of Security. Security seems to be this ooh-so-magical castle that's innately "more important" than anything else that your company does and any and all pain and sacrifice demanded at the altar of security is justified (because oooh! Users! & Privacy!!!). Well, let's face it - security is a pain... a massive pain that provides very little reward in return.

Security is like brushing. Do it once everyday and you'll be happy, your teeth will be healthy and you'll have a long life. And that's all the importance it deserves. 

ACLs are like flossing. You know you should be doing it, but you keep putting it off as long as possible. ACLs give you a sense of security, a warm fuzzy feeling of being in control. An easy way of saying - yeah, we know what we're doing - look! ACLs! And every single day, an engineer's creativity dies a little bit when he or she spends 3 hours trying to get permission to look at just one, just one instance of the data that you're trying to protect so that you can write just one line of code. 

And then there's security "best practices". Ooh, I just cry a bit every single time someone in security says "no, you can't do this" without providing a half-decent, reasonable technical alternative to the egregious hack that's already in the system to work around some other limitation of some other security system. Half-assed security systems do no good to anyone. 

There are only 2 reasonable threat models in the world: you're either being attacked by a Nation state or you're being attacked by a script kiddie. Nothing you can do will stop a determined Nation state. Electronic security be damned. They're just going to pick you up from your neighborhood coffee shop and you're going to quietly give up your passwords and hope to get your life back. If they're trying to be a bit subtler, you're just going to get key-logged. 

If you're getting attacked by a script kiddie, reasonable passwords, https, instructions about phishing and 2 factor authentication is all that you need. 

This entire hoopla about "insider threats" being a reasonable threat model in enterprise just doesn't work. If you can't trust your employees with the broadest of powers, they just can't work with each other or with the data that you're working so hard to protect. There's just no way you can protect against the exponential possibilities of compromise. The *ONLY* reasonable alternative is "trust but verify". Put in auditing, look for patterns, look for data exfiltration but for heaven's sake, just don't make daily life hard. Because your average engineer is working really, really hard to move your company forward. Making your productive workers jump through hoops kills productivity and at the end of the day, talent leaves a non productive organization and with it, your long term future disappears.

Saturday, June 25, 2016

I love my lawyer lyrics - Ofelia K

Black suit... morning... wakes up... window... 
His face... got it covered... bouquet.

I want your pheromones, messing with my hair on my time.
Feeling self destructive, I want someone who wants to fight.
Nervous bloody nose, smiling for a poloroid again.
Let me be your witness, everybody is looking for a way out, I want in.

I wanna lose control x 4

I love my lawyer, my angel in a black suit.
Looks like he's mourning, everytime he wakes up.
If you were my age or close to my age, anywhere near my age,
I would marry you.... You, whoo oo ou.

I want you in the shadows, want you in the blinding light of day.
Nothing fricking matters, tell me do you feel the same way.

I wanna lose control x 4

I love my lawyer, my angel in a black suit.
Looks like he's mourning, everytime he wakes up.
If you were my age or close to my age, anywhere near my age,
I would marry you.... You, whoo oo ou.

I wanna lose control,
Do you ever feel the same way.
Oh yeah, I gotta know... You didn't say.
I've got a night time jones, it never goes away
I wanna lose control. Tell me, do you feel the same way.

I love my lawyer, my angel in a black suit.
Looks like he's mourning, everytime he wakes up.
I love my lawyer, my angel in a black suit.
Looks like he's mourning, everytime he wakes up.

Like when the window shuts, and lands on his face,
He gets it covered like a sweet cake on a bouquet.

If you were my age or close to my age, anywhere near my age,
I would marry you. I would marry you.

Black suit... morning... wakes up... 
I would marry you... 
Window... His face... Got it covered... Bouquet.
I would marry you....
You, whoo oo ou.

I love this song! Let me know if you find any mistakes in the comments.