tag:blogger.com,1999:blog-341737772024-02-29T21:39:38.695-08:00DeeKaying @ PinterestFinding people you never knew you needed. :)Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.comBlogger138125tag:blogger.com,1999:blog-34173777.post-11087306484435852992024-02-29T11:49:00.000-08:002024-02-29T11:49:12.612-08:00BUCK2 import Maven dependencies<p>Meta's BUCK2 doesn't yet have Java and Android support (see <a href="https://github.com/facebook/buck2/issues/394" target="_blank">github issue 394</a>), however, it's useful to have a script to autogenerate remote_file and prebuilt_jar rules from Maven coordinates, if the code did work. If you're trying to get BUCK2 to compile Java, save yourself the trouble just yet and use <a href="https://bazel.build/" target="_blank">Bazel</a> - it should work for you. </p><p>That said, here's a quick attempt at a script to import maven jars for BUCK. If it's useful to you, feel free to use.</p><div style="text-align: left;"><br /></div>
<pre>
#!/usr/bin/env bash
#
# Import a Maven dependency based on the coordinates.
#
#
# Print debug log
# set -x
# Exit on error
set -e
if [[ $# != 1 ]]; then
echo Usage: $0 com.pinterest.optimus:library:version
echo
echo Imports the given Maven target into a BUCK file in third_party/
exit 1
fi
# Change to third-party directory
cd `dirname $0`
MVN_COORDS=$1
GROUP_ID=$(echo $MVN_COORDS | cut -f 1 -d:)
ARTIFACT_ID=$(echo $MVN_COORDS | cut -f 2 -d:)
VERSION=$(echo $MVN_COORDS | cut -f 3 -d:)
MAVEN_CENTRAL_URL_BASE=https://repo1.maven.org/maven2
if [[ $VERSION == "" ]]; then
VERSION=$(curl -s ${MAVEN_CENTRAL_URL_BASE}/$(echo $GROUP_ID | tr . /)/${ARTIFACT_ID}/maven-metadata.xml | grep '<latest>.*</latest>' | xargs echo | tr -d latest\<\>/)
echo Latest Version: $VERSION
fi
echo Importing $MVN_COORDS
echo $GROUP_ID $ARTIFACT_ID $VERSION
JAR_SHA1=$(curl -s ${MAVEN_CENTRAL_URL_BASE}/$(echo $GROUP_ID | tr . /)/${ARTIFACT_ID}/$VERSION/${ARTIFACT_ID}-${VERSION}.jar.sha1)
SRC_JAR_SHA1=$(curl -s ${MAVEN_CENTRAL_URL_BASE}/$(echo $GROUP_ID | tr . /)/${ARTIFACT_ID}/$VERSION/${ARTIFACT_ID}-${VERSION}-sources.jar.sha1)
JAVADOC_JAR_SHA1=$(curl -s ${MAVEN_CENTRAL_URL_BASE}/$(echo $GROUP_ID | tr . /)/${ARTIFACT_ID}/$VERSION/${ARTIFACT_ID}-${VERSION}-javadoc.jar.sha1)
NAME=$(echo ${GROUP_ID} | tr . _)_$(echo ${ARTIFACT_ID} | tr .- _)
echo JAR_SHA1=${JAR_SHA1} SRC_JAR_SHA1=${SRC_JAR_SHA1} NAME=${NAME}
echo "
remote_file(
name = '${NAME}_sources',
out = '${ARTIFACT_ID}-${VERSION}-sources.jar',
url = 'mvn:$MAVEN_CENTRAL_URL_BASE:$GROUP_ID:$ARTIFACT_ID:src:$VERSION',
sha1 = '${SRC_JAR_SHA1}'
)
remote_file(
name = '${NAME}_jar',
out = '${ARTIFACT_ID}-${VERSION}.jar',
url = 'mvn:$MAVEN_CENTRAL_URL_BASE:$GROUP_ID:$ARTIFACT_ID:jar:$VERSION',
sha1 = '${JAR_SHA1}'
)
remote_file(
name = '${NAME}_javadoc',
out = '${ARTIFACT_ID}-${VERSION}-javadoc.jar',
url = 'mvn:$MAVEN_CENTRAL_URL_BASE:$GROUP_ID:$ARTIFACT_ID:javadoc:$VERSION',
sha1 = '${JAVADOC_JAR_SHA1}'
)
prebuilt_jar(
name = '${NAME}',
binary_jar = ':${NAME}_jar',
source_jar = ':${NAME}_sources',
javadoc_url = ':${NAME}_javadoc'
)
"
</pre>Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-37112876801052311212024-02-26T15:54:00.000-08:002024-02-26T15:54:38.419-08:00How to correctly compile Thrift using Maven<p>The <b>org.apache.thrift.tools:maven-thrift-plugin</b> is old and largely unmaintained (last updated 2013). It should not be used anymore. The correct plugin to use is <b>org.apache.thrift:thrift-maven-plugin</b> which was last updated in 2017.</p><p> The main problem with the maven-thrift-plugin and the thrift-maven-plugin is that it doesn't correctly integrate with the maven generate-sources command (at-least for Maven 3.8.4). The way to fix that is to change the generated sources to be placed in the src/main directory of the maven package. The `<i><generator>java</generator></i>` is to fix some of the backward incompatible changes on the thrift compiler's command line (usually shows up as an error: "<span style="background-color: white; color: #0c0d0e; font-family: -apple-system, "system-ui", "Segoe UI Adjusted", "Segoe UI", "Liberation Sans", sans-serif; font-size: 13px;">unknown option java:hashcode"). </span></p><p>Adding the following to your pom.xml will allow a process where the thrift files can be manually updated with a local command the rest of the compile is straightforward. </p><p><br /></p><blockquote><div style="background-color: #2b2b2b; color: #a9b7c6; font-family: 'JetBrains Mono',monospace; font-size: 9.0pt; white-space: pre;"><span style="color: #179387;"><plugin><br /></span><span style="color: #179387;"> </span><span style="color: grey;"><!-- Note: trigger rebuild of java classes from thrift with<br /></span><span style="color: grey;"> $ mvn thrift:compile<br /></span><span style="color: grey;"> or alternatively<br /></span><span style="color: grey;"> $ mvn org.apache.thrift:thrift-maven-plugin:compile<br /></span><span style="color: grey;"> --><br /></span><span style="color: grey;"> </span><span style="color: #e8ba36;"><groupId></span>org.apache.thrift<span style="color: #e8ba36;"></groupId><br /></span><span style="color: #e8ba36;"> <artifactId></span>thrift-maven-plugin<span style="color: #e8ba36;"></artifactId><br /></span><span style="color: #e8ba36;"> <version></span>0.10.0<span style="color: #e8ba36;"></version><br /></span><span style="color: #e8ba36;"> <configuration><br /></span><span style="color: #e8ba36;"> </span><span style="color: #54a857;"><thriftSourceRoot></span>${basedir}/src/main/thrift<span style="color: #54a857;"></thriftSourceRoot><br /></span><span style="color: #54a857;"> <thriftExecutable></span>/usr/local/bin/thrift<span style="color: #54a857;"></thriftExecutable><br /></span><span style="color: #54a857;"> <generator></span>java<span style="color: #54a857;"></generator><br /></span><span style="color: #54a857;"> <outputDirectory></span>${basedir}/src/main/java<span style="color: #54a857;"></outputDirectory><br /></span><span style="color: #54a857;"> </span><span style="color: #e8ba36;"></configuration><br /></span><span style="color: #e8ba36;"> </span><span style="color: #179387;"></plugin><br /></span><span style="color: #6e7ed9;"></plugins></span></div></blockquote>Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-39298017189012457762023-07-27T02:53:00.000-07:002023-07-27T02:53:14.223-07:00Quality documentation requires Diátaxis<p>It was an eye opening day for me when I first learnt that there's a theory around how documentation should be organized. That grand unified theory of documentation is called <a href=" https://diataxis.fr" target="_blank">Diátaxis</a>. </p><p> <a href="https://diataxis.fr">https://diataxis.fr</a></p><p>Diátaxis states that documentation belongs in one of 4 quadrants:</p><p></p><ol style="text-align: left;"><li>Tutorials</li><li>How To Guides</li><li>Explanations</li><li>Reference documentation</li></ol><div>They illustrate the idea using the following image:</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-p79zLpT-Q5LPvkSRCj-tMajbINeYqXKY_ww5BUzY9_z37GPXTuSKQ6yWNNhVB3dKI1BHx1zHUxwIwB-ugKZcETFbL6TvxhrVdnJ3b_791Cx0Dt_n5TMFi6q5t6K0FOaTSZplGJ-7MEeWZeHVkz3BwEQHhMt1izzC-F5yKdHOTdk9kdqpj5s1DA/s1802/diataxis.webp" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1008" data-original-width="1802" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-p79zLpT-Q5LPvkSRCj-tMajbINeYqXKY_ww5BUzY9_z37GPXTuSKQ6yWNNhVB3dKI1BHx1zHUxwIwB-ugKZcETFbL6TvxhrVdnJ3b_791Cx0Dt_n5TMFi6q5t6K0FOaTSZplGJ-7MEeWZeHVkz3BwEQHhMt1izzC-F5yKdHOTdk9kdqpj5s1DA/w640-h358/diataxis.webp" width="640" /></a></div><br /><div>You'll see that the space of documentation was broken down along 2 axes:</div><div><ol style="text-align: left;"><li>Personal State: Understanding or Executing</li><li>Task state: </li><ol><li>For Understanding: Theoretical vs Real-World </li><li>For Executing in the Real World: Processes (User Flows) vs Tasks (micro actions)</li></ol></ol><div>The 2x2 breaks down as follows:</div><div><ul style="text-align: left;"><li>Understanding a Theoretical Scenario: (What's my model of the world? How is this system solving it?)</li><ul><li>Architecture Explanations</li><li>Architecture Diagrams</li><li>Data Flow Diagrams</li><li>Algorithms & Theory</li><li>Design Docs</li><li>Problem statement / Problem setting</li></ul><li>Understanding a Real-World Scenario: (Walk me through how I can solve a problem on the happy path, without error conditions).</li><ul><li>Tutorials & Training</li><ul><li>Hands on guides against a contrived end goal (eg. Deploying Hello World on K8s)</li></ul><li>User Stories / User walkthroughs</li><ul><li>Common cases being run through without any error cases triggering. (eg. Scaling a cluster up and down)</li></ul></ul><li>Doing a Process: (eg. Development, Onboarding, Debugging, Deployment, Monitoring)</li><ul><li>How to documentation: Walkthroughs of Real-World User Journeys</li><ul><li>This is often the most important documentation set and the last to be written.</li><li>This documentation covers the main "User-Facing Processes" that allow a user to derive value from the system. </li></ul></ul><li>Doing a Task (Micro Action): (eg. Triggering a system, Issuing a command line)</li><ul><li>Man Pages</li><li>Command line reference guides</li><li>API Pages</li><ul><li>These identify the limits of the system, are very low level and show one "how" to force a particular system component to do what you want. </li></ul></ul></ul></div><div>Most software teams do a great job on the "Understanding a Theoretical Scenario" -- the design docs, the data flow diagrams, the architectural diagrams are usually in place. Most software teams also do a great job on Doing a Task (the micro actions) -- usually the corresponding code or class is documented, there might be some API documentation, some Javadoc and the reference parameters on the API contracts are well specified. Where most teams lose their users is that they don't proceed to the next level of maturity: the Tutorials and the How-Tos. </div></div><div><br /></div><div>This level of maturity can be assessed with the following hierarchical pyramid of documentation:</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjW5ZPz28nqdMeqo2FOWdT2RvbVbGcYeopNrTOSrvY-_cwLeThZoiGcULdDrYUrl7mFOyKO3RomRhlD9Cu88hORF56FhKD4YEKbXz8RUXCEVoQG0ZwRMeF9Ycj7vPurIMNYbOavAGJsVrOpHyqXzyNed3ULOKgzIqdp7YbetnZ7-obRS4h_YCD4og/s651/Screen%20Shot%202023-07-27%20at%203.15.27%20PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="440" data-original-width="651" height="432" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjW5ZPz28nqdMeqo2FOWdT2RvbVbGcYeopNrTOSrvY-_cwLeThZoiGcULdDrYUrl7mFOyKO3RomRhlD9Cu88hORF56FhKD4YEKbXz8RUXCEVoQG0ZwRMeF9Ycj7vPurIMNYbOavAGJsVrOpHyqXzyNed3ULOKgzIqdp7YbetnZ7-obRS4h_YCD4og/w640-h432/Screen%20Shot%202023-07-27%20at%203.15.27%20PM.png" width="640" /></a></div><br /><div><br /></div><div>The main difference between the Tutorials and the How-Tos is that the Tutorials only cover the happy path in a contrived scenario and the How-Tos cover both the Happy Path and all of the ways the happy path can become unhappy in a real-world scenario. The main goal of the How To documentation is to get a user that's out of the happy path back on the happy path and resolve any open issues. As you can see, how-to documentation is much more challenging to write because it is set in a real-world context in the presence of all the real-world error conditions. </div><div><br /></div><div>If we were to use the analogy of a compiler, the tutorials show how to compile and run the program, the man pages show how to customize the compile to suit the user and the how-to documentation shows how to recover from error conditions triggered by the program. </div><div><br /></div><div>Only mature programs hit the how-to stage of documentation writing. In order to write the how-to documentation, the program must be robust enough to have the ability to handle all the major error conditions (ideally with a really good error message with detailed instructions on how to address the error condition) and these error conditions must be user-errors and not bugs. This takes time. Once the error conditions have been handled in the application, the how to documentation is written based on how the users are encountering these error conditions (and we are certain that they are not bugs), the program is nearly done and dusted.</div><div><br /></div><div>It is also for this reason that how to documentation is not written for software that is still under active development. Any error conditions encountered by the user are more likely to be bugs rather than user error. Once how-to documentation has been written for a product, you can be reasonably certain that the software has reached a certain level of maturity. It's also a shortcut for identifying good software -- if the howto documentation is detailed and well organized, the software itself is likely to be mature. </div><div><br /></div><div>From a career perspective, you can also use how-to documentation to identify if a software team has reached a level of maturity. Joining a team with a mature piece of software allows you to focus on best practices -- it's good for rapidly learning the best way to do things. It also means that most changes are likely to be incremental. Conversely, a team having little to no documentation, with components running undocumented, with no clear usage guides and regular fires is a team where there's a lot of work to be done to build and stabilize the software. The latter teams are good for early career engineers where doing gets you promoted, the former teams are good for late career folks where getting things done quickly and reliably is paramount. It's also why the more gray haired engineers prefer old-school, rock solid, reliable software - it gets things done with a minimum of fuss. Remember - <a href="https://www.simplethread.com/relational-databases-arent-dinosaurs-theyre-sharks/">old software is not a dinosaur, it's a shark</a>. </div><div><br /></div><div>If you liked this take on software documentation, please take a couple of minutes to write a comment. I would love to know what you took away from this piece. </div><p></p>Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-8598362041542999082023-01-05T01:12:00.003-08:002023-01-05T01:12:45.812-08:00The day I met Whit Diffie (of the Diffie-Hellman crypto fame)<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpfJv2bVvZxCK0aHWMW0InvkmYb5SKxe0C3DVdoqPsXiv9pNG5BnVCwyn-Wzsp6I7Plc22mtF6cP4o8m7Hn_pHmGzkbmnsoOzSrWoB8VddL_eaYAME1hYhK6DSVXATzZVNvUO2LMljb5jT-pRsceUfhc4DrqFKVqpuYtK5FTvEO8wsNC22Jak/s917/Screen%20Shot%202023-01-05%20at%201.04.13%20AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="367" data-original-width="917" height="256" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpfJv2bVvZxCK0aHWMW0InvkmYb5SKxe0C3DVdoqPsXiv9pNG5BnVCwyn-Wzsp6I7Plc22mtF6cP4o8m7Hn_pHmGzkbmnsoOzSrWoB8VddL_eaYAME1hYhK6DSVXATzZVNvUO2LMljb5jT-pRsceUfhc4DrqFKVqpuYtK5FTvEO8wsNC22Jak/w640-h256/Screen%20Shot%202023-01-05%20at%201.04.13%20AM.png" width="640" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtg5Wdt-u4Ugy6J_-0PGMbTyaN5MKGy1XxO-YRFz3HNJOdRoHIw8SxanPaFRx5uQwGZw2ULI2hGUF31Zt0p9_sDzaicVR-eqlr-1y7T_4QsUC7dYD4TyDMnTXvTih8KsPUEj79994n9phW6dvsvWck4KuypUfrgaKuVyb3bCGRFhM_JOZWjvM/s677/d55847c7-1e47-43cf-bacb-83a6054be80c%20(cropped).JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="426" data-original-width="677" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtg5Wdt-u4Ugy6J_-0PGMbTyaN5MKGy1XxO-YRFz3HNJOdRoHIw8SxanPaFRx5uQwGZw2ULI2hGUF31Zt0p9_sDzaicVR-eqlr-1y7T_4QsUC7dYD4TyDMnTXvTih8KsPUEj79994n9phW6dvsvWck4KuypUfrgaKuVyb3bCGRFhM_JOZWjvM/w640-h402/d55847c7-1e47-43cf-bacb-83a6054be80c%20(cropped).JPG" width="640" /></a></div><br /><p>It was either December 11th or December 18th. It was a neighborhood party at Didi's place. Honored to meet a Turing Award winner. I was too tongue tied to say anything beyond platitudes though. Tons of respect for what he's achieved. </p>Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-76234293086697772722022-02-12T00:47:00.000-08:002022-02-12T00:47:18.285-08:00public static is harmful. It has no home in modern programming.<p>In the modern era of software development, "public static" is a relic of a past age. It reflects a bygone era where the concepts of composition and dependency injection were not yet well understood. In the modern world of software, where we understand both of these concepts relatively well - it's important to know that public static has no future. Let's see why:</p><h2 style="text-align: left;"><b><span style="font-family: helvetica; font-size: large;">1. public static forces static coupling, breaks composition and dependency injection</span></b></h2><p>This one is fairly obvious - it's in the name. A call to a public static function typically looks like:</p><p><span style="font-family: courier;">void myCallingFunction() {<br /></span><span style="font-family: courier;"> MyUtils.publicStaticFunction(args);<br /></span><span style="font-family: courier;">}</span></p><p>As we can see clearly, the call to the public static function is happening with a direct reference through the containing class. The calling function is tied down to using only the implementation in MyUtils. The caller may not choose another implementation even if the caller may be calling the code in a test environment (<b>brittle composition</b>). This<b> "hidden composition"</b> <b>breaks dependency injection and testability</b>.</p><h2 style="text-align: left;"><b><span style="font-family: helvetica; font-size: large;">2. public static produces "virulent" recursive refactors</span></b></h2><p>Assuming that there are stateless "utility" classes containing public static methods: if these public static methods become typical OOP methods (non-static) [and there are good reasons to make these changes], it will force a <b>recursive-refactor</b> of ALL call sites of the public static methods within the codebase, causing <b>painful cascading changes</b>. This one is really nasty and a bit tricky to see - here's an example:</p><h4 style="text-align: left;"><b>Old code:</b></h4><p><span style="font-size: x-small;"><span style="font-family: courier;">class ExceptionReporter {<br /> </span><span style="font-family: courier;">// Public Static method; Hardcoded urls & libs.</span><span style="font-family: courier;"><br /> public static void logException() {<br /> HttpClient.call("http://exceptions.example.com/report"); <br /> }<br />}<br /><br />class MyUtils {<br /> public static void myWidelyUsedUtilsFunction() {<br /> ExceptionReporter.logException(); // Brittle dependency.<br /> }<br />}</span></span></p><p><b>Problems:</b></p>The ExceptionReporter will always make the network call (even if it's running within a test environment). The URL is non controllable, the Http client library is non controllable, the protocol is non controllable.<br /><br />Let's say we decide to make this code test friendly and we parameterize ExceptionReporter:<p></p><h4 style="text-align: left;"><b>Refactor to:</b></h4><p><span style="font-family: courier; font-size: x-small;">class ExceptionReporter {<br /> // Parameterized constructor.<br /> public ExceptionReporter(URI uri, HttpClient httpClient) { ... }<br /> public void logException() { ... } // Use member vars.<br />}</span><br /><br />The call sites are all badly impacted. We can either:<br />1. Pass through the <span style="font-family: courier;">ExceptionReporter</span> as a function argument<br /> (<b>Con</b>: the URLs and HTTP Clients are strewn in 100s of call sites -- a large number of identical ExceptionReporter objects get created for little value, the copy-paste is tech-debt). OR<br />2. Convert <span style="font-family: courier;">MyUtils</span> to contain a member variable.<br /> (<b>Con</b>: MyUtils cascades through function arguments and constructor changes to clients. If classes are constructing MyUtils objects, they have to pass through constructor dependencies, <b>forcing recursive refactoring. </b>Additionally, the stateless method call becomes a stateful class + object dispatch, this may also affect thread safety guarantees). To put it mildly, this is a difficult change.</p><p>To see this clearly, see these options in code:<br /><br /></p><h3 style="text-align: left;"><b><u>Option 1: Pass through ExceptionReporter as function arguments.</u></b></h3><p></p><p><span style="font-size: x-small;"><span style="font-family: courier;">// All call sites need a new function parameter passed in.<br />class MyUtils {<br /> public static void myWidelyUsedUtilsFunction(ExceptionReporter e) { ... }<br /> public static void myWidelyUsedOtherFunction(ExceptionReporter e) { ... }<br />}<br /><br />// The constructor arguments keep getting copy-pasted<br />public static void myCallSite1() {<br /> ExceptionReporter e = new ExceptionReporter(<br /> "http://exceptions.example.com/report", <br /> new HttpClient()); // Copy-paste everywhere.<br /><br /> MyUtils.myWidelyUsedUtilsFunction(e);<br /></span><span style="font-family: courier;"> MyUtils.myWidelyUsedOtherFunction(e);</span><span style="font-family: courier;"><br /></span></span><span style="font-family: courier;"><span style="font-size: x-small;">}</span> </span></p><div style="text-align: left;"><b><u><span style="font-size: medium;">Option 2: Convert all the public static call sites to proper OOP function calls.</span><br /></u></b><br /><span style="font-size: x-small;"><span style="font-family: courier;">class MyUtils {<br /> // Convert to "proper" class + object.<br /></span><span style="font-family: courier;"> public MyUtils(ExceptionReporter e) { ... }; <br /> ... </span></span></div><div style="text-align: left;"><span style="font-size: x-small;"><span style="font-family: courier;">}<br /><br />// All call sites create 2 objects! <br /></span><span style="font-family: courier;">public static void myCallSite1() {<br /> // The number of objects created may be large here.<br /> // Depends on constructor parameters.<br /> ExceptionReporter e = new ExceptionReporter(<br /> "http://exceptions.example.com/report", new HttpClient());<br /> MyUtils myUtils = new MyUtils(e);<br /><br /> myUtils.myWidelyUsedUtilsFunction();<br /></span><span style="font-family: courier;"> myUtils.myWidelyUsedOuterFunction();</span><span style="font-family: courier;"><br /></span></span><span style="font-family: courier;"><span style="font-size: x-small;">}</span><br /><br /></span>This cascading process of adding constructors, member variables, object creation and function arguments is a long, painful "virulent" recursive refactor. The compile errors while doing this may seem like a virus spreading outwards from the site of the original public static methods to all their call sites and all their further call sites and all enclosing objects and methods. If this is a widely used library, several days of intensive, laborious work will need to be done. I've been there and done it - it's not enjoyable.</div><div style="text-align: left;"><br /><h2 style="text-align: left;"><b><span style="font-size: large;">3. Every public static method can trivially be replaced with a non-static implementation at the cost of a single stack pointer bump.</span></b></h2><p>The only difference is the call site:</p><p>Before: <br /><span style="font-family: courier;"><span style="font-size: x-small;">public void callSite() {<br /> MyStaticClass.myUtilsFunction(); // no object.<br />}</span><br /></span><br />After:<br /><span style="font-size: x-small;"><span style="font-family: courier;">// testable, dependency injected code.</span><span style="font-family: courier;"><br />public void callSite(MyNonStaticClass dependency) {<br /> dependency.myNonStaticUtilsFunction(); // requires object alloc.<br />}</span></span><br /><br />The JVM has a nifty feature called <a href="https://blogs.oracle.com/javamagazine/post/escape-analysis-in-the-hotspot-jit-compiler" target="_blank">Escape Analysis</a>, which will determine whether an object is retained on the stack or placed on the heap. If an object never escapes a function or a set of functions (through a data-flow trace), the JVM will avoid allocating it on heap (instead, it's space will be allocated through an %(esp) stack pointer bump). As soon as the function returns, the %(esp) register reset automatically deallocates the object (no GC needed!). In practice, this means that the empty object allocation costs (approximately) 1 add instruction (0.3 nanoseconds) only! In my view, this is a <b>very cheap price</b> to pay.</p><h2 style="text-align: left;"><span style="font-size: large;">To Conclude</span></h2><p>For the price of a single add instruction and (often) zero impact to GC, we can:<br />a) Improve testability and dependency injection, make the code cleaner.<br />b) Encourage well encapsulated, well parameterized codebases and good coding practices. <br />c) Completely remove the possibility of "virulent" refactors.<br /><br />The result is clear - "public static" should be eliminated. Please avoid it your code and discourage its use in code reviews. If there's a debate, please refer them to this post. Happy hunting! </p></div>Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-56795628724207505842021-11-10T11:34:00.000-08:002021-11-10T11:34:24.754-08:00Use the Heilmeirs for Project Funding<p>I learnt about the Heilmeirs from <a href="https://www.linkedin.com/in/vgill/" target="_blank">Vijay Gill</a>, an ex-SVP at Databricks (now an SVP at RapidAPI). He said that this was the fastest way to focus the discussion around project funding. I've found it to be useful - sharing this with you.</p><h1 id="ProductivityTips-TheHeilmeirs" style="background-color: white; border-bottom-color: rgb(196, 196, 196); color: #172b4d; font-family: -apple-system, "system-ui", "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 24px; font-weight: normal; letter-spacing: -0.01em; line-height: 1.25; margin: 30px 0px 0px; padding: 0px;" talk-marker="43" talk-page-id="58735044" talk-page-version="32">The Heilmeirs</h1><p style="background-color: white; color: #172b4d; font-family: -apple-system, "system-ui", "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="44" talk-page-id="58735044" talk-page-version="32">Project funding is governed by this one-pager process:</p><ul style="background-color: white; color: #172b4d; font-family: -apple-system, "system-ui", "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px;"><li><span style="display: block;" talk-marker="45" talk-page-id="58735044" talk-page-version="32">What are you trying to do? Articulate your objectives using absolutely no jargon.</span></li><li><span style="display: block;" talk-marker="46" talk-page-id="58735044" talk-page-version="32">How is it done today, and what are the limits of current practice?</span></li><li><span style="display: block;" talk-marker="47" talk-page-id="58735044" talk-page-version="32">What is new in your approach and why do you think it will be successful?</span></li><li><span style="display: block;" talk-marker="48" talk-page-id="58735044" talk-page-version="32">Who cares? If you are successful, what difference will it make?</span></li><li><span style="display: block;" talk-marker="49" talk-page-id="58735044" talk-page-version="32">What are the risks?</span></li><li><span style="display: block;" talk-marker="50" talk-page-id="58735044" talk-page-version="32">How much will it cost?</span></li><li><span style="display: block;" talk-marker="51" talk-page-id="58735044" talk-page-version="32">How long will it take?</span></li><li><span style="display: block;" talk-marker="52" talk-page-id="58735044" talk-page-version="32">What are the mid-term and final “exams” to check for success?</span></li></ul>Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com1tag:blogger.com,1999:blog-34173777.post-28598073602197097942021-11-10T11:01:00.003-08:002021-11-10T11:01:29.967-08:00The Effective Executive<p> <span style="background-color: white; color: #172b4d; font-family: -apple-system, "system-ui", "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 24px; letter-spacing: -0.01em;">The Effective executive</span></p><p style="background-color: white; color: #172b4d; font-family: -apple-system, "system-ui", "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="33" talk-page-id="58735044" talk-page-version="32">“An effective executive does not need to be a leader in the sense that the term is now most commonly used. Harry Truman did not have one ounce of charisma, for example, yet he was among the most effective chief executives in U.S. history. Similarly, some of the best business and nonprofit CEOs I’ve worked with over a 65-year consulting career were not stereotypical leaders. They were all over the map in terms of their personalities, attitudes, values, strengths, and weaknesses. They ranged from extroverted to nearly reclusive, from easygoing to controlling, from generous to parsimonious.</p><p style="background-color: white; color: #172b4d; font-family: -apple-system, "system-ui", "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="34" talk-page-id="58735044" talk-page-version="32">What made them all effective is that they followed the same eight practices:</p><ul style="background-color: white; color: #172b4d; font-family: -apple-system, "system-ui", "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px;"><li><span style="display: block;" talk-marker="35" talk-page-id="58735044" talk-page-version="32">They asked, ‘What needs to be done?’</span></li><li><span style="display: block;" talk-marker="36" talk-page-id="58735044" talk-page-version="32">They asked, ‘What is right for the enterprise?’</span></li><li><span style="display: block;" talk-marker="37" talk-page-id="58735044" talk-page-version="32">They developed action plans.</span></li><li><span style="display: block;" talk-marker="38" talk-page-id="58735044" talk-page-version="32">They took responsibility for decisions.</span></li><li><span style="display: block;" talk-marker="39" talk-page-id="58735044" talk-page-version="32">They took responsibility for communicating.</span></li><li><span style="display: block;" talk-marker="40" talk-page-id="58735044" talk-page-version="32">They were focused on opportunities rather than problems.</span></li><li><span style="display: block;" talk-marker="41" talk-page-id="58735044" talk-page-version="32">They ran productive meetings.</span></li><li><span style="display: block;" talk-marker="42" talk-page-id="58735044" talk-page-version="32">They thought and said ‘we’ rather than ‘I.’</span></li></ul><h1 id="ProductivityTips-TheHeilmeirs" style="background-color: white; border-bottom-color: rgb(196, 196, 196); color: #172b4d; font-family: -apple-system, "system-ui", "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 24px; font-weight: normal; letter-spacing: -0.01em; line-height: 1.25; margin: 30px 0px 0px; padding: 0px;" talk-marker="43" talk-page-id="58735044" talk-page-version="32"><br /></h1>Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-89589786542017067172021-06-03T12:23:00.003-07:002021-07-02T15:52:39.228-07:00For Effective Meetings, leverage the Meeting Bill of Rights<p>Meetings become enormously important as a medium of work execution as people become more senior over time. I've seen that the quality of meetings vary from person to person and organization to organization. Over the years, I've found the following principles very useful to frame my approach to meetings. I'm sharing this with you - hope you find it useful as well. Cheers!</p><p> <a class="external-link" href="https://www.leadstrat.com/blog/we-have-to-stop-meeting-like-this-your-meeting-bill-of-rights-2/" rel="nofollow" style="color: #0084ff; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; text-decoration-line: none;">Adapted</a><span face="-apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif" style="background-color: white; color: #172b4d; font-size: 14px;"> from "The Secrets to Masterful Meetings" (</span><a class="external-link" href="https://www.leadstrat.com/wp-content/uploads/2016/02/MeetingBillofRights-LeadershipStrategies-20160202.pdf" rel="nofollow" style="color: #0084ff; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; text-decoration-line: none;">PDF</a><span face="-apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif" style="background-color: white; color: #172b4d; font-size: 14px;">); How <a href="https://www.inc.com/justin-bariso/jeff-bezos-knows-how-to-run-a-meeting-here-are-his-three-simple-rules.html" target="_blank">Jeff Bezos runs meetings</a>.</span></p><p align="center" style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="1" talk-page-id="58738787" talk-page-version="4">Your Meeting Bill of Rights</p><ol style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px 20px;"><li><span style="display: block;" talk-marker="2" talk-page-id="58738787" talk-page-version="4">Meeting Notice.You have the right to be informed about the purpose and proposed agenda for a meeting, verbally or in writing, at least twenty-four hours in advance of the meeting.</span></li><li><span style="display: block;" talk-marker="3" talk-page-id="58738787" talk-page-version="4">Timely Start.You have the right to attend meetings that start on time. 3 minutes late: issue a reminder; 7 minutes late: meeting cancelled. </span></li><li><span style="display: block;" talk-marker="4" talk-page-id="58738787" talk-page-version="4"><strong>Right People</strong>.You have the right to have all major viewpoints critical to decision-making represented at the meeting. Conversely, If you're not useful to the meeting, feel free to leave.</span></li><li><span style="display: block;" talk-marker="5" talk-page-id="58738787" talk-page-version="4"><strong>Right Information</strong>.You have the right to have the information necessary to facilitate decision-making available at the meeting. <u>Come prepared.</u></span></li><li><span style="display: block;" talk-marker="6" talk-page-id="58738787" talk-page-version="4">Ground Rules.You have the right to have agreed upon ground rules respected in the meeting.</span></li><li><span style="display: block;" talk-marker="7" talk-page-id="58738787" talk-page-version="4"><strong>Focused Discussion</strong>.You have the right for meetings to stay focused on the topic of the meeting.</span></li><li><span style="display: block;" talk-marker="8" talk-page-id="58738787" talk-page-version="4">Input Opportunity.You have the right to have the opportunity to provide input and alternative views before decision-making occurs in the meeting.</span></li><li><span style="display: block;" talk-marker="9" talk-page-id="58738787" talk-page-version="4"><strong>Meeting Recap</strong>.You have the right to hear a recap of (a) decisions made during the meeting, (b) actions to be taken, when and by whom, following the meeting, and (c) any outstanding issues to be discussed at a future meeting.</span></li><li><span style="display: block;" talk-marker="10" talk-page-id="58738787" talk-page-version="4">Timely Completion.You have the right to have your time respected by having meetings finish at or before the scheduled end time.</span></li><li><span style="display: block;" talk-marker="11" talk-page-id="58738787" talk-page-version="4">No Retribution.You have the right to exercise <em>Your</em><em>Meeting Rights</em> without fear of retribution or other consequences.</span></li></ol><p style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="12" talk-page-id="58738787" talk-page-version="4">If you have the right people, the right information, a focused discussion and a recap of the decisions and action items of the meeting, the meeting is successful.</p><h2 id="MeetingBillofRights-What'sexpectedintheAgenda?" style="background-color: white; border-bottom-color: rgb(196, 196, 196); color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 20px; font-weight: normal; letter-spacing: -0.008em; line-height: 1.5; margin: 30px 0px 0px; padding: 0px;" talk-marker="13" talk-page-id="58738787" talk-page-version="4">What's expected in the Agenda?</h2><ol style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px;"><li><span style="display: block;" talk-marker="14" talk-page-id="58738787" talk-page-version="4">Specify the <a class="external-link" href="http://meetingsift.com/the-six-types-of-meetings/" rel="nofollow" style="color: #0084ff; text-decoration-line: none;">type of meeting</a>: <ol style="list-style-type: lower-alpha; margin: 0px;"><li><span style="display: block;" talk-marker="15" talk-page-id="58738787" talk-page-version="4">Problem Solving</span></li><li><span style="display: block;" talk-marker="16" talk-page-id="58738787" talk-page-version="4">Decision Making</span></li><li><span style="display: block;" talk-marker="17" talk-page-id="58738787" talk-page-version="4">Knowledge Sharing</span></li><li><span style="display: block;" talk-marker="18" talk-page-id="58738787" talk-page-version="4">Status Update</span></li><li><span style="display: block;" talk-marker="19" talk-page-id="58738787" talk-page-version="4">Innovation</span></li><li><span style="display: block;" talk-marker="20" talk-page-id="58738787" talk-page-version="4">Team Building</span></li></ol></span></li><li><span style="display: block;" talk-marker="21" talk-page-id="58738787" talk-page-version="4">For Problem Solving and Decision Making meetings: clearly specify the issue or the decision as part of the agenda. <ol style="list-style-type: lower-alpha; margin: 0px;"><li><span style="display: block;" talk-marker="22" talk-page-id="58738787" talk-page-version="4">At close, <u><em>recap</em></u> and ensure that a decision was reached.</span></li></ol></span></li><li><span style="display: block;" talk-marker="23" talk-page-id="58738787" talk-page-version="4">For Information Sharing / Status Update meetings: come prepared with the shareable information (ideally written down)<ol style="list-style-type: lower-alpha; margin: 0px;"><li><span style="display: block;" talk-marker="24" talk-page-id="58738787" talk-page-version="4">At close, <u>recap</u> information and determine action items.</span></li></ol></span></li><li><span style="display: block;" talk-marker="25" talk-page-id="58738787" talk-page-version="4">For Innovation meetings: Suspend judgement.<br talk-br="0" /><ol style="list-style-type: lower-alpha; margin: 0px;"><li><span style="display: block;" talk-marker="26" talk-page-id="58738787" talk-page-version="4">Brainstorming happens in 2 phases: Flare and Focus. They should take place separately.</span></li><li><span style="display: block;" talk-marker="27" talk-page-id="58738787" talk-page-version="4">Flare: Use "Yes, and..." language instead of "Yes, but...". "Yes, and..." is <a class="external-link" href="https://www.huffingtonpost.com/karen-hough/yes-but-the-evil-twin-to-_b_5669640.html" rel="nofollow" style="color: #0084ff; text-decoration-line: none;">collaborative and stimulates discussion</a>. </span></li><li><span style="display: block;" talk-marker="28" talk-page-id="58738787" talk-page-version="4">If you have concerns, hold them aside for the focus meeting.</span></li></ol></span></li><li><span style="display: block;" talk-marker="29" talk-page-id="58738787" talk-page-version="4">Team Building: Have fun! Get to know the team and skip talking about work. <ol style="list-style-type: lower-alpha; margin: 0px;"><li><span style="display: block;" talk-marker="30" talk-page-id="58738787" talk-page-version="4">The goal is to know and trust your teammates. </span></li></ol></span></li></ol><h1 id="MeetingBillofRights-Here'safewexamples" style="background-color: white; border-bottom-color: rgb(196, 196, 196); color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 24px; font-weight: normal; letter-spacing: -0.01em; line-height: 1.25; margin: 30px 0px 0px; padding: 0px;" talk-marker="31" talk-page-id="58738787" talk-page-version="4">Here's a few examples</h1><p style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="32" talk-page-id="58738787" talk-page-version="4">Meeting: Qingxian : Divye - Realtime Feature Monitoring Status<br talk-br="1" />Date: Jan 4, 2019</p><p style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="33" talk-page-id="58738787" talk-page-version="4"><span class="ynRLnc" style="color: #3c4043;">Description:<br talk-br="2" /></span><span style="color: #3c4043;">Agenda: Knowledge Sharing<br talk-br="3" />=====================<br talk-br="4" />1. Qingxian: Status of RT Feature Monitoring; Demo of current state.<br talk-br="5" />2. Any action items for Divye?</span></p><p style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="34" talk-page-id="58738787" talk-page-version="4"><br talk-br="6" /></p><p style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="35" talk-page-id="58738787" talk-page-version="4">Meeting: David : Divye – follow up on HF Training Doc<br talk-br="7" />Date: Jan 3, 2019</p><p style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="36" talk-page-id="58738787" talk-page-version="4">Description:<br talk-br="8" /><span style="color: #3c4043;">Agenda<br talk-br="9" />=======<br talk-br="10" />1. Sync on Homefeed training</span><span style="color: #3c4043;"><br talk-br="11" />Outcome: Knowledge sharing.<br talk-br="12" />Preparation: Divye to look at code pointers in the doc before the meeting.</span></p><p style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="37" talk-page-id="58738787" talk-page-version="4"><br talk-br="13" /></p><p style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="38" talk-page-id="58738787" talk-page-version="4">Meeting: Divye : P - Sell Call - Software Engineer, ML Platform<br talk-br="14" />Date: Jan 3, 2019</p><p style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="39" talk-page-id="58738787" talk-page-version="4">Description:<br talk-br="15" /><span style="color: #3c4043;">=========<br talk-br="16" />- Expecting an offer from X<br talk-br="17" />- Interviewing with A, B, C, D, E, F <br talk-br="18" />- Top choices are Pinterest, Airbnb, and Lyft<br talk-br="19" />- Priorities: good culture, likes the product, team/project, tech lead, eventual mgmt<br talk-br="20" />- Concerns: does team/project have enough scope, leadership opportunities, how do we support IC to managers?</span></p><p style="background-color: white; color: #172b4d; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="39" talk-page-id="58738787" talk-page-version="4"><span style="color: #3c4043;"><br /></span></p><p style="background-color: white; font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; margin: 10px 0px 0px; padding: 0px;" talk-marker="39" talk-page-id="58738787" talk-page-version="4"><span style="color: #3c4043;">It's not necessary to be explicit about the type of meeting in the agenda (as long as it's clear what needs to get done by the end of the meeting). A few maxims - don't leave a decision meeting without the decision made, if not made, have clearly identified next steps. For a problem solving meeting, leave only after next steps are identified. Recap is the most important part of a Knowledge sharing meeting and people in a Team Building event should always leave happy after a round of chit-chat. </span></p>Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-82131925667628750462020-12-28T02:30:00.001-08:002020-12-28T02:47:26.161-08:00Can white prevent checkmate?<div dir="auto">I reached here as black. White to play - Qh5 was my last move. I had just sacked my bishop for 2 pawns to open up the king. Can white save checkmate?</div><div dir="auto"><br /></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMjdSxMqW6hsqVNziRpIhzd-6tf59r_3WBeeABVkbacqmPo4azbvm4U4UK6LNTMesvr8lJefAgxTjv2-suAjWckDVzcmwa-9gvZCAR-_mowbzH8a65F6U74xVlBbYubWMW-pMkZQ/s1600/2718-722704.jpg"><img alt="" border="0" id="BLOGGER_PHOTO_ID_6911252739811996210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMjdSxMqW6hsqVNziRpIhzd-6tf59r_3WBeeABVkbacqmPo4azbvm4U4UK6LNTMesvr8lJefAgxTjv2-suAjWckDVzcmwa-9gvZCAR-_mowbzH8a65F6U74xVlBbYubWMW-pMkZQ/s16000/2718-722704.jpg" /></a></div><div>Full game:</div><div>Nimzowitsch-Larsen Attack</div><div>Fairly interesting in its own way:</div><div>1. b3 e5</div><div>2. Bb2 Nc6</div><div>3. Nf3 d6</div><div>4. e4 Nf6</div><div>5. Bb5 a6</div><div>6. Bxc6+ bxc6</div><div>7. Nc3 d5 </div><div>8. exd5 cxd5</div><div>9. O-O Bg4 </div><div>10. h3 Bh5</div><div>11. Qe2 e4</div><div>12. g4 Bxg4</div><div>13. hxg4 Nxg4</div><div>14. Nd4 Qh5</div><div><br /></div><div>0-1</div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com2tag:blogger.com,1999:blog-34173777.post-31068611468972061772020-10-16T13:35:00.000-07:002020-11-18T18:40:28.087-08:00Expectations of an Engineering Manager<div dir="ltr"><a href="http://blog.rlmflores.me/2020/10/14/what_is_expected_of_an_engineering_manager/">This is a beautiful article</a> that very succinctly and accurately defines management responsibilities:<div><a href="http://blog.rlmflores.me/2020/10/14/what_is_expected_of_an_engineering_manager/"></a></div><div><br /></div><div>The key principles being:</div><div><ul><li>Support the members of your team and help them grow.</li><li>Follow along the deliveries, setting quality standards, making sure the team has the support they need and upper management the feedback they need [through you and without you]</li><li>Keep a constant practice of creating, improving or eliminating team or company processes.</li></ul></div><div>There are some issues with the article as well: for example, the reliance on Andy Grove's definition of management is something that I don't fully agree with. It's a reasonable definition, don't get me wrong. It's even valuable in certain ways, I just don't fully agree with it though.</div><div><br /></div><div>Traditionally, Andy Grove defined a manager's output <i>as the sum of their team's output and the teams under influence output</i>. This is great if the goal of your organization's managers is to play politics at promotion time (hey I influenced this guy, this is my project! this win is from my team! etc.). It's not so great if the goal is to objectively define a manager's performance with respect to items under their control and that work they do on a daily basis.</div><div><br /></div><div>For example, a team or organization can be highly effective and yet have a terrible manager or leader who pits the team members against each other. A prima donna manager can "influence" teams by being a pain. That's typically not what's desired. A manager (for better or for worse) is an institutional position. It's institutional because where there's a team, there needs to be a goal, someone who sets goalposts and a referee that calls the goal when it's achieved. It's inevitable. That's the process nature of management => things need to happen, there needs to be one or more people who can take responsibility for the goal and process. That's the manager. However, unless the manager actually directly contributed (positively) to the goal, the process, the execution; taking credit for the team's work feels a bit ingenuous. Similarly, a great manager joining a team that needs a turnaround can't be adequately measured by the total team output. The change in the team out is a valid benchmark.</div><div><br /></div><div>My preferred benchmark of a manager's work is in a term called <b>effectiveness</b>. Effectiveness is defined in terms of the actions of the manager with respect to their team. </div><div><br /></div><div>How effective is the manager in growing the team or growing the org? </div><div>Is hiring well thought out? </div><div>Are the roles clear before a hire? </div><div>Are new hires well integrated? </div><div>Are existing team members aware of their growth charts? </div><div>Does the team interact well with each other? </div><div>Are process blockers simplified or removed?</div><div>Are decisions made quickly?</div><div>What is the quality of those decisions?</div><div>Are non-consensus decisions documented?</div><div>Is communication within the team smooth?</div><div>Are processes within the team smooth?</div><div>Does the work of the team get adequately documented and promoted in the org?</div><div>Is the team correctly benchmarked for effectiveness against its peers?</div><div>How accommodating is the manager of diverse perspectives?</div><div>How much autonomy does the team have? Is there micromanagement?</div><div>Is the manager inclusive of opinions that diverge from their own?</div><div>Are the right members of the team rewarded?</div><div>For team members that are struggling, is corrective action taken? </div><div>How is that corrective action communicated and managed?</div><div>Are performance conversations continuous and regular?</div><div>Does the manager actively and regularly seek feedback about their own performance?</div><div>Does the manager provide opportunities to lead?</div><div>Do you feel you have enough scope that stretches you in your current role?</div><div><div>Does your manager provide technical guidance and support? [(in-my-view) a delicate question].</div><div></div></div><div><br /></div><div>The team's eventual output is a derivative of a well run team and facilitative management trounces management by control and process any day. If we agree with the previous statement, a manager should then. be <b>assessed by their team and their own peers and managers</b> on the questions listed above - how well does the manager facilitate the work of the company? For the questions listed above are specific, they are positive and a negative reaction on any of them indicates things that the manager can do to be more effective. There are probably more questions that can be added to the list and I'd love to hear some from you. If you come across this, please leave a comment!</div><div><br /></div><div>Other articles that I found interesting in this domain:</div><div><ul style="text-align: left;"><ul><li><a href="http://www.defmacro.org/2014/10/03/engman.htm" target="_blank">44 engineering management lessons</a></li><li>Know your own strengths with a <a href="https://www.gallup.com/cliftonstrengths/en/253715/34-cliftonstrengths-themes.aspx" target="_blank">Clifton Strengths report</a> [helps greatly with understanding how you'll do as a manager]</li><li><a href="https://faculty.washington.edu/ajko/papers/Li2019WhatDistinguishesEngineers.pdf" target="_blank">What distinguishes great software engineers [pdf]?</a> (<a href="https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2015/whatmakesagreatsoftwareengineer_slides.pdf" target="_blank">slides</a>) talks about how managers view senior, capable engineers and the set of personality traits that help them be effective at their jobs.</li></ul></ul></div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-64738125459787961422020-09-28T20:09:00.001-07:002020-10-04T22:24:17.086-07:00Pearls of wisdom: taxing into prosperity<div>
For a nation to try and tax itself into prosperity is like a man standing in a bucket trying to lift himself up by the handle -- Winston Churchill <br/>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi34SCdu28X9JKFNUKpeMSW2vf5lO943AkqQTiFX_ZyGrzGERx5ibeSw_mkgYPxgT_JP2uSrmCqRCZ1YFCH9QNlM5W6VGlz7PBzCYkevjFfrbq-CVM-nN72XqQhjHmjazsoSVf9wQ/s1600/61559-799040.jpg"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi34SCdu28X9JKFNUKpeMSW2vf5lO943AkqQTiFX_ZyGrzGERx5ibeSw_mkgYPxgT_JP2uSrmCqRCZ1YFCH9QNlM5W6VGlz7PBzCYkevjFfrbq-CVM-nN72XqQhjHmjazsoSVf9wQ/s320/61559-799040.jpg" border="0" alt="" id="BLOGGER_PHOTO_ID_6877741583489185282" /></a></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-55541416044018134372020-06-26T15:09:00.001-07:002020-06-26T16:49:25.564-07:00Checked exceptions break composition<div dir="ltr"><b>A.K.A. Always Throw Runtime Exceptions or their subclasses</b><br><div><br></div><div>A typical Java or C++ function can potentially come with an exception specification: for example, a method can declare that it throws exceptions of a particular type (eg. IOException, std::bad_alloc etc.) and clients need to handle that exception being thrown with a try-catch block. This seems good at the outset till we spend time thinking through what this does to the <b>type</b> of the function.</div><div><br></div><div>A typical function in a happy-go-lucky world either succeeds or fails with an exception because of something beyond its control. If it succeeds, it returns with a value of the provided return type (let's call it SuccessValueType). If it fails with an exception (eg. a file read error or a mem allocation error), it throws the exception and the error handling parts of the code run. In type terms, the <b>return type</b> of the function is Either<SuccessValueType, RuntimeExceptionType> (where the RuntimeExceptionType is an implicit return type of the function). If all functions agree that RuntimeExceptionType is the implicit secondary return type, functions and try-catch blocks compose beautifully. This is because every function call site becomes an implicit early return point with a valid return value from the function. As a corollary, every try-catch block wrapping the function also makes little to no assumptions of the kinds of exceptions its likely to receive and that builds in flexibility for code evolution.</div><div><br></div><div><b>Here's an example:</b></div><div>Function 1 => calls => Function 2 followed by Function 3; both Function 2 and 3 can only throw RuntimeExceptions</div><div>If either of these functions throws, the RuntimeException is propagated as an "early return" from Function 1 without any changes. You can stack as many layers of nesting in the code and the return types and early return behavior remain compatible (because all the functions agree that RuntimeExceptionType is an implicit return type).</div><div>The application then adds error handling code close to the top-level of the processing hierarchy and presents the error to the user (as a form of recovery) or retries or notifies an engineer to take a look. If we need to add additional context to the exception, at any level a try-catch block can be introduced to attach context information to the exception and rethrowing the RuntimeException. This introduction of an intermediate try-catch is a<b> purely local change</b> that composes well with try-catch blocks further up the stack (removing a try-catch similarly composes well). Adding new libraries or call paths to the code remains a purely local operation and does not affect the type hierarchy or the error-handling try-catch structure.</div><div><br></div><div>Contrast with what happens <b>when a checked exception is introduced</b>. The function type changes from bi-valent to tri-valent: Either<SuccessValueType, RuntimeExceptionType, CheckedExceptionType>. Note that avoiding the RuntimeExceptionType is not possible (else you'll have code littered with redundant bad_alloc, io_exceptions and the like that are meaningless). With a trivalent return type from a function, we have 2 options: </div><div><br></div><div>1. Convert the function back into the bi-valent return type by introducing a try-catch block, catching the checked exception and rethrowing as a RuntimeException.</div><div>2. Propagate the checked exception and ask our clients to update their code.</div><div><br></div><div>(1) is of-course the reasonable thing to do. It's a local operation, client code doesn't have to change and we're back to having to deal with only a single type of failure (either the function succeeds or the function fails with a RuntimeException).</div><div>(2) is a world of pain. If we're in this world, every new introduction of a checked exception means that significant chunks of the program have to change to include the new checked exception type.</div><div><br></div><div><b>Going back to our original example:</b></div><div>If Function 1 => calls => Function 2 followed by Function 3 and both of them throw checked exceptions of different types, the return type of Function 1 becomes Either<SuccessValueType, RuntimeExceptionType, F2CheckedExceptionType, F3CheckedExceptionType> (essentially the union of all the checked exceptions show up in the return type signature). As we keep adding more nested functions, this type list keeps expanding. </div><div><br></div><div>In practical terms, this means that the developer adds "throws F2CheckedExceptionType, F3CheckedExceptionType, ..." to each of the caller functions in order to get them to compose. All the try-catch blocks similarly bloat to handle all the possible failure cases. Beyond small-sized codebases, this is completely infeasible because these signature changes and the try-catch handlers keep propagating out throughout the codebase. This hurts dev velocity.</div><div><br></div><div>From a recovery perspective, these checked exceptions are typically handled just one-level up the call stack at the lowest level of library code (to avoid the exception signature blowout) and a local resolution is done (retry a few times and then fail). This is typically not an optimal solution (eg. for an out-of-disk-space error, a batch processing application might prefer an immediate crash, a streaming application might prefer a continuous retry but without propagating the error all the way up to the application, this choice of recovery can't be made reasonably and the only way out is to pass down configurations to control this behavior... a gargantuan mess). </div><div><br></div><div>In the RuntimeException only world, the retry configuration stays at the top level where things can be handled based on the execution environment.</div><div><br></div><div>In summary, as a practical matter, professional software engineers should <b>ensure</b> that their functions <b>only throw unchecked exceptions</b> (RuntimeExceptions or similar). Checked exceptions are actively harmful to dev velocity in large codebases and should be avoided. Google avoids this tar pit by banning exceptions from C++ code (for historical reasons), LinkedIn & Pinterest actively utilize a RuntimeException based Java codebase and you should encourage this too.</div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-78268514417983565202020-01-14T17:02:00.000-08:002020-01-14T17:03:06.353-08:00IO numbers that everyone should know<div dir="ltr"><p style="margin:0px 0px 1em;padding:0px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit;line-height:inherit;font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;font-size:15px;vertical-align:baseline;box-sizing:inherit;clear:both;color:rgb(36,39,41)">In the <a href="https://gist.github.com/jboner/2841832">Numbers Every Programmer Should Know</a>, one set of numbers that I've always found missing were IO numbers (HDD vs SSD - random reads / writes). I found a <a href="https://superuser.com/questions/1108034/why-are-4k-reads-in-hdd-ssd-benchmarks-slower-than-writes">really good source on StackExchange</a> for these numbers and for the sake of posterity, I'm documenting this here (for me and for you):</p><ul style="margin:0px 0px 1em 30px;padding:0px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit;line-height:inherit;font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;font-size:15px;vertical-align:baseline;list-style-position:initial;box-sizing:inherit;color:rgb(36,39,41)"><li style="margin:0px 0px 0.5em;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">SSD | HDD Sequential Read/Write : 700 MB/s+ | 115 MB/s (6x diff) </li><li style="margin:0px 0px 0.5em;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">SSD | HDD Random Read 512KB : 160 MB/s | 39 MB/s (4x diff)</li><li style="margin:0px 0px 0.5em;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">SSD | HDD Random Write 512KB : 830 MB/s | 57 MB/s (14x diff)</li><li style="margin:0px 0px 0.5em;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">SSD | HDD Random Read</span> 4KB<span style="font-weight:inherit">: </span><strong style="font-weight:inherit;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">27 MB/s</strong><span style="font-weight:inherit"> | 0.5 - 1.5 MB/s (17x diff)</span></li><li style="margin:0px 0px 0.5em;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit"><span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">SSD | HDD Random Write</span> 4KB<span style="font-weight:inherit">: </span><strong style="font-weight:inherit;margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">135 - 177 MB/s</strong><span style="font-weight:inherit"> | 0.7 MB/s (192x+ diff!)</span></li></ul><div><font color="#242729" face="Arial, Helvetica Neue, Helvetica, sans-serif"><span style="font-size:15px">The bottom line is that unless you're thrashing the HDD with lots of 4KB random writes, the <u><b>HDD</b> should not be tapped out till about <b>30+ MB/s</b></u> (and an <u><b>SSD</b> should be just fine till about <b>150 - 300 MB/s</b></u>). If you're seeing an HDD tapped out at 3 MB/s, then you're either not writing sequentially or your block size of writes is too small. If you're seeing an SSD tapped out at < 100 MB/s, it's almost certainly a software bug and not an IO limitation. In either case, the basic norm holds - if you can, always use SSDs, they usually save you money in CPU time.</span></font></div><div><font color="#242729" face="Arial, Helvetica Neue, Helvetica, sans-serif"><span style="font-size:15px"><br></span></font></div><div><font color="#242729" face="Arial, Helvetica Neue, Helvetica, sans-serif"><span style="font-size:15px"><br></span></font></div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com2tag:blogger.com,1999:blog-34173777.post-21500455683517817642019-06-19T12:38:00.001-07:002020-08-11T11:01:40.122-07:00Modern Programming: never use Inheritance; use Composition instead<div dir="ltr">Inheritance vs Composition is an age old debate. The world has evolved enough that it's time to put this discussion to rest. There is no good reason to ever use inheritance in new code. Composition is functionally identical to Inheritance, produces superior outcomes, flatter class hierarchies and more flexible code than inheritance. Composition also does not violate encapsulation and avoids classes of issues produced by unexpected polymorphic dispatch to implementations. We get better class design for free as well. <div><br></div><div>In short, never use inheritance - always express the same code re-use through composition and be happy. Let's go through each of the points one by one:</div><div><br></div><div><b>1. Composition is functionally identical to inheritance.</b></div><div>This one is easy to work with. When inheriting implementations, all subclass methods have an implicit parameter (an instance of the superclass). Composition just makes this parameter explicit as a constructor argument and takes away the superclass. Code reuse through calling the superclass method can be done by just calling the same method on the composed object.</div><div><br></div><div><b>2. Composition produces superior outcomes</b></div><div>When operating in an inheritance class hierarchy, a code dependency and a type dependency become coupled. Very often, this is not necessary. For example, a 2DSurface class may have a computeArea method that you want to re-use, however your class is not a 2DSurface. With inheritance, the 2DSurface type and the code reuse are coupled. With composition, the two are separate. This is advantageous when you want to introduce code between the levels of the hierarchy. Eg. You want to add a "TimingClass" that wraps calls into to the base class. Without composition, the inheritance hierarchy deepens and a TimingClass also ends up being a 2DSurface.</div><div><br></div><div><b>3. Composition produces flatter class hierarchies</b></div><div>When operating within an inheritance hierarchy, the more variants or ways the code gets reused, the levels of the hierarchy grow deeper with "leaf" level code overriding the base class code. Code with more than 2 layers of hierarchy is very difficult to manage since you have to trace up and down layers of code that are mutually calling each other. Composition forces a single, clean extension point in the code and adding in classes like the "TimingClass" above do not increase the depth of the inheritance hierarchy.</div><div><br></div><div><b>4. Composition does not violate encapsulation a.k.a. friendly refactoring</b></div><div>Changing a base class method's access and mutations of a base class member is virtually impossible in a non trivial class hierarchy (because subclasses may rely on protected methods or direct access to state which violates encapsulation). The solution is to solely depend on the public api methods of a class (so that internal state may be refactored without affecting behavior). Composition forces this in a direct manner (enforced by the compiler).</div><div><br></div><div><b>5. Composition prevents polymorphic dispatch bugs</b></div><div>A common (bad) design pattern is to have an abstract base class call the abstract method that must be implemented by subclasses. This produces bugs when code is called across different levels of the hierarchy. Check out the <a href="http://igstan.ro/posts/2011-09-09-how-inheritance-violates-encapsulation.html">first example on this page</a>.</div><div><br></div><div><b>6. Polymorphic dispatch can be implemented better through generics and type bounds</b></div><div>The common process of dispatching through a single "super-class" hierarchy is antiquated. Generics with type bounds provide a much better substitute. Eg. Shape.getArea() doesn't need to be implemented by writing all code against the abstract Shape class with the Shape class providing .getLength() and .getWidth() that get overridden. A much better way is to implement finer grained interfaces (traits) of the form: ShapeWithArea (.getArea()), RectilinearShape (.getLength(), .getWidth()) . Composition can then fully express just the needed dependencies (eg. Class<T extends ShapeWithArea> if it just needs a ShapeWithArea but doesn't need a RectilinearShape and if it needs both, the type parameter becomes Class<T extends ShapeWithArea, RectilinearShape>). </div><div><br></div><div><b>7. Dependency injection / Testability is much easier with composition</b></div><div>In composition, the superclass is just another dependency of the class and thus can be mocked and faked. This is much harder to do when trying to mock out the superclass itself in the type hierarchy if inheritance is used. Even if we tried to stub out the superclass methods, we wouldn't know if new ones got added.</div><div><br></div><div>Overall, all the advancements in our understanding of type theory, category theory point to composition being the "right" abstraction for composing code. Inheritance is a special case of composition (a trivial subset), has worse properties around unit-testing, dependency injection and code maintenance. Given the above, inheritance from concrete or abstract classes (especially those with any form of state) should be strongly avoided. Instead, implementation of interfaces + composition should be the strongly preferred approach. </div><div><br></div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-76459013572693164022019-05-21T02:01:00.001-07:002019-05-21T02:01:20.844-07:00Managing Humans with Math<div dir="ltr">I was in a funky corner of the internet, reading about reinforcement learning when I chanced upon <a href="https://thegradient.pub/the-promise-of-hierarchical-reinforcement-learning/">this article</a> that compares a number of reinforcement learning algorithms. Since we as humans are glorified neural networks amenable to reinforcement learning and companies are nothing but hierarchical relationships between humans, it was very interesting to go through that article with a business point of view.<div><br></div><div>Long story short, the article figures out various conditions in which the algorithms are able to break down a task recursively, the optimality convergence conditions and the "knowledge" required of the lowest level agents to learn successfully. From my reading of the article, I'm most drawn to the "Options" style of management: work with high quality people, give them maximum freedom to act and step in only at decision points.</div><div><br></div><div>This is the style of management that I'd grown used to at Google (and that I've seen work really well there) and is quite in contrast to other managements that I have experienced. Not to say that the other managements were ineffective, they just took different routes to get to the same goals.</div><div><br></div><div>In summary: prefer the "Options" style of management, work with smart people, give them all the freedom they want and deserve, provide a strong learning function and clear feedback on how they're performing. Step in only at "Choice" points and then too defer to people with context of the problem. Most decisions are reversible, so just take one and let's keep going!</div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-63811436347648754962019-02-12T11:12:00.001-08:002019-02-12T11:12:39.157-08:00Hermetic MySQL in the modern world<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">So you're looking to run a MySQL docker instance without any environmental dependencies? Here's how to do that:<div><br></div><div><font face="monospace, monospace">$ docker run --name mysql-test -e MYSQL_ALLOW_EMPTY_PASSWORD=true -e MYSQL_DATABASE=testingdb -e MYSQL_USER=scott -e MYSQL_PASSWORD=tiger -p 3306:3306 mysql/mysql-server:latest</font></div><div><br></div><div>Documentation on the various environment variables are located here:</div><div><a href="https://hub.docker.com/_/mysql">https://hub.docker.com/_/mysql</a> <br></div><div><br></div><div>To connect to your MySQL instance without depending on my.cnf:</div><div><br></div><div><font face="monospace, monospace">$ mysql --no-defaults --host 127.0.0.1 --port 3306 --user scott --password --protocol tcp testingdb</font><br><div><br></div></div><div>That's it. You have a running instance of mysql from a docker image and you're connecting to it with a generic mysql command line client. Have fun.</div></div></div></div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-7254916118826825742019-02-01T17:46:00.001-08:002019-02-01T17:46:52.949-08:00Alerts should be actionable a.k.a.: Do not email on success!<div dir="ltr"><div>Following the Unix philosophy: Do one thing, do it well and be quiet about it.</div><div><br></div><div>In software engineering, if you're writing a system that's useful and suddenly, one day, you think it's nice to notify users via email that their useful thing is being done, you're making a mistake. </div><div><br></div><div>Emails from software systems should be actionable: if a system is sending an email to a user, it should be helpful, provide enough context about who it is, where it's running, who owns / runs it and what the problem is that requires human attention. Ideally, the alert email should clearly specify the next steps and the dashboards that can be used to ensure that the problem is fixed.</div><div><br></div><div>The worst offense of the system is to send out success emails. This fails on 2 counts:</div><div>1. Success emails are not actionable - if I read a success email, I am informed and I promptly create a Gmail filter to never see another success email from the system again. The system made me do active work to ignore it.</div><div>2. Success emails are not trackable - if I want to see what's the ratio of success to failures of the system, Gmail is a terrible way to do it. From first hand experience, measuring alert volume over time in Gmail is a time sink. Please build a dashboard and make the world a better place. Your future self will thank you.</div><div><br></div><div>The best alert emails are those that just tell me the commands to fix the problem. Oh happy me, I don't have to read a runbook, talk to people, poke around dashboards to see what the problem is. Run a few commands and presto, it's fixed.</div><div><br></div><div>Take some time and make your life better, don't send success emails and send actionable emails on failure.</div><div><br></div><div>Cheers!</div><div>Divye</div><div><br></div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-46802645452704730412019-01-30T13:46:00.001-08:002019-01-30T13:46:56.419-08:00How to debug a crashing docker container<div dir="ltr"><div dir="ltr"><div>If want to run your docker process with some tweaks because it's crashing in your docker container and causing the container itself to stop (without giving you a way to inspect the files on the image), here's the magic command to start it with just bash.</div><div><br></div><div dir="ltr">(I found this after quite a bit of hunting on the internet, the magic flag is --entrypoint and don't forget the -s at the end)<div><br></div><div>Here's a sample command:</div><div>docker run -it --entrypoint /bin/bash $IMAGE -s<br></div><div><br></div><div>Sourced from:</div><div><a href="https://vsupalov.com/debug-docker-container/">https://vsupalov.com/debug-docker-container/</a><br></div></div></div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-86231728481630909402019-01-23T08:04:00.001-08:002019-01-23T08:04:52.384-08:00How to be an Effective Engineer?<div dir="ltr"><div>Read this book by Edmond Lau: <br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhy3j_blWzNtfrOnFewbchFt-GaoGBm-ikG7EduwkW96gjoSeIfIt6_EkityNboJxT_7IFY_bMLtV0QiCyCyf3Ej1FAxk0Q818ntxDTexj3ZznD4mamc4No7QNWd5E446XrqQR0ig/s1600/Screen+Shot+2019-01-23+at+8.02.14+AM-792438.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhy3j_blWzNtfrOnFewbchFt-GaoGBm-ikG7EduwkW96gjoSeIfIt6_EkityNboJxT_7IFY_bMLtV0QiCyCyf3Ej1FAxk0Q818ntxDTexj3ZznD4mamc4No7QNWd5E446XrqQR0ig/s320/Screen+Shot+2019-01-23+at+8.02.14+AM-792438.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6649723890874841138" /></a><br></div><div><br></div><div>Highly recommended. His experiences parallel my own. The book is written engagingly, quick to read and succinct in its delivery. </div><div><br></div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-60844265955690874892019-01-04T09:16:00.001-08:002019-02-01T18:47:35.910-08:00Understanding Divye<div dir="ltr" style="text-align: left;" trbidi="on">
<span id="gmail-docs-internal-guid-2f0b7693-7fff-9430-76fb-1258a1adbdc4"></span><br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span id="gmail-docs-internal-guid-2f0b7693-7fff-9430-76fb-1258a1adbdc4"><span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">I typically share the contents of this document the first time I meet someone I'm going to be working with for a long time.</span></span></div>
<span id="gmail-docs-internal-guid-2f0b7693-7fff-9430-76fb-1258a1adbdc4"><br /></span>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span id="gmail-docs-internal-guid-2f0b7693-7fff-9430-76fb-1258a1adbdc4"><span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Style of communication:</span><span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"> </span></span></div>
<span id="gmail-docs-internal-guid-2f0b7693-7fff-9430-76fb-1258a1adbdc4">
</span>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span id="gmail-docs-internal-guid-2f0b7693-7fff-9430-76fb-1258a1adbdc4"><span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Direct and clear. I don't do well reading between the lines. If you would like me to know something, please say something. </span></span></div>
<span id="gmail-docs-internal-guid-2f0b7693-7fff-9430-76fb-1258a1adbdc4">
</span><br/>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span id="gmail-docs-internal-guid-2f0b7693-7fff-9430-76fb-1258a1adbdc4"><span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">What is important to me:</span></span></div>
<span id="gmail-docs-internal-guid-2f0b7693-7fff-9430-76fb-1258a1adbdc4">
</span>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span id="gmail-docs-internal-guid-2f0b7693-7fff-9430-76fb-1258a1adbdc4"><span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">High quality code and a highly productive, functional team. I don't believe in posturing. I believe in commitments and delivery. Delivering more and talking less is a good way to show impact. </span></span></div>
<span id="gmail-docs-internal-guid-2f0b7693-7fff-9430-76fb-1258a1adbdc4">
<br /><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">What kind of leader I'm trying to become:</span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">One who can grow people. I'm driven by Mission, Vision, Values.</span></div>
<br /><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">What I value in an organization:</span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Transparency, Trust and Integrity. If you have a problem, speak up. Don't ever go behind someone's back. Bring problems up together so that I can see both sides represented fairly. Be direct.</span></div>
<br /><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">What I seek in people I work with:</span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Ability to execute and potential to grow. I will endeavor to make you a better engineer or leader.</span></div>
<br /><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">How I believe our industry operates:</span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Our industry is a factory - our deliverables are software and running services. Computer Science is a predictable industry with unpredictable schedules. The when might be unclear but the deliverable should be rock solid.</span></div>
<br /><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">My planning philosophy:</span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Plan your work and deliver on your own benchmarks. Look out and adjacent. </span></div>
<br /><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">My operating style:</span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">I will work with you and always give you the "why". On the rare occasion where "because I say so" is required, I will let you know so. </span></div>
<br /><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">My background:</span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Born and brought up in India. Son of two doctors, have a sister who is a doctor. Small town guy.</span></div>
<br /><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">My favorite quotes:</span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">"If I do what I say I will do, I guess that means you can trust me. If I don't, I guess that means you can't."</span></div>
<br /><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">My biggest weakness:</span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">I evolve, I don't let you know that fast enough. </span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">If you talk to me and I tell you I support X and 24 hours later, I'm supporting !X, it means that I've gained new information that has changed my mind. Talk to me and clarify what changed. I have "strong opinions, loosely held". </span></div>
<br /><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Meetings</span><span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">:</span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<a href="https://conantleadership.com/ceo-manifesto-better-meetings/" style="text-decoration-line: none;"><span style="background-color: transparent; font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">https://conantleadership.com/ceo-manifesto-better-meetings/</span></a><span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<a href="https://conantleadership.com/a-highly-effective-leadership-habit-for-building-relationships/" style="text-decoration-line: none;"><span style="background-color: transparent; font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">https://conantleadership.com/a-highly-effective-leadership-habit-for-building-relationships/</span></a><span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<li style="background-color: transparent; color: #000000; font-family: arial; list-style-type: disc; vertical-align: baseline; white-space: pre;"><span style="font-size: 10pt;">What's working?</span></li>
</ul>
<ul style="margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<li style="background-color: transparent; color: #000000; font-family: arial; list-style-type: disc; vertical-align: baseline; white-space: pre;"><span style="font-size: 10pt;">What's not working?</span></li>
</ul>
<ul style="margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<li style="background-color: transparent; color: #000000; font-family: arial; list-style-type: disc; vertical-align: baseline; white-space: pre;"><span style="font-size: 10pt;">What's needed?</span></li>
</ul>
<ul style="margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
</ul>
<br /><div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Things I would like you to know:</span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<a href="https://www.divye.in/2018/08/great-workplace-habits.html" style="text-decoration-line: none;"><span style="background-color: transparent; font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">https://www.divye.in/2018/08/great-workplace-habits.html</span></a><span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<a href="https://www.divye.in/2018/05/actionable-production-escalations.html" style="text-decoration-line: none;"><span style="background-color: transparent; font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">https://www.divye.in/2018/05/actionable-production-escalations.html</span></a><span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"><a href="https://www.divye.in/2016/03/what-is-quality-as-it-relates-to.html" style="text-decoration-line: none;">https://www.divye.in/2016/03/what-is-quality-as-it-relates-to.html</a></span><span style="background-color: transparent; color: rgb(0 , 0 , 0); font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
</span><div style="text-align: left;">
</div>
</div>
Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-51950073198246310622018-12-04T08:19:00.001-08:002018-12-04T08:19:40.281-08:00AI Expo 2019: Tim Jurka (LinkedIn, Director Feed AI) - Part 4 of 4<div dir="ltr">I recently attended the AI Expo 2019 at the Santa Clara Convention Center. Notes are from my understanding of the talk. Any errors are mine and mine alone.<div><br></div><div> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">LinkedIn: A look behind the AI that powers the LI feed</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Tim Jurka (Dir. Feed AI)</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">The talk was focused on the objectives of LinkedIn's Feed. The talk was focused to a high level (exec) audience. While I was familiar with the space, the objective function formulation and presentation was interesting:</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">The recommendation problem for LinkedIn is maximizing Like/Comment/Share CTR + downstream network activation (virals) + encouraging new creators.</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Problem Formulation:</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">P(click) + P(viral) * (alpha_downstream + alpha_creator * e ^ (- decay * E[num_response_to_creator])</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">alpha_downstream accounts for downstream effects; alpha_creator penalizes popular creators to induce diversity.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">General approaches (Toolbox):</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Multi Objective Optimization (ads vs organic content).</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Logistic Regression: Features, Embeddings and Decision Trees (XGBoost for Feature Importance), occasional pruning<span class="gmail-Apple-converted-space"> </span></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Auto tuning of weights of the MOO to correct for drifts in accuracy of the component models (to meet a product goal).</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Running the Team:</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">The goal of the team is to maximize: Successful Experiments / Total Experiments Run</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Two approaches: maximize successful experiments, minimize unsuccessful experiments.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Maximize successful experiments:</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">1. Hire the best talent</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">2. Increase the total number of experiments being run online.</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">3. Automate deployments, parameter tuning, retraining, rebasing, ramping to maximize developer throughput.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Minimize unsuccessful experiments:</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">1. Replay models over historical data to figure out whether they would perform better than the current model before moving to online.</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">2. Compute actual business metrics, Determine precision @ 1, precision @ top3/top5 over a randomized sample of data.</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">3. Use bandits to figure out how to be intelligent about collecting data and exploiting the current model.</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">4. AI for Ai: Auto retrain and evaluate models. Identify promising features and ramp online. Find optimizations for existing models automatically. Highlight promising variants to engineers.</p></div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-74519654316927295562018-12-04T08:16:00.001-08:002018-12-04T08:16:51.388-08:00AI Expo 2019: Emilio Billi (CTO, A3Cube) - Part 3 of 4<div dir="ltr"> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Why and How the computational power influences the rate of progress in the technology</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Emilio Billi CTO A3Cube Inc</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Background: ML, Big Data & Analytics, AI, HPC.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">This was a big data infra focused talk. The speaker had a background in systems infra with past DoD experience. Not the most engaging delivery, but really nice takeaways:</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Moving 128 bytes on a CPU using 100Gbit ETH: CPU waits 8900ns for nothing (~7.1M compute ops lost);</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Moving the same 128 bytes using optimized RDMA intra-cluster costs 1200ns CPU time (~0.96M compute ops lost)</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">You get 6M ops extra per second for ML. That's a great acceleration for ML workloads.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Basic contention: ETH, TCP, slow storage is legacy technology.</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">The clusters of the future will look like the supercomputer systems of today:</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">1. Low latency converged parallel file systems (think S3 for the cluster).</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">2. Built in Distributed Resource scheduler (think Kubernetes for the cluster).</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">3. Cooperative RAM over network fabric (RDMA over Infiniband)</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">4. Cluster wide sharing of Accelerators (eg. GPUs / FPGAs) (</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Do the above without changing the server setup.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Case Study:</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Optimizing the stack with these features got 64 nodes to the same compute capacity as 360 AWS nodes (6x speedup).</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Work was done for a DoD project.</p></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-45007658536641332462018-12-04T08:14:00.000-08:002018-12-04T08:15:39.315-08:00AI Expo 2019 - Prakhar Mehrotra (Walmart Sr. Director of ML) - Part 2 of 4<div dir="ltr">I attended the AI Expo 2019 at the Santa Clara Convention Center where Prakhar gave a talk. Notes are my summarization of the talk. Any errors are mine and mine alone. <div><br></div><div> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Walmart - Prakhar Mehrotra (Sr. Director of ML, previously at Uber)</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Walmart has huge scale: 0.5Trillion+ revenue, 3000+ stores with massive physical footprints, a massive global supply chain, Jet.com, Walmart.com, Shoes.com, Flipkart.com and it keeps growing.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">The talk was focused on Walmart's application of ML, the contrasts of Uber-style surge pricing vs Walmart's fixed in-store pricing ("everyday low pricing"). A focus point was causality over correlation: understanding Walmart's customer and its supply chain (the Why?). Their primary domain was solving for shelf placement of inventory. Other interesting problems were inventory management, bridging the online and offline worlds (if we ship from warehouse, it's going to cost you X but if you pick up at this store where it's in stock it's X-3). The takeaway: Omni-channel shoppers are changing the customer profile for Walmart but adjusting for that isn't as simple as taking the .com purchase data and feeding it into the models.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">The crux of the talk was a discussion of (NP-hard) Causality finding Bayesian Networks; Walmart worked around the NP-hardness by manually decomposing the Walmart supply chain into relatively independent units and their interrelations (suppliers, merchants, inventory, pricing, warehousing, transportation etc.). There was also a discussion about counterfactuals ("would sales have declined if I had not placed this item on promotion?"). A/B Testing is hard in the Walmart space because pricing is fixed in-store (and not per-shopper).</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Fundamental datasets and metrics: Imagery, Similarity, Variants, Attributes, Classification, Quality, Scoring, Analysis</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">They're solving for:</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">1. Interventions (should I provide an offer to this user or intervene in another way?)</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">2. Associations (shelf placement, cart composition, suppliers)</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">3. Counterfactuals (what-if analysis)</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Walmart's tech stack and algorithms:</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Models/Algos: NN, Bayesian nets, Structural models</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Tech Stack: Hive, Hadoop for big data, Teradata for medium-data, ETL systems, Jupyter for ad-hoc, hyper parameter optimization systems, CPU&GPU training, Scala and Python as primary languages for data scientists.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">The final food for thought was: how fast will the online and offline worlds converge on dynamic pricing? What does that look like to the customer?</p></div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-81773854383363613002018-12-04T08:11:00.000-08:002018-12-04T08:12:12.609-08:00AI Expo 2019 Notes: Ameen Kazerouni (Zappos) - Part 1 of 4<div dir="ltr">I recently attended the AI Expo 2019 at the Santa Clara Convention Center where there were talks on various ML platforms. I'm leading the ML Training Infra platform at Pinterest. These notes are from those talks and are summarized from the speaker's presentations. Any and all errors are mine alone. Hope you find the below useful.<div><br></div><div> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">Speaker: Ameen Kazerouni (Zappos)</p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">The scope of the talk was Zappos' ML Platform ecosystem, the problems they faced after solving the basic 5: Problem specification, dataset design, model selection, training and validation. A condensed list of their issues is:</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">1. Data management: data lifetime (how long?), security footprint (who should have access?), governance issues (who did have access?), data scrubbing and anonymization (avoiding privacy issues under GDPR).</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">2. Team: There are very few unicorns that can do PhD statistics, ML math and write distributed systems. They hire for domain competence and the ability to communicate to others in the ML domain. They're seeing a changing ecosystem where ML scientists and engineers are collaborating rather than a waterfall model where a model is built and a data engineer puts it into production.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">3. Language dependencies: ML researchers and Statisticians are using Python Scikit, TF, R for analysis; the data team in production is using Scala and Java - finding a way to bridge the language gap: Protobufs for all data intermediates and their generated code for typed access to the data across languages and systems; serialized (typed) model representations for delivering into production.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">4. Tier 1 SLAs: Making the latency - feature value tradeoffs (how slow is too slow?): precomputed models, ensemble models (personalization on top of precompute) help break down complex problems into simpler solutions that are manageable from a latency perspective.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">5.<span class="gmail-Apple-converted-space"> </span>Model Interoperability: Their Java serving systems use DL4J for TF model serving. Some systems use TFServing. For model representation, they rely heavily on PMML and (unclear) but they round trip through ONNX for model format conversions across language boundaries and DL systems (eg. Scikit, Spark, TF).</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">6. Productionization: There's an emphasis on constant communication between teams, they decouple systems through microservice APIs, they aim to serve product features (not recommendation algos) and they degrade gracefully from high precision to high coverage models.</p> <p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica;min-height:14px"><br></p> <p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:Helvetica">7. Product Roadmap: Learning: ML everywhere doesn't work as a driver of the product roadmap. ML is supporting cast. Build the product roadmap first and fit ML later.</p></div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com0tag:blogger.com,1999:blog-34173777.post-79302911784383980982018-10-27T16:22:00.001-07:002018-10-27T16:22:49.966-07:00The toughest interview questions asked recently<div dir="ltr">During a recent round of interviews, I was asked these 5 questions that I found particularly interesting / challenging. They each covered a different interesting pieces of computer science that I thought worth sharing. Check them out and see what you think about them:<div><br></div><div>1. A set of overlapping ranges are provided in [start, end) format. Find the max number of ranges that overlap each given range. Improve your solution to O(N log N) complexity (basic solution is: O(N^2) complexity for a set of ascending ranges [1,4), [2, 4), [3, 4) etc.).<br></div><div>2. Implement a multithreaded producer / consumer queue using condition variables.</div><div>3. Implement a multithreaded rate limiter (token bucket with defined capacity) using no hard-coded poll durations and without a background thread for "filling". Discuss fairness vs head of line blocking tradeoffs of the implementation.<br></div><div>4. Implement a multithreaded scheduler that executes tasks repeatedly at specified time intervals which manages task overruns (task time > interval time) and does not skip scheduled run points. Estimate the maximum number of outstanding threads that a task may produce.</div><div>5. Given a matrix of size NxM with 0 and 1 entries, find the number of connected regions in the matrix. Extend your solution to handle updates of 0 entries to 1s that may connect existing regions. Expected solution is O(NxM) complexity in the worst case even for the second case with updates being approximately O(1) and space being O(NxM). Hint: Disjoint sets. </div><div><br></div><div>For System Design, the following were interesting questions:</div><div>1. Design typeahead search (given a query prefix, provide completions). </div><div>The interesting part was managing hotspots for small query lengths, the use of a KV store for serving given the low latency requirements and managing personalization.</div><div>2. Design an ad click prediction system (ML infra pipeline).</div><div>3. Design a lineage system for tracking input data to ML model training to avoid sample bias.</div><div><br></div><div>Overall, interviewing was a pleasant process that refreshed a lot of skills. Let me know about other interview questions that might have been interesting for you.</div><div><br></div><div>Cheers!</div><div>Divye</div></div> Divye Kapoorhttp://www.blogger.com/profile/05987327171759168375noreply@blogger.com1