With the announcement that DC/OS will be deprecated and the imminent death of Mesos, it feels like the right time to write about the awesome journey I’ve been through these past 6 years with Mesos and the Mesos community.
I had originally enrolled in the PhD track after earning a scholarship from the Louis Stokes Alliances for Minority Participation (LSAMP). At the time I earned the scholarship wasn’t really sure if I wanted to finish a Doctorate but my interview with Microsoft in Seattle at the end of senior year had brought me down a few notches and I was unable to clear Bloomberg’s interviews. I wasn’t having much luck getting any interest from any other outfits either.
So with seemingly nowhere to turn to, I decided that maybe it wouldn’t be such a bad idea to stick to school and re-evaluate down the line.
2014
It was now 2014 and I was on the verge of finishing my Master’s when I got a e-mail from my advisor. One of his former students, Meghdoot Battacharya, was looking for a intern to work for the summer at eBay. I didn’t know it at the time, but this was the start of a superbly important chapter of my life. It would lead me to publishing my doctoral thesis and getting hooded. It would also lead me to the job I’ve been working at during the past three years. But most importantly, it would introduce me to a number of people that continue to be huge influences and have been incredible mentors.
During my first internship I had the pleasure of meeting Ben Hindman (co-creator of Mesos) who was at Twitter at the time. I even had the privilege of saying a few sentences about my work on allowing Apache Hadoop and Mesos to coexist in a single cluster with my mentor Mohit Soni at Mesoscon ‘14 in Chicago. It was surreal to be talking right after John Wilkes to whoever decided to stick around after his awesome talk. This was before I’d ever done a talk about one of my published works so this was by far the largest audience I’d ever talked to. The project we worked on during that summer would be the seed for what would later become Apache Myriad.
Since (for some reason) I was still hell bent on finishing my doctoral degree, I went back to Binghamton, NY in August. I immediately started talking to my advisor about what I’d worked on that summer and how Mesos could be a great platform to carry out academic experiments. After all, I thought at the time, Mesos had initially started as a research project at Berkeley AMP Labs so it had Academic credit. Eventually this would lay the ground work for Elektron, a power aware Mesos framework, and of course my doctoral dissertation.
2015
The team I’d worked with that summer wanted me to come back for another internship the following summer. Things had changed since my first internship at eBay. Plans were in motion to split PayPal from eBay with the team I was working with landing on the PayPal side. (Side note: I would not wish being an intern at a large company during a split to my worst enemy, but that is a story for another time.)
Upon my return to eBay in the summer of 2015, plans to run a joint Mesos and Hadoop cluster had been scrapped internally (most likely due to the upcoming split). Instead, we’d be working towards building a platform using Mesos and Docker Compose. The first step in the process was to pick a framework to run our workloads. The team had picked Aurora after extensive benchmarking and testing. Unfortunately, Aurora at the time only had support for the Thermos executor, which was hard-coded as the executor of choice. It would be my summer job to convince the Aurora devs to make former Aurora tech lead Bill Farner’s 2014 prediction come true and modify Aurora to enable the use of custom executors.
At the end of the second internship, I got to showcase what I had worked on with my mentors Mohit Soni and Tejas Varma at the Mesoscon 15’s hackathon winner’s circle. Custom executor were now a possibility in Aurora but it’d be some time before the feature would actually land in the project. Bill Farner, who has since gone on to become a friend and mentor, helped me tremendously in those days to land the feature and made me become a much better programmer in the process.
So I left San Jose around August with a sense of accomplishment dampened by the fact the patch was not able to land before the internship finished. During the next few weeks I’d contribute here and there to the Aurora project while Bill would be paving the way for my custom executor’s patch to land by shipping:w some preliminary patches needed for mine. In parallel, work began on my first academic paper focused on exploring the opportunity for improvements in scheduling done by Aurora in terms of power and energy consumption.
2016
Fast forwarding to the summer of 2016, I was rejoining Meghdoot’s team as an intern for the summer one last time just as the new platform was beginning to take shape. The team had added a few members in my yearly 9 month absence and was making progress at a fast pace. With the experience I had gained during the past year with Aurora, I set out to quickly land the first version of the custom executor patch. Thanks to a patch Bill had landed earlier, it was pretty straight forward to get my patch accepted. Maxim Khutornenko helped me polish up a few rough edges and it finally landed.
This time around, the team had finally settled upon the programming language we’d use for building our platform: Go. The Aurora Scheduler, however, was written in Java and the Aurora Client in Python 2. To allow the two components to communicate, Apache Thrift was used. Since the team wanted to build a platform with the Aurora Scheduler at the core, we now needed to be able to interact with Aurora in a programmatic way, preferably without leaving the Go ecosystem. Thus, that summer Gorealis was born. Inspired by go-marathon, it would allow us to talk directly to the Aurora Scheduler via Thrift calls without needing to use the client.
The third and final internship was now finished and I was back in Binghamton, NY for one last year. In December, 22nd, 2016, now with Go experience under my belt, the first few commits transforming RENDLER (a sample Mesos framework) into Elektron were landed. Two Master’s students, Pradyumna Kaushik and Abhishek Jain, would eventually take interest in my research and join the lab I was part of. PK continues to work with Elektron to this day as he is nearing the completion of his own thesis.
2017 and onwards
In June of 2017 it was time to wave goodbye to Binghamton to join my partner in the west coast. By that time, the first paper using Elektron was accepted for publication and a second one in the pipeline.
We decided to open source Elektron and even got an awesome comment from Tobi Knaup saying he’d check it out right after we announced our release.
The journey didn’t end there though. I joined PayPal as a full time employee in mid 2017 and I was eventually invited to join the Apache Aurora project by the Project Management Committee first as a committer and later as a PMC member. I worked with a number of talented engineers including: Bill Farner, Stephan Erb, David McLaughlin, Dave Lester, Kevin Sweeney, Joe Smith, John Sirois, Jake Ferrell, Santosh Shanmugham, Jordan Ly, Joshua Cohen, Zameer Manji, Mauricio Garavaglia, Nicolas Donatucci, Rogier Dikkes, Se Choi, and many others.
As the project started to wind down, I was chosen by the PMC to replace our current PMC Chair, Jake Ferrell. We would eventually vote to move the project out of the Apache umbrella as the overhead became too much for what now seems a relic of days past in the current Kubernetes dominated world.
Final words
Today’s announcement by d2iq doesn’t come to me as a surprise, but it is a good chance to stop for a moment and look back on a fantastic journey. I’ve been extremely fortunate and grateful to have been in the right place at the right time. Mesos, and by extension, its community, has had a profound impact on my life and it’s sad to see the train pull up to its final stop on its final voyage. But the memories, experiences, and friendships made are unforgettable and I hope (and fully expect) the paths of those I met through Mesos and my own will meet again.
Until then, well, just like d2iq, I’m off to kubernetes…