46. Go at Heroku

Rishabh: Yeah, so I've been at Heroku for about a year and a half. I'm pretty new to the industry. This is actually my first job, and a bit of a brief history about myself. I recently graduated from UC Berkeley about two years ago, after studying some computer science. And prior to that, I was actually a Futureforce intern on one of the other clouds at Salesforce, called Salesforce IoT Cloud. And most recently I've started as a software engineer on the runtime infrastructure team.

Johnny: Very cool. So as for me, I'm relatively new into Heroku still. I'd say at least relative to the two of you. It's been what? Maybe my six... I think I'm entering my sixth month at Heroku? I joined as an SRE, and Ed and I are also part of the SRE org within Heroku. And this has been an awesome journey for me.

Johnny: So Ed, I know you've been at Heroku the longest, so I know internally right now that you are definitely one of the people who looks after the buildpack. So, whenever there's a new version of Go that comes out, you are one of the first people the org looks to, to make sure that we're running the latest and greatest. We're patching, and we're basically keeping up with the releases of Go.

Johnny: Can you give us a little bit of insight into what the process is like? Is it something that's difficult to do, hard to manage? Do you have automation around it? How easy is it to keep the buildpack fresh, and serving our customers?

Ed: Keeping Go up to date on the buildpack itself is relatively straightforward. We have a little script inside the buildpack repo itself, that we can use to do the work, to pump versions. And then things like CI and CD for the most part validate that the existing tests and everything are good.

Ed: There are challenges supporting every new major release, so like Go 1.12 to Go 1.13 et cetera, et cetera. For instance, the defaults is currently still one of the Go 1.12 lines, because there are a whole bunch of implications with turning to modules being the default on... And yeah, we're not yet sure we understand all of the implications of it from the standpoint of the buildpack anyway. And how that affects all the users of the buildpack. But the general process of bumping for point versions, and things like that is really simple.

Johnny: Pretty much since I've been in Heroku every new project, it sounds like it's being written in Go. But we still have a lot of projects that had been written in Ruby. Some new projects are either Ruby, and we have some Elixir as well.

Johnny: So we're truly a polyglot sort of organization. We have a set of go-to languages that we use, but Go is increasingly being one of the first ones we look at. Especially for backend kind of systems, and services that needs to talk to each other, that kind of thing. So how we do metrics, and how we do... when folks hit the API at heroku.com, most of that stuff is in Go. Private Spaces are run that most of that stuff is controlled in Go. So we have a lot of Go.

Johnny: So does that mean we have a lot of Go expertise in-house? And along those lines perhaps Rishabh, you can sort of chime into this one. How easy has it been for you, right, coming out of school and this being your first job? Were you doing Go at school, or did you learn Go on the job? How easy has it been for you to sort of learn Go, and be productive with the language at Heroku?

Rishabh: At college actually I never even heard of Go. The main languages that I used was actually Java and Python. And coming to Heroku, and starting my first job, it was quite a bit of a surprise. Just hearing that I would be working in Go and Ruby, and learning a new language.

Rishabh: So specifically there were like a couple of resources that really helped me gain a bit of an understanding about the Go ecosystem, and learn how I can actually start writing idiomatic Go. For example, the runtime team, specifically we have a program called Runtime University.

Rishabh: And it has a huge section on Go, which it also references Go by Example, and other resources where an engineer who is relatively new to Go, or wants a refresher, can actually go in, and do some practical exercises and learn. For example, syntax about the language, or how to actually write good Go.

Rishabh: Also, for myself, one of the biggest things that has really helped me learn Go at a good rate, is reading a lot of Go code. So, taking an existing microservice that we have, and then reading it end-to-end really helped me understand how to build a service in Go.

Rishabh: And then also when I'm making changes to an existing service, reading that whole code base was very helpful. So, I think overall as a new engineer, we have some patterns at Heroku such as Runtime University that really support a new engineer to learn some Go.

Johnny: So one of the things that you, as a Go developer start to hear as a new Go developer, you start to hear more and more of this concept of idiomatic Go, right? So you could write Go that looks kind of like Java a little bit, or some other language that you're familiar with. But you wouldn't be doing it the way most Go developers do. Right?

Johnny: And this is something that basically is more subjective than it is a specific set of... If you don't write it a certain way, then your code will come back kind of thing. But it's more along the lines of this is the expectation right out in the Go community. When you look at a Go code base, and Ed, I know you've been the leading person internally behind our Go Design Guide. Maybe you can add some flavor there, that says basically how much of the community's concept of idiomatic Go... How much of that basically is impacting, or influencing how we approach Go development internally?

Ed: I like to think of our Go Design Guide. First of all, it's a living doc, right? So, I expect it to evolve over time. It's an attempt to basically try to get all the engineers on the same page about how to structure projects, and how to write Heroku's version of idiomatic Go.

Ed: I do feel that a lot of things captured in that Guide are inspired by the things you see in the community at large. For instance, the Guide has many call-outs to everything from Dave Cheney's blog, to Jada's blog, to the Go Wiki, and things like that.

Ed: And usually has texts for purposes of clarification, or to better relate the contents that are linked out, to the context of Heroku, and the types of services and tools that we need to write.

Ed: And like I said, some of the main motivation for me, starting at several years ago was, as the Go language owner then, I would read a lot of Go code internally, and different projects were trying different things, which is something we want to continue to encourage.

Ed: But as the code bases go from experimentation phase into production, and things like that, in order to have that code base easily maintainable and modifiable by a large swath of engineers, my opinion is that it's better if we do Thing X in the same way across as much of our code Go basis, as possible.

Ed: And the attempts for the design guide is to state what those ways are, and the reason and motivation behind why we feel that is the right way for us to do it. And to provide examples, and all that context for posterity.

Ed: And again, since it's a living doc though, it doesn't mean it's going to all, like if we say do this for these reasons, it doesn't mean in six months from now, it might not give different advice for the same thing. Because we've learned something in the interim, and we have a process in place through an internal RFC, or request for comment, process to modify that doc.

Johnny: Mm-hmm (affirmative). Rishabh, I know you want to add some flavor to that.

Rishabh: Yeah, so this is actually something I forgot to mention earlier. The Go Design Guide for myself, has actually been a fantastic resource as a new engineer. I think, for example, when I start working in a new code base with Go, learning the language as a beginner, you're always looking and asking around for patterns or questions around, is the way that I'm writing this correct?

Rishabh: And does it make sense? And the Go design guide being there as an artifact that you can look at, and see patterns and standards, which to most people make sense, provides a sense of proof, or like prior art to say, "Hey this code makes sense because the Go Design Guide references this pattern. So, I'm going to implement it this way."

Rishabh: And also, when doing code reviews for example, I also referenced the Go Design Guide as a reason as to why certain code should be written in a certain way. So it provides a lot of examples, and provides a lot of explanation as to writing a good standard of Go.

Johnny: So, as someone who say, sort of learned Go on the job, I do want to sort of ask though, as a newcomer to go, what have you found to be the hardest thing to learn about the language? What would have been your stumbling blocks? What things you keep coming back to, and over and over, that it has taken some time to sort of sink in for you?

Rishabh: Yeah. So I think there were a few things that I found difficult along the way when learning Go. One of them, was actually dependency management. So different code bases, used different tools to manage their dependencies. Some code bases that we have, use dep. Others, use Go mod, or modules. So initially when I was starting, I had no idea how dependency management worked in Go. So knowing when to use either, was a big issue.

Rishabh: Also, a difficulty along the way that I had, was knowing that the code that I was writing was idiomatic. So I would look to other resources, such as the Go Design Guide, or Dave Cheney's blog, to really validate whether or not the code that I was writing, was idiomatic.

Rishabh: And another issue, or another difficulty along the way was knowing if the pattern that I was using to solve a problem was appropriate at the time. And it's very similar to writing idiomatic Go, but knowing if the solution that I had was appropriate.

Johnny: And have you found basically team members to be a go-to resource as well? So how much of a face-to-face have you had to do, right? And how much of it do you think you spend, in pull requests, going back and forth? Did you find... Have you thought... I guess what I'm asking, what has the experience overall been like for you? For somebody who didn't know Go before, and has sort of been learning on the job?

Rishabh: Yeah, definitely. So I think being very new to Go, I'm just asking a lot of questions, and always questioning why something was done, the way it was done. So just being open to asking a lot of questions, especially to senior engineers who've been writing Go for a multitude of years, and specifically in pull review or pull requests, asking targeted questions to the reviewers, and saying, "Hey, does this section of code make sense, based on this pattern that I found here?"

Rishabh: And really soliciting specific feedback. So, yeah. I also definitely took advantage of pairing, while doing code reviews. So, we would hop on a call together, look at the code together, and I would explain, and do some kind of a walk-through, explaining the reasons why I took, when writing a certain piece of code. And really got a chance to add some more flavor to the code review process.

Johnny: So Ed, I'm curious. Do you think your journey, I mean you've been doing Go for many, many years. Where do you see yourself at this point in your career, in terms of your expertise with Go? Do you still think the language has a lot to offer you? Do you still think there's a lot to learn? Has the way you've written Go along those lines, has the way you write Go today, has that changed over the years?

Ed: Oh, it's totally changed. Go's not my first language. Maybe my seventh, eighth, sixth? Something like that, language, professionally. I did what everybody does, or at least I assume what everybody does, when they first learn a new language.

Ed: And that is mix in styles picked up, and/or used in the language that they used previously most. Previous to Ruby for instance, I had used Python a lot. So when I first came to Ruby, I wrote my Ruby like Python, and was quickly yelled at by a bunch of Rubyists. So when I came to Go, I wrote what I currently called Guby, the mish-mash of Go and Ruby, which I've seen plenty of. I've also seen plenty of Gython, I've seen plenty Gava.

Ed: I've seen all of these, and I'm not, I don't say that to cast derision on the people doing that, but I believe it's also important to recognize that's part of the process of transitioning, and learning a new language. As far as a new people coming into Go though, I think the best thing we can do for new engineers, is pair with them, and pair with them on specific tasks.

Ed: I have offered, and continue to offer pairing with any engineer, on any project, and those who have taken me up on it, the sessions have been most productive. I believe for myself as well, experiencing the beginner mindset, which I'm so removed from now.

Ed: So experiencing that firsthand. And then I hope for them as well, because through that process of pairing, we can explore the design philosophies, that I think are idiomatic Go, where some early attempts at this people I was pairing with, attempted to use me as a Go encyclopedia. And I feel like neither one of us, neither party, came away from that experience feeling they had gotten good use of that time.

Ed: So that's something else I recommend for more senior Go engineers. When you offer something, or do pairing with somebody, try to do it on a... Like have a specific focus, and a goal to accomplish. Refactor a new, implement something new. Both of those, whatever. Not just like, "Hey look at this code, and tell me how it could be better." Because there's no absolute right answer to that. It involves too many contexts that are specific to the project. How the deployment, the engineer's involved, et cetera, et cetera.

Johnny: And where do we see Go going at Heroku? Obviously we've already mentioned that a lot of our control playing software is written in Go. And everywhere I turn, I think... And you know within the organization, I see more and more Go. Do you think that's the language, the bulk of the engineering teams at Heroku are going to be writing software for your foreseeable future, in your mind?

Ed: I'd certainly like that. I suspect a lot of Rubyists might be more immediately attracted to something like Elixir. That's a theory. Not something I know. But I think Elixir can give the Rubyists, some of the advantages that you would get from Go, while keeping some of the things that they like about Ruby, and those are the same.

Ed: A lot of those things are the same things that kind of drive me crazy about Ruby, and Ruby and Go are different. With that said, I do think Go is embedded into our infrastructure, to the extent that it will likely continue to be one of one to four primary languages, that Heroku ends up using. Especially while things like Docker and Kubernetes, and stuff like that, continue to dominate mindshare in the industry.

Johnny: So Rishabh, what are you most looking forward to on your end with the gloss of working with Go? Any project you get to work on, that involves Go you're happy with? Or do you have a specific kind of project that you find the most fun to write in Go? Where's are your head at?

Rishabh: So, currently I have the pleasure of actually writing a new service, that actually deploys to our Kubernetes clusters in a secure, and scalable fashion. And that service was done in Go. So, it's been a journey for myself in the sense that, it's a foundational project, where I get to actually bootstrap a new service in Go, and really figure out for myself, good design patterns, and writing good Go code. It gives me the space, and the time to actually experiment and learn, which I'm pretty happy about.

Johnny: Well, I am glad we're able to have this chat. And I personally learned a bit more about how Heroku uses Go, and knowing that there are folks who are brand new to Go in your organization as well, that are learning how to write Go, and how to write idiomatic Go. And folks like Rishabh, and also folks like Ed, who have been doing it for a while, providing some guidance.

Johnny: And me, having joined the organization, and knowing that Go was one of the determining factors for me joining the organization. I'm glad to see that there's a lot more to the org, a lot more to the services, and everything else that we're doing that's written in Go, and going that makes me happy. So, I'm grateful that we're able to get them to have this chat, and thank you for being a guest on our show today.

Rishabh: Yeah. Thank you. Thanks for hosting.

Ed: Thank you Johnny, for having me.