Spinnaker Summit 2019 Preview: As you use an open source tool like Spinnaker, the more you develop best practices and learnings. These are useful to share with others internal to the enterprise and other open source users.
Jing Vergara, principal software engineer at Salesforce, joins DevOps Chats to share a preview of her upcoming talk “Demystifying Spinnaker VM Image Baking and Deployment.” Jing shares with us what to include, and not to include, when baking a VM image, how to avoid configuration drift and how to deploy to multiple cloud providers using Cloud-init files. Jing also shares how to use open source Packer templates with Ansible, Chef and Docker, and how to build your own Packer templates.
Jing Vergara’s talk is on Sunday, Nov. 17th at 10:45 AM PT. Spinnaker Summit 2019 is San Diego on Nov. 15-17.
Transcript
Mitch Ashley: Hi, everyone. This is Mitch Ashley with DevOps.com and you’re listening to another DevOps Chat podcast. Today, I’m joined by Jing Vergara, who is principal software engineer at Salesforce. Now, Jing is going to be talking at the Spinnaker Summit in San Diego in November, and she’s gonna be talking about a really interesting topic, “Demystifying Spinnaker VM Image Baking and Deployment.” Now, don’t think we’ve changed into a baking show. We’re baking VM images, and that’s what Jing’s gonna tell us a little bit about, that whole process—pipelining and creating VMs and baking them. [Laughter] So, her talk is on Sunday, November 17th, at 10:45 a.m.
Jing, welcome to DevOps Chat.
Jing Vergara: Thanks, Mitch.
Ashley: Great to have you.
Vergara: Thank you for the opportunity to have me talk here.
Ashley: Well, I’m honored you’re on. Thank you for being here. Tell us about yourself. Tell us a little bit about what you do and maybe what parts of Salesforce product or code that you work on.
Vergara: Sure. So, I’m part of the continuous deployment team at the company. We offer Spinnaker as a Service for all of the organizations within the company. And I help teams onboard their delivery pipelines into Spinnaker.
Ashley: Oh, interesting.
Vergara: Yep, and as part of my job, I also build custom extensions to Spinnaker to support their internal business needs.
Ashley: So, you’re kind of the Spinnaker service center expert person or group that everybody goes to for help implementing Spinnaker. Is that close?
Vergara: Yeah—very, very close, yes. So, my role within the team has been changing, but yeah, it has transformed into that one. Another thing that I do for Salesforce is that I represent Salesforce at a couple of the Spinnaker special interest groups for stakes insured so that I could learn more about upcoming features of Spinnaker, so we can leverage those in our roadmap as well so that we know we can offer it to the business.
Ashley: Mm-hmm.
Vergara: And then we also contribute back to the ________ and let them know what would and would not work for us as users of Spinnaker.
Ashley: Well, and I noticed in your bio on the Spinnaker site, you have a Master’s in Computer Science from the University of Illinois at Urbana-Champaign. I used to live not too far, from there, though I didn’t go to that school. [Laughter]
Vergara: [Laughter]
Ashley: That’s a very strong C.S. degree, and it sounds like you’re putting it to good use with Spinnaker and DevOps style deployment with these kinda pipeline creation.
Vergara: Yes, right.
Ashley: Excellent. Well, tell us a little bit about what—so, we’re talking about your talk. So, describe for us a little bit about what are some of the things that are maybe a little mystical that need demystifying about Spinnaker VM image baking and deployment. What are some of the things people don’t know that you usually have to help them understand with, “Here’s how it works and here’s how we do it”?
Vergara: Sure. So, for companies that are traditionally in a bare metal world where you don’t use VM images, it’s kind of a definite shift in the mindset, right?
Ashley: Mm-hmm.
Vergara: So, the way I usually explain it to them is using the baking metaphor, right? So, when you talk about baking, you think of putting the ingredients together like flour, salt, what have you, right? You form a dough, shape the dough, then you cook it. You start with a base ingredient—like, again, the flour—and you mix it up.
So, in the VM world, when you bake an image, you start with that base OS image. You add your other ingredients, which means installing your applications or your agents to run on that VM. And then you have your standard or common configurations and possibly some data, right?
Ashley: Mm-hmm.
Vergara: You add them all up in the VM, you create a snapshot. So, that snapshot, you can now call it the VM image. So you can save this image into a store, and once you have that image, you can use it for deployment. So, deployment, it’s basically building or provisioning one or multiple VMs out of that same image.
Ashley: Okay. Now, leading into this, the thing that kinda kicks it off is that, I’m not sure how it works at Salesforce, but is it something like a Jenkins process that then kicks off this creating of the image? How does that work?
Vergara: Right. So, we have our continuous integration system. So, we have a few and we’re standardizing that at Salesforce as well, right? So, we have those artifacts built by those CI systems. We have our Docker images so they get stored, we put them into the cloud, and then we have triggers. So, we listen for those changes. So, Spinnaker supports a Pub/Sub subscription that triggers the pipeline. So, then, once we have those architects, we can start the baking process.
Ashley: Great. Excellent. Well, that makes a lot of sense. Think of it as all of the pieces of software, OS on up, that are ingredients, right? Just elements of what’s gonna go into that image.
Now, I imagine this is something that’s happening frequently if you’re doing a lot of deploys, right? Is this happening every time that you do a deploy or just as part of the integration process, and then it may go through a deployment process, too, or how frequently would you see this running?
Vergara: Yeah, so it’s for every code change that gets checked in, right? Ideally, you would want that change to go all the way, so for continuous—again, for continuous delivery and deployment, that’s the ideal process. Check in, generate the artifacts, and deploy that artifact, create a baked image out of it, and then go and deploy it.
Ashley: Okay, great. So, this is—
Vergara: Right, so, but—yeah, yeah, right. But like, when we say delivery and deployment, of course, you know, there’s release management and change management policies at every company that says, “Okay, this is actually good for production deployment,” right? So, some of them are continuous deployment up to development or test environments, but for production, we still want to have that gate system in there.
Ashley: Mm-hmm, yeah.
Vergara: And Spinnaker actually supports it really well, you know? It’s got those manual judgment stages that you can add into the pipelines to get those kinds of approvals.
Ashley: Yeah. Well, we’ve had some other folks talking about some of those stages like Run Job and Webhook and some of the capabilities that are built in.
Vergara: Yes.
Ashley: Easy to extend as well in Spinnaker. So, I think one of the things that you’re planning on talking about in your talk at Spinnaker Summit is maybe some lessons learned or best practices that you’ve figured out from doing this a lot at Salesforce. Any thoughts on that you want to share with us about the kind of best practices you’re gonna get into in your talk?
Vergara: Sure, yeah. So, some of the best practices that I’d like to share with you is more around the baking and deployment. So, for example, when you do baking and never, ever bake secrets during bake time. [Laughter]
Ashley: Yeah. No tokens, no keys, no certificates, no—
Vergara: Exactly. Yes, yes. So, that’s very, very important. So, those kinds of sensitive data, you can actually perform that during the deployment stage. There is a trap stage there during deployment where you can go and, you know, connect to a secret store, like, Vault, HashiCorp’s Vault, right? So, you can pull down those secrets and install it on the resources or the VMs, right? So, that’s the safer way to go about it rather than baking it during bake time.
And then another thing as well, you know, as much as possible to avoid configuration drift, right? So, if you have any common or standard configuration, be it your application configuration or your OS configuration, right, do it during the baking step, right? So, as much as possible, limit those VM configurations towards them. Again, you can do it during deployment.
Ashley: Mm-hmm.
Vergara: And then, for companies that support multiple cloud providers—so, for example, you need to go and deploy to, say, Amazon Cloud or Google Cloud or other clouds that are available and supported by Spinnaker, you can use Cloud-init files so that it’s not tied to one particular provider and you don’t have to repeat your bootstrap steps during that same time. So, you write your script once, your Cloud-init file, and then you can use it in those different cloud providers, so that’s, I think, that’s very, very helpful.
Ashley: Now, is that capability something that’s built into Spinnaker if you write your script corrects, and I think you said init file for each—
Vergara: Cloud-init.
Ashley:—each cloud, Cloud-init.
Vergara: Yes.
Ashley: Is that something that, it’s a feature if you know about, you can use it up front and design that into how you’re writing your scripts?
Vergara: Yeah, so, there’s—there are fields within the Spinnaker deploy stage where you can put that in. So, in, I think in GCEVMs, you can use the user data field in the deploy stage, and then in AWS, there’s the metadata field.
So, the field kind of differs by provider, but it’s the same concept, right?
Ashley: Mm-hmm.
Vergara: So, you put it in there, you plug it in. And then, during the bootstrap process of those VMs, I think the OS has the support Cloud-init, though, so when they choose an OS, make sure it’s got the Cloud-init, and then Cloud-init will pick up those files and then launch the steps or commands from what you have in those Cloud-init scripts.
Ashley: Mm-hmm, great. What are some of the things that people that are maybe new to Spinnaker or new to creating VM images and doing baking, what are some of the typical struggles someone might have that would be helpful to come to your talk for?
Vergara: Right, so I experienced the same thing when I started out with Spinnaker. I’m like, “Oh, this is all new to me! How do I go about this?” [Laughter] So, one of the things that I’m gonna share in the Summit is, like, “Hey, you know what? Spinnaker actually uses Packer, it’s a HashiCorp tool that actually performs the baking behind the scenes.
Now, Spinnaker offers a set of Packer templates, right? So, those are the very basic ones. And it actually varies per provider. So, you can only use one at a time. So, if you’re building for, say, GCP, then use the GCP Packer template. And if you’re using AWS, you have to use a different AWS Packer template.
Ashley: Mm-hmm.
Vergara: So, and then, if you have a more advanced use case, you can actually customize and build your own Packer template, so that’s one of the powerful teachers of Spinnaker as well, right? So, if you end up—like, you know, so for in-house, you use Chef or Ansible or Puppet. There are Packer templates that can use those kinds of provisioners, but you have to go and deploy those into Spinnaker.
So, at the talk, I’m gonna share how you can deploy those custom Packer templates. So, there are a couple of ways on how to do it. So, one is the simpler way, but it’s kind of limited. Those Packer templates are stored like Kubernetes secrets in Spinnaker, so it’s got a limitation in size. So, if you have complex scripts or Packer templates, then you have to deploy it as a sidecar container.
Again, I’m gonna share those tips at the Summit and give more details around it.
Ashley: Mm-hmm. Yeah, folks may be familiar with Packer, too, it’s an open source project in and of itself for creating images, machine images.
Vergara: Yes.
Ashley: Great. Are there some other struggles that you find folks typically have, or may have when they’re new to this?
Vergara: What else? So, Spinnaker by itself mostly focuses on releasing software changes, but it has a very flexible pipeline management system. Like, at Salesforce, we actually use it for building resources such as load balancers, DNS records, topics, queues. You can actually create custom job stages to perform those.
Ashley: Mm-hmm.
Vergara: So, we found it really awesome, because we can rebuild a full environment carrying down all of those existing resources, running the same set of pipelines that we can deploy together with those infrastructure resource setup. So, we can build it up right away, right, and then it also answers our scaling requirements. Because, you know, large scale enterprises such as Salesforce, you know, we have to build all of these resources and deployments. We’re talking about thousands of servers across different regions and geographical locations.
Ashley: Mm-hmm. You know, sometimes tools, open source, or even commercial tools that are highly flexible and very powerful, sometimes with that comes complexity, so it helps to have some guidance, some experience mentoring from someone like yourself that’s gone through that process and can help you kinda get through it and understand how to use it, how to get things done, right?
Vergara: Right. So, Spinnaker has a Slack channel. So, you know, when people like me or others who are new to Spinnaker have any questions, we go and ask in that channel. There’s a few channels there that are specific to certain features. So, for example, if you’re struggling with templatizing pipelines, then you go into the Pipelines as Code channel, right? Or if you have questions about how to use Kubernetes in Spinnaker, then you use the Kubernetes Slack channel.
That was very, very helpful for us, especially in the beginning when we were trying to figure out how things are working.
Ashley: Mm-hmm, I’ll bet, I’ll bet. How long have you been using Spinnaker, then, at Salesforce?
Vergara: We started our journey last year.
Ashley: Okay. [Cross talk]
Vergara: Late last year, so almost one year now. [Laughter]
Ashley: Wow, okay.
Vergara: Yeah.
Ashley: Wow. I bet a lot has happened in that one year. I know one of the things that you’re planning on also talking about is some contribution ideas for those that might want to help make Spinnaker better. And those are things that you—ideas you had or lessons you’ve learned implementing Spinnaker there, I assume?
Vergara: Right. So, some of the contribution ideas that we have is with regards to a new effort called Kill. I’m not sure if you have interviewed the speakers.
Ashley: I haven’t yet, but I may. [Laughter]
Vergara: Okay. Yeah, so I’ll let them—you know, I won’t steal their thunder.
Ashley: Don’t steal their thunder—okay, great. [Laughter]
Vergara: [Laughter] Right, so, it’s one of those, they’re focused more around the managed delivery, so that’s when. Another contribution idea is centered more around each of the providers. So, again, if you’re building for one provider, it’s a different, totally different stage. So, for example, like GCEVM, right? So, they have their own deploy stage for that. And then AWS, they would have their own deploy stage for that. They’re kind of not, like, at par right now. Like, you know, they focus on certain things.
I think another contribution there is trying to add more options or features into that, so it can fully support that provider. And so, for example, for GCE, I think one of the features lacking there, if I remember correctly—or they might have fixed it. But it was, say, so if you have a persistent disk and you want this particular persistent disk, then it’s not supported. It always creates a new one. So, it doesn’t give you an option to actually reuse an existing one that you’ve built earlier. So, things like those. I think that would be important.
Same thing with AWS, right? So, there’s some features for EC2 VMs during deployment that could be added in there as well.
Ashley: Great. I’m wondering, too, thinking about your talking that’s coming up, who are the kinds of folks that would be, get the most out of it, coming to your—obviously, people that wanna perform this function, but is that usually, is it developers, is it DevOps engineers, is it CI/CD automation specialists? Who are folks that would get the most out of your talk?
Vergara: It would be around the Spinnaker administrators within their companies. So, if they want to learn how to configure Spinnaker to enable features for baking and deployment, then they would be the ones who would learn the most out of it. But I would say, like, all the DevOps engineers who want to figure out how they actually use Spinnaker for baking and deployment as well. Like, how do they set up their pipelines?
I’m gonna give a demo on how it actually works from start to finish so that they can have a visual. Sometimes, you need that visual to say, “Oh, yeah! Oh, that’s how it works!” Right?
Ashley: [Laughter] Mm-hmm. Well, that guidance is very helpful. And I can see, too, what you’re saying about the folks that are obviously administering Spinnaker in that deploy process are ideal to attend, and of course, other DevOps engineers automating and wanna know how this part of it works certainly would be good to go as well.
Well, I wish you the best of luck. I know you’ll do real well. You obviously know, have learned a lot and know a lot from your experience at Salesforce. So, I wish you the best.
Vergara: Well, thank you. Thank you, Mitch, again, for the opportunity for having me talk here.
Ashley: Well, absolutely. Thank you for being on. I wanna thank my guest today, Jing Vergara, who’s princ—I can’t say it [Laughter]—principal software engineer at Salesforce. There, I got it out. Now, Spinnaker Summit 2019 is November 15th through the 19th in San Diego, and Jing’s talk is on Sunday, November 17th, at 10:45. It’s titled, “Demystifying Spinnaker VM Imaging Baking and Deployment.”
So, thank you for joining us today for this DevOps Chat podcast. This is Mitch Ashley with DevOps.com. Have a great day, and be careful out there.
