Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk about the spectrum of continuous delivery and where continuous delivery will and will not work.
Edith is CEO and co-founder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.
Paul is founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.
Edith: So Paul, does one have to be all in to do continuous delivery?
Paul: So I think the obvious answer is no. And in particular, you know that the answer must be no because nobody starts at all-in, or at least if you’re starting a new project you can be all in, and many projects are not all-in.
So someone starts somewhere and they say, “Now we are going to do continuous delivery in some way,” and so they have to go from zero to full continuous delivery along some way.
Edith:
People have this conception that continuous delivery just means pushing any time you want, and a lot of very old school development shops did that. They would just hot patch and production constantly.
Paul: Right. So, I mean, are we calling that continuous delivery? I feel that the idea of SSH-ing into the server is not continuous delivery somehow.
Edith: But people have this misconception because they’re like, “Oh, I had a continuous delivery before this was a thing, I just patched it production.”
Paul: I’m going to put a lot of money on the line here and say that if you define continuous delivery as “I SSH into the production server,” then you are not doing continuous delivery right. In fact, you are doing it considerably wrong.
Edith: What exactly are you doing wrong then? What steps are you skipping?
Paul:
The whole point is that you have this automated process for releasing software. And SSH-into the production server is the very opposite of that.
That’s not to say you shouldn’t do it in the occasional emergency. And occasionally, if we have some sort of catastrophe, we will SSH into the production server, replace some code, execute some code on the server to keep things as they are and prevent customers from noticing or from things from going down.
But that isn’t the same as a continuous delivery pipeline or a release workflow, or an automated release workflow, which is essentially what continuous delivery is.
Edith: So continuous delivery isn’t just shipping stuff out to production, it’s more having all the processes behind it.
Paul: It’s very much the process. So when you think about can you be a little bit continuous delivery, I think what you’re looking at saying is there a lightweight version where we get some benefits of the continuous delivery as we go between all and nothing?
Edith: Can you be, do you have to be all in, or can you be a little bit continuous?
Paul:
I think the very obvious instance of a little bit continuous is where you deliver the code where you have the automated release process builds some sort of artifact and then someone manually deploys that by a single click.
Or, in fact, where a build happens and test happens and the remaining automated release process is triggered by a human saying, “Now we are going to release,” and then everything else is scripted, and so they just have one input, which is now.
Edith: Oh, that’s really interesting. So do you think to truly be continuous it has to all be completely automated with everything always pushed?
Paul: No, no. I don’t think so. I think it largely depends on the size of the team and that sort of thing. But I’ve seen a couple of examples where our customers were asking for particular things, and it indicated that they had sort of semi-continuous workflows.
And one example of that is a team that only wants to do continuous delivery during work hours.
Edith: Yeah.
Paul: And there was another team, which only wanted to do continuous delivery at midnight because that was the only time that they didn’t have enough customers on their site to be able to clear the caches.
Edith: Yeah.
Paul: Now if they wanted to go to complete continuous delivery they would have to say, “We are going to clear a subset of the caches,” or have some other caching policy that allows us to continue or that allows us to deploy at any time, but where they are is basically were semi-continuous.
Edith: I had a really interesting chat with Douglas Squirrel, who actually listened to our podcast, and he does a lot of consulting for companies in England, and he said he goes to a lot of companies that have all the tooling to be continuous.
Paul: Okay.
Edith: But they don’t actually use it.
Paul: And what does that mean exactly?
Edith: They have set-up continuous integration, they could, in theory, deploy whenever they wanted but they still, for whatever reason, still cling to their old schedules.
Paul: Okay. So, I was looking at this very interesting article on the First Round Capital blog.
Edith: Oh Jocelyn’s.
Paul: Yes, that sounds right.
Edith: Jocelyn Goldfein, who is at Facebook, right?
Paul: Yes, that’s right. Yes. So, what she talked about was this idea that delivery schedules are useful in certain contexts. And in some cases velocity matters, and that’s kind of the Facebook case, and in some cases predictability matters a lot more. And if you’re delivering enterprise software, for example.
An enterprise is not going to allow you to continuously deliver your product into their firewall or into their enterprise. What they’re going to want is every quarter a new thing.
Multiple times a year preferably, but each time you release to them it creates a new workflow, and they want to be able to audit it and see a change log and validate that things are, they might have a change process internally.
And so it’s not necessarily going to be a good idea to continuously deliver there. But a team that is building for that can do a continuous delivery process internally where they produce all of the artifacts and they do a complete release on every single push. Even if that is not released.
Edith:
There are a lot of reasons why an enterprise doesn’t want to take new features constantly, and a lot of it comes down to training. If there are big new features that need to be rolled out to people then they have to take on the cost of training.
Paul: Right.
Edith: It was actually interesting. I chatted with a guy from Square and he talked about how they really try to limit the amount of pushes they put out to the actual cashiers.
Paul: Okay.
Edith: Because of the training cost. I mean, a cashier at a coffee shop is doing a gazillion other thing and they can’t just suddenly pop up some new functionality and expect them to grok it or even take a tour.
Paul: Right. That’s an interesting idea.
Edith: So, although they have all the processes in place to do a lot of changes, to do continuous delivery, they actually really limit what they push out to the true end users.
Paul: And it’s funny that the people who have far, far more users, someone like Facebook, can just do those changes because they don’t necessarily value an individual user on a…
Edith: You’re walking down a path.
Paul: No, I’m not saying that they don’t value their users. It is safe to piss off an individual user. Less so for someone like Square. And as you go up the value chain, the more they’re paying you or the more involvement they have to have in the product decisions, the less able you are to piss them off.
Edith: Facebook is actually a fascinating example because at any time they’re going to be pissing off somebody. It’s just a matter of, it’s kind of like Apple.
Ya know?
Paul: Right.
Edith: They’re always pissing off somebody, it’s just who.
Paul: Did you hear about today at Facebook where all the feature flags went on?
Edith: It was LinkedIn.
Paul: Oh, was it LinkedIn? Okay.
Edith: Ya. Well, it probably has happened to everybody that’s done feature flags. There’s the day that that happens.
Paul: Right.
Edith: And, it’s ugly, yet. Facebook is fascinating. I actually, I wrote a blog post about Facebook and how they use feature flags. They have a product called Gatekeeper.
Paul: Right.
Edith: That controls everything, everything. And after they had a huge PR disaster, i think it was in 2011, with their news feed, they actually do really care if they piss off a lot of people.
Paul: Right, right.
Edith: And that’s why they implemented Gatekeeper where they do a lot of 1% roll-outs.
Paul: So, I remember when I was setting up Circle I sat down and talked to my friend who worked at Facebook and talked about their workflow and that sort of thing, and their release process was that code got released on Tuesdays and everyone who was releasing code, or who had code in that bit that was going out, had to be available during the release process.
And the release process didn’t turn on any features because that was all done by Gatekeeper, but the code itself had to go out. And they had to validate that everything stayed the same and everyone had to be there. And so they had a fairly continuous delivery process but were still, manually there was a release going out.
Edith: Well ya. You want to have some sort of enforcement that people look at stuff and care.
Paul: Care in what sense?
Edith: So, the example I heard about also at Facebook is it’s so large that there’s just no way that all the groups can coordinate with each other. So literally what they do is when they roll stuff out with Gatekeeper they roll it out internally to Facebook-only employees and that’s how they find out if stuff breaks. It’s not like at a smaller company like CircleCI.
Paul: Right. But there’s a difference between the code being rolled out and the feature being rolled out.
And they don’t do, I don’t know what the story is nowadays, but at the time they certainly didn’t do continuous delivery on code. Or continuous releases or whatever you might call it.
Edith: I don’t actually work at Facebook, but what I did hear was that– that they just literally can’t coordinate. When you’re a small company you can still coordinate in a HipChat or a Slack room like, “Hey, we’re pushing this now, start looking for changes.”
Paul: Right.
Edith:
But with Facebook, the only way they know if something is broken is if somebody complains.
Paul: So there’s an interesting discussion about microservices at AWS. I guess they didn’t call them microservices at the time, but having fully service oriented architecture.
So, one of the things they talked about is how when you roll, or when you continuously deploy a service, you might break other services. So, the thing that actually breaks may not be in any way related to the people who’s alarms start going off. And, in fact, there may be multiple steps away from who’s alarms are going off.
Edith: Google has a similar system, and I heard that they monitor 200 plus different metrics. And one of the ones they monitor is just do people say, “Gmail sucks more?” on Twitter.
Paul: Oh. Interesting.
The thing that I’m getting at in terms of how little or much can you continuously deliver is, if you are Google or you’re Facebook and you have the monolithic code repos, the thing that you’re releasing is it’s not the same release process as someone like Amazon who has what I would describe as pure, continuous delivery.
Every service does it’s own continuous delivery and your dependencies may be changing at any time and you don’t actually know what’s going on. Versus Facebook, we are taking this multi-gigabyte binary and we’re uploading it to a set of servers, and sure, we’re slowly releasing and maybe we’re doing that multiple times a week or multiple times a day.
But it’s not quite the same as Amazon releasing every 11 seconds. And I think we can see, even between these multiple large companies who have incredibly advanced release process, that you could say that they are clearly at different spectrums along continuous delivery.
Edith: So, what do you think is suitable for, let’s say, a medium company with, let’s say, 10 developers?
Paul: I mean, there’s a lot of different things going in. Who is your customer base? What is the level of safety that you need? Let’s suppose that it’s a consumer web app. If it’s a consumer web app, I would expect that that company should be continuously delivering all the time. Especially if it’s slightly okay to break things.
Edith: Yeah.
Paul: Or at least, you break things you can kind of roll them back or whatever within a couple of minutes. A team of that size might not have the resources to make their continuous delivery process perfect. But it seems likely that they would have more success shipping with continuous delivery than shipping without it. So lower risks, higher velocity. That sort of thing.
Edith: It’s funny how continuous delivery has permeated even the VC world. Jason Lemkin, the guy behind SaaStr,
Paul: Right.
Edith: He tweeted yesterday that one of his due diligence questions was how,”How often can you roll back, and does it work?”
Paul: Right.
Edith: And it’s funny that I think of that as a very technical thing, like, “can you roll back?” But to the VC is seen as a sign of technical sophistication.
For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.
