SauceCon 2020 by the good folks at Sauce Labs is just about a month and half away (April 27). As part of our lead-up to the show and our coverage including live broadcasting on Digital Anarchist, we caught up with newly minted Chief Product Officer, Matt Wyman.
Matt discusses how continuous testing, automation and DevOps has led to better testing. With that though, we are more complex and sophisticated than ever in how, what and when we test. Can’t we just keep it simple? Do we want to keep it simple? Have a listen to what Matt has to say, as well as why he is so excited about this coming event.
If you are interested in testing at all, do check out the great lineup for SauceCon. Hope to see you there.
Transcript
Alan Shimel: Hey, everyone, itโs Alan Shimel for DevOps.com, and you’re listening to another DevOps Chat. Todayโs DevOps Chatย is meant to give you a little bit of a flavor, a preview for the upcoming SauceCon put on by Sauce Labs, which will be at the end of April. So, about a month and a half away, almost two monthsโwell, a little less than two months away. It promises to be another great event from the folks at Sauce, and they have, really, just a packed lineup of speakers from a wide range of enterprises and startups and tech companies, manufacturing companies. I mean, just a whole gamut.
Anyway, speaking with me today about SauceCon, about maybe some new developments and trends in the testing market is Matt Wyman. Matt is chief product officer at Sauce. And Matt, welcome.
Matt Wyman: Thank you. Glad to be here.
Shimel: My pleasure. So, Matt, letโs get this out of the way first. You recently joined Sauce in, like, January, correct?
Wyman: I did, January 6th. I am a recent addition to Sauce Labs, and very excited to be here. I’ve been in the DevOps community for some time, worked as the head of product for CircleCI, so definitely a space that I know well.
Shimel: Sure
Wyman: But I see this massive opportunity and impact around test, and this fragmented space, and hopefully, we’ll get into this and talk about complexity a little bit. But it just really seems like a huge opportunity and a company that I love and a great reputation, so I’m excited to be here.
Shimel: Very cool. You know whatโs interesting? I mean, of course, CircleCI is in and of itself a great company, too. We’veโyou know, I’ve done a couple podcasts with Rob, the CTO over there.
Wyman: Yep.
Shimel: And, you know, they recently, they raised a bunch of money, they’re really becoming a player in the CI/CD space. But itโs interesting that you moved from CI and CD to testing here, and you know, when we look at, letโs say, CloudBees, right, and Jenkins, the Jenkins founder, KK, you knowโI’m not even try to pronounce his name; letโs just call him KK. But anyway, KK recently left CloudBees and the Jenkins community to start a new company also focused in the testing space.
So, I think your move here, KKโs moveโI think it shows that, as the whole CI/CD, you know, software delivery pipeline type of industry and community matures, people are recognizing that testing is kinda where itโs at right now, right? Thatโs the new battleground. And maybe it is, as you mentioned, Matt, because you know, the mission has grown so much more complex than, you know, letโs say pre-DevOps, right?
Wyman: Absolutely.
Shimel: So, why donโt youโif you donโt mind, give me or give our audience an idea of where you see or how you see testing has become, you know, more complex.
Wyman: Absolutely. So, speaking to the CI/CD platforms just for a moment, orchestration in CI/CD has become a very complex problem, and thatโs reflective of changes in the software landscape and how people build software, which itself is a reaction to where we started.
So, we had all these monoliths, these, you know, large systems, we’ve all talked about it, they still exist out there in the world. Most every system that says that they do microservices actually has a monolith somewhere, like, micro-monoliths all around.
When you’re in the monolith world, testing is all about end-to-end testing. You’re trying to put all the pieces together, validate that everything works top to bottom, and thatโs really where quality and test have their roots. And as much as possible, itโs all about trying to automate those difficult processes and make it easier and more efficient to move through all your different test cases.
But as those entities split up, and as we started to recognize that each component of a larger application needed different types of tests, testing itself became more complicated. And furthermore, it became more than just clicking buttons. At one point, testing was about validation, that if you click a button, you go to the correct page. If you click a button, the right business logic executed.
But these days, itโs a lot more than that. Itโs also validating that the CSS is correct, that the experience is performant and is wonderful and delightful. Companies that have bad user interfaces and bad user experiences donโt win.
And so, you have this testing world that, on the one hand, is trying to move faster, dealing with a more complex software landscape, and on the other hand, is now broadening in the types of tests that are being run, not just functional testing, but also, if you will, delight testing and trying to validate that the user experience, the customer experience is going to be an excellent experience at the end of the day
Shimel: Yeah.
Wyman: And so, thatโs how I see this space splitting up.
Shimel: Absolutely. I donโt disagree with you there. You know, Matt, I almost wanna put myself in the shoes of the test professional and the QA person who, you know, maybe six or seven years ago was hearing that DevOps and this move, you know, agile to DevOps and beyond was gonna somehow make them obsolete. And then other people were saying, โOh, no, itโs not gonna make you obsolete, itโs gonna free you up to do higher level kind of tasks that are more worthy of your talent and skill set.โ
And I think that has come true, certainly. I think what we’ve seen with test automation has freed up testers to do higher level stuff, but I also think it shifted theโnot the burden, but the emphasis on people who know how to set up automated testing, right? You’ve gotta know maybe a little bit of coding, a little bit of scripting, you gotta know some of these open source tests.
Wyman: Thereโs a bit of all of the above, donโt you think?
Shimel: Yeah, it is. Thatโs exactly it.
Wyman: Theโand itโs further complicated by the fact that the number of devices and environments we’re testing on has also fragmented and increased. And you canโt run the same type of test in every environment, so most software applications have some sort of mobile component to them.
Shimel: Yeah.
Wyman: And yet, mobile testing is not automated in the same way that you would automate browser testing. And in fact, in some cases with mobile testing, itโs more effective to be manual and to go through it piece by piece, or to use visual testing to go through it.
And so, you have to pull together all these different techniques. I think thereโs often this perception that, with any new technology or any new change, that itโs going to somehow upset the apple cart and suddenly everyone who has a job doesnโt. And it just doesnโt typically work out that way. I think what we’ve seen is just this increasing complexity, and then also the opportunity to use the right type of testing in the right place. Meanwhile, a lot of the people out there doing quality testing continue to do so, because the number of applications has only multiplied.
You know, the fact this whole digital transformation and all the buzz words that you hear about thatโfundamentally, itโs true. Every business is now a software business, and one of my favorite examples recently is, where I live, thereโs a highway called 101, a pretty big highway, and thereโs a big billboard on the side that has a, itโs from Wells Fargo, and it has a mobile app on it. And the billboard is not advocating or talking about Wells Fargoโs core business, they’re not talking about loans or lending or savings accounts, itโs talking about their mobile app and the experience of their mobile app. And you better believe that they care about testing it, right?
And so, itโs more than justโagain, I covered this earlier, itโs more than just feature function right now. You haveโyou need more people who need to look across all the different touch points and then utilize the right type of testing in each area. And so, there isnโt one test solution that solves all of it. You have to use different techniques, because ultimately, you’re managing business risk, and you have to do it all the way from the moment a developer checks in code all the way through to production and validate, if you will, a virtual user. Someone has to go validate that itโs actually working as expected when a customer reaches out to interact with the tool.
Shimel: Yeah, yeah. And, you know, Matt, you hit on a coupleโyou hit on web testing, mobile testing. I think one of the things, you know, thinking back to when I was running software companies, testing was always something that meant that we were a day late and a dollar short with, you know?
Wyman: Absolutely.
Shimel: It seemed like we never had enough budget to do all the testing we thought we should do, and we never had enough time to finish the testing at least we were going to do. But when I look at the kinds of testing and the amount of testing that goes in today, and all of it pre-deployment and everythingโyeah, itโs grown more complex, but I think just beyond pure complexity, I mean, would you say we’re testing more now than we did before.
Wyman: Well, I think part of the answer there is to dig in on what does it mean to test. So, these days, itโs about risk management. And so, if we’re going to have developers able to check in code and 10 minutes later update production, then we’ll need mechanisms to ensure the production is safe. And typically, thatโs done with feature flagging, itโs done with canaries. Thereโs all sorts of ways to ensure that each change doesnโt blow out production.
But along that path, itโs also about automating as many of the different possible negative outcomes as you can. And that is what we would typically call testing. So, thereโs visual testing, thereโs browser testing, thereโs unit testing, thereโs end to end testing, integration testing. All these are necessary, and you’re trying to compress them all into that 10 minute window, whichโyou know, most organizations canโt hit that kind of number, but thatโs the gold mark if you can hit that 10 minutes from change to production.
And so, I believe that itโs a matter of thinking about risk and saying itโs not about quote-unquote test coverage with a go/no go meeting where you look at the test coverageโand frankly, thereโs so much peer pressure to go that typically organizations do. Itโs more about saying, โWhat is the possible negative outcome, and how do I ensure that I have appropriate coverage for each of those negative possibilities?โ And in some cases, you’re going to build tests to simply block and fail that particular change, and in other cases, you’re going to provide monitoring and observability to ensure that, if something negative happens, you can pull it back quickly.
And if I go very meta for a moment and go well above even what Sauce Labs is doing as a platform, I would suggest that all of that is a part of the overall test or risk management perspective. And ultimately, I think our goal is to then create visibility across the whole thing, so you can manage your risk, manage your test all from one place.
Shimel: Got it. Makes sense. Now, ScottโMatt, excuse meโletโs talk a little bit about SauceCon. How is all of this manifesting itself, you think, at SauceCon?
Wyman: Well, one of the fun things for me is, this is gonna be my first SauceCon. So, I havenโt had the opportunity to seeโ
Shimel: Oh, you never went? Okay.
Wyman: I’ve never been, so, this is gonna be my first one. I mean, I’ve seen the lineup, it looks fantastic. I’m hoping to see not only a whole series of really good conversations within our existing community, but also, there are quite a few people that are coming who are Selenium, who are Appium users, developers and whatnot, to share in some of their experiences.
I know, as a product person, one of the conversations I’m very eager to have is in how we help people shift left. And so, itโs more than just running Selenium tests, itโs about the headless tests, itโs about getting earlier upstream so that we can provide a little bit more of a developer oriented experience. And so, I’m very excited to go have those conversations, and I think the lineup looks good to have it.
Shimel: I agree as well. So, I’ll tell you, my experience is, you know, if you’re looking for conferences around testing and howโlike, whatโs pushing the frontiers there, right, and also pulling in the open source testing community, SauceCon is one of the best. And I’ll tell you what I like about what Sauce Labs has done there is, you will see other testing companies represented, mentioned, participating, right? I remember, you know, CA who had bought BlazeMeter, for instance, was involved in SauceCon. I don’t know if BlazeMeter will be there this year or not, but they were there. I know the Tricentis folks also like to participate.
And part of the reason is because testing is such a big world with so many different facets today, you know, everyone kinda doesโyou know, they’re not necessarily head on competitors. A lot of this is sort of complementary. And so, itโs really nice to see the community come together. You see it a lotโand maybe this is something that is specific to DevOps in general, I don’t know, but you see it a lot at these DevOps events where, yeah, itโs Sauce Labs throwing the party, but the whole testing community partakes. And that, I think, to me is one of the things that makes SauceCon really special.
The other thing is, is just the speakers. I mean, you know, like in any vendor/user type conference, you’re gonna get the CEO, the CTO, and I’m sure Matt, you’ll do some presentation. But when you look at the speaker lineup here, you know, you have everything from, like, Google kind of engineers to startups to large enterprises, you’re really getting a good 360 degree view.
Wyman: What excites me is that the way in which we think about this event, we donโt see it as a sales event. We’re not sending salespeople there.
Shimel: Right.
Wyman: A number of my product folks will go, because we’re interested in having conversations. The test market overall is fragmented, but itโs fragmented for a good reason, which is, there are a lot of different use cases, a lot of different problems that people are trying to solve, and quite a few of those folks are going to be there, and so, itโs a good chance to have a conversation if you wanna do, if you wanna know more about visual testing, if you wanna know more about how to use headless effectively or deal with the grid.
So, thereโs this opportunity to share that information and share that knowledge, and I think thatโs really what we’re aiming for is an environment thatโs a practitionerโs environment. Itโs [Cross talk].
Shimel: Yeah, definitely, in years pastโin years past, you know, SauceCon, thatโs exactly what itโs been, and I think Sauce Labs has done a great job fostering that, right? Itโs good stuff, all around.
We donโt have a lot of time left, but you know, you just joined in January, you’re fairly new, your first SauceCon, you’re getting a handle around product. But do you feel up to giving us a little projection, maybe, of what we’ll see coming out from Sauce Labs over the next, you know, six months?
Wyman: Well, I donโt wanna give away too much, but we will have something that I expect to be quite exciting, a new beta labs experience that I think people will be able to go get their hands on and play with, something that moves, letโs call it shift left, a bit.
And we’ll continue to talk a bit about our analytics and in particular, thereโs a new report that we are going to be showing. I think some folks may be aware of it already, but itโs relatively new, itโs called Failure Analytics, where we’re able to aggregate all of the different log outputs and provide intelligence and cohorting of different common errors so that, instead of having to hunt through hundreds, potentially, of failures, we will highlight the common problems, which gives developers an easier time of going and fixing things. Also, it makes it a lot easier to debug across large automated test suites.
So, we’ve got quite a bit that we’ll be showing from a product standpoint, and like I said, there is gonna be a pretty exciting unveil around a beta project that we’re moving forward with.
Shimel: I think thatโs enough of a tease right there. Hey, Matt, you may not know this, but I’ll mention it for our audienceโanyone interested in going to SauceCon, itโs actually SauceCon, S-A-U-C-E-C-O-N dot com, and you can go on there and you can still register to attend. I should also mention that our MediaOps, DevOps.com team will be there under our Digital Anarchist platform. We will be broadcasting live interviews from SauceCon.
So, if anyone listening is going to be in Austin for the eventโhey, come find the Digital Anarchist broadcast booth or me, and weโd love to hear what you have to say and maybe interview you. You never know. So, yeah, please do that, but check out SauceCon.com for more information.
Matt, before we wrap up, anything else you wanna add?
Wyman: I think this is great. I appreciate the opportunity to have the conversation.
Shimel: Well, now that you’re part of Sauce, you know, weโd likeโwe do these DevOps Chats pretty regularly, so maybe we can have, maybe we’ll do another one even before SauceCon itself.
Iโd like toโyou know, I didn’t wanna put you on the spot today, [Laughter] but maybe in the future, Iโd like to hear, what are your top three sessions at SauceCon that you’re looking forward to. You donโt have to answer now, though, but I’m gonna give you a chance to do a little digging, but I’m gonna invite you back on in about two weeks and letโs talk about it.
Wyman: That sounds great. I think itโd be a good idea to, I assume that we’re at the end, here?
Shimel: Yeah, we’re wrapping up, and you know what, for our next one, Matt, I’m gonna pick three, too. So, folks, there you have itโstay tuned for our next chat with Matt where, between us, we’re gonna highlight his three and my threeโ
Wyman: That sounds like fun.
Shimel: Favorite talks for SauceCon that we’re looking forward to. Sound good?
Wyman: Alright, sounds great. I look forward to it.
Shimel: Alright! Hey, Matt, welcome aboard to Sauce Labs, I thinkโand you already know it, but you joined a great outfit, a great bunch of people there, and looking forward to seeing you in person at SauceCon.
Wyman: And likewiseโthank you.
Shimel: Okay, and we’re looking forward to seeing you, our listeners, there at SauceCon, but if you canโt make it, do log onto digitalanarchist.live and you can follow along with us from there.
Until then, this is Alan Shimel for DevOps.com, and you’ve just listened to another DevOps Chats. Have a great day, everyone.
โ Alan Shimel



