Podcast: How DevOps Combines Classic Lean Manufacturing Techniques with Modern Agile IT Operations to Deliver Significant Competitive Advantage

Devops leads to significant competitive advantage.
Devops leads to significant competitive advantage.

If your industry is being disrupted, chances are a strong new competitor is utilizing DevOps to reduce the time required to get from an idea for software development to secure, effective implementation.   Cris Daniluk of Rhythmic Technologies provides insight and background on the DevOps movement, comparing the impact it can have to Lean Manufacturing, which revolutionized how many businesses operate by reducing waste and optimizing production.

In this informative conversation, Cris shares:

  • How a few large social media and search companies initiated the DevOps movement, and the ways in which it is now transforming organizations in all sectors by drastically reducing the time from idea to implementation.
  • Why the elimination of large-scale, periodic updates of software and websites is so beneficial, and how a DevOps mindset enables development and operations to work together to make changes seamlessly and efficiently.
  • The actual, tangible results that businesses and even large government agencies are enjoying with DevOps as they radically improve speed of implementation, reliability and security.
  • The relative ease with which an organization can implement DevOps and the first step they should take.
  • How CEOs, CFOs, and COOs can get the conversation started with technologists in their organization about the adoption, or more rapid implementation, of DevOps principals.

DevOps is radically changing many industries, and allowing smaller competitors to quickly catch up to, and in some cases, dominate established industry leaders. This conversation will help you more fully understand the benefits of DevOps and how DevOps principles can be implemented in your organization.

You can listen to the conversation with Cris in the player below, and/or you can read the full transcript beneath the player.




Kevin O’Neill, Data Center Spotlight: This is Kevin O’Neill of Data Center Spotlight, and today we’re talking about DevOps with Cris Daniluk at Rhythmic Technologies. I’ve had the pleasure of working with Rhythmic on a number of different things and Cris is one of the brighter technologists you’re going to across when it comes to using technology, especially using technology to solve business problems and add value. Cris, thanks for joining me today.

Cris Daniluk, Rhythmic Technologies: Thanks for having me, Kevin.

Data Center Spotlight: Cris, we’re going to share with our audience today what DevOps is and the results that companies are achieving by adopting DevOps, but first, why don’t you share with our audience a little bit about Rhythmic Technologies and what you guys do there?

Cris Daniluk: Sure. We’re an infrastructure management company and we work with, usually, development and engineering teams to build and run platforms that they can run their apps on. DevOps is a huge part of what we do to give our customers, as particularly the Developers’ Tools ,they need to deploy fast, build fast, release fast, and know that their application is going to be well taken care of in the cloud.

Data Center Spotlight: Okay, great. Now Cris, DevOps is a term that, really, barely existed five years ago, and it’s been the domain of technologists since then. But now it seems to be bubbling up in the business discussions and business conversations. Can you define DevOps for us, please?

Cris Daniluk: Sure. If you ask ten different people, you’ll get ten different answers to this question, which means that it’s both impossible to answer right, and impossible to answer wrong. To me, DevOps is a culture of different teams in a company working together to move fast, and a set of tools and processes to support moving fast without making quality tail off. In fact, the result, when it’s done correctly, is that by moving faster and releasing more frequently you increase quality rather than decreasing quality, so you get to deliver value more often and you get to deliver a higher quality output than you would otherwise.

Data Center Spotlight: Cris, do you think we will decide on a definition for DevOps that’s universally accepted or we’ll just muddle along like with cloud for years and years with no formal definition forever.

Cris Daniluk: I hope that we don’t settle on one definition for it, because if we do I’m sure it’ll be the wrong one. I think what’s important is the goal of knowing that teams need to work together and knowing that you can’t go from where Dev does their thing for six months, and then Ops gets it dropped on their lap. We’re already behind schedule and then security come last in line. It just doesn’t make sense. As long as everybody’s agreeing on that common principle, I don’t think the precise definition is important.

Data Center Spotlight: And I think it’s important to note, as we get started here, Cris, that DevOps, it’s not like some new fancy application of technology, it’s closer to a process movement, like Lean or Agile. Correct?

Cris Daniluk: Absolutely. The core of DevOps is to say that delivering giant bunches of functionality delays value to the customer unnecessarily, and decreases quality because 1) you’re making a ton of changes at once and that’s just always risky, and 2) what are the odds that you’ve precisely identified what you needed to build nine months ago when you first sat down at the design table? DevOps changes that, and it says, “Let’s work in small batches. Let’s release frequently, and let’s be flexible to change because we know that there’s going to need to be change.” And that is the embodiment of Lean manufacturing, saying that we need to have more opportunities to inject communication in some of the feedback groups back to the people in front of us and behind us, and that we need to have more opportunities to adapt to unexpected changes.

Data Center Spotlight: Before we get into the nuts and bolts of how companies adopt DevOps principles, can you share just a little bit about what’s being achieved with DevOps, what sort of results?

Cris Daniluk: Sure. At the end of the day, DevOps only matters if it produces something. What DevOps produces, most importantly to me, is an accountability between teams, an involvement between them, so that when development shows up and they say, “I’m two months behind schedule, but here’s my code. You need to figure out how to get it working, and by the way, it needs to be working tomorrow.” A typical response from Operations is to say, “That’s nice. I don’t care.”

One of the favorite lines in Ops is, “You’re failure to plan does not constitute my emergency.” While that is somewhat true, it also doesn’t really help the business one bit. What DevOps does is it gets everybody working from the beginning of the project, or the release, or whatever it is on day one, so that there’s shared understanding, there’s shared ownership, there are no surprises, and if dev is behind for legitimate reasons, operations is not only aware of it, but they’re supportive of it, and they’re doing what they can to help get dev back on schedule.

One of the biggest things to get dev behind schedule is not having the environment they need to run their code in, and by pushing Operations up to the beginning alongside them, ops can then say, “Oh, I understand why you need that, and I understand that I’m hot helping myself by not getting that to you as fast as possible.” Now, as a result, there certainly have been a lot of creative tools and processes that have sprung up to help support that, but those are really a means to an end.

Data Center Spotlight: It seems, Cris, that everyone talks about their infrastructure wanting to help them move faster. Everyone wants to move faster, and it seems like DevOps goes a long way towards helping companies just move faster. Move faster with new versions of whatever they’re offering their customers, move faster with updates to their websites, just a lot of speed.

Cris Daniluk: Absolutely. There are already startups that come out on the market and they’re going up against well-entrenched competitors and they don’t have the baggage of this behemoth of a system that they have to manage, and so they can just push out the functionality at a rapid clip. In Enterprise, it’s sitting there thinking, “I wish I could move that fast.”

DevOps isn’t something that is owned by startups full of hipsters. Anyone can do DevOps and there’s great examples of established companies making that pivot to do it. Google wasn’t doing DevOps and started having problems, and now they are. HP, Etsy, Amazon – all of these enormous companies that have huge infrastructures and started dying under their own weight, adopted DevOps techniques and principles. Slowly, at first – you can’t just do it all like a light switch, but they started gradually building these in, and as a result, they’re able to keep pace with the startups that are trying to challenge them on the market and in the case of Google and Amazon, they’re just smokin’ ’em.

Data Center Spotlight: You know, Cris, as we all know, traditional software and websites have sort of converged greatly in the past decade or so, and the result has been complex apps that customers expect to update just as often as, say, a news site would update a breaking news article or a blog would give updates. I think this is a problem, and most development teams have been trained to work in a large release cycles, sometimes as long as a year. And even Agile teams that release monthly can’t necessarily keep pace with customer demand. How does DevOps solve some of these problems?

Cris Daniluk: One of the key attributes of a well-run DevOps infrastructure is that your systems are loosely coupled together. You want to have teams tightly coupled, but you want to have systems loosely coupled. What that means is that each system is a small piece of your overall application and it makes a contract out with all the systems around it that says, “You don’t care how I work, but I will always deliver you this result in this way.”

What that gives you is the ability to make changes whenever you want to that system without having to fear that you’ve affected the systems around you. That’s incredibly powerful. Agile doesn’t necessarily give you that. In fact, Agile can make a tightly coupled system even harder to maintain because you’re moving so fast that you never really have the time to really assess your risk, but when you have that loose coupling it gives you a lot of power.

Amazon is a great example of this. Amazon has what’s called the two-pizza-box-rule for their teams, which means that there should be no team large enough that they can’t be fed by two pizzas. The corollary of that is that because each team owns the system, they really can’t own a system all that big, they can’t own a system any bigger than about 10 people can maintain.

Amazon obviously has thousands of systems, then, if they’re that small and yet they can present all of the functionality you see in Amazon.com and in AWS. So, clearly, the result of that is that they’re very loosely coupled and that they’re all able to move independently of each other in a safe and controlled way.

Data Center Spotlight: That’s great. I haven’t heard it explained just like that before. Cris, a little bit ago you referenced the security advantages that DevOps can provide. I know you’re very security conscious, and for me it’s been kind of a canary in a coal mine on a number of security issues before they become public knowledge, and covered in the tech media. Can you give me, maybe, a little more color on how DevOps provides a security advantage?

Cris Daniluk: First I want to thank you for making die on your podcast here. It doesn’t end well for the canary. I think that security is always last in line and, as a result, the InfoSec teams – they’re people, too – they don’t like to be short-changed. Their whole life is protecting the organization and they never really get the opportunity to do that. DevOps involves them sooner, which is critical because it gives an opportunity for them to show, not just, “This is what you have to do to comply,” but more importantly, “Why do you want to? What’s the risk?” It gives them an opportunity to be part of the process and to help say, “I understand why you’re pushing this off. This one, though, is really important. You shouldn’t do that.”

The other benefit is that the small batch sizes that you’re working in in DevOps – these rapid release cycles – means that 1) you can make changes often. So if you can’t get a change in in your Wednesday release for security, maybe you can get it in your Friday release or your next week release, whatever your schedule is, whatever your cadence is. It gives you more opportunities to meet their needs. It also has, because we have these loosely coupled systems, there’s not the fear of upgrade that so often keeps people from patching their systems.

Verizon does a report every year on top causes of breaches that lose customer credit card data, and some of them, like 95% of all of those breaches, are accounted for by 10 vulnerabilities, most of which are over 6 months old, and organizations just haven’t addressed them yet. We know they’re critical. We know that they can be used to steal customer data, and yet the organization doesn’t make the time. There’s only one explanation for that which is the fear that if they fix it, they’ll break something else that’s even worse.

DevOps gets you to a place where each service can make their own security improvements and know they’re not going to affect the services around them. That makes it a whole lot easier to say, “Yes, I’m willing to take the risk and make that security update for you.”

Data Center Spotlight: Cris, you referenced some of the social media web-scale behemoths that were at the forefront of the DevOps mindset, the DevOps movement, if you will. I know you’ve got a lot of relationships at some of those companies. Can you give us some idea as to what sort of results they’ve had, what sort of tangible results they’ve had by transitioning over to a DevOps mindset and a DevOps-type process?

Cris Daniluk: Absolutely. A lot of those companies – you mentioned at the beginning of the call that DevOps was barely around and barely talked about five years – it wasn’t those startups that are so effectively using DevOps today that came up with it, it was actually those web-scale companies like Google and Amazon that developed those techniques in response to… They grew so fast that they grew the problems that a 30-year-old enterprise would have in the span of a few years, and they didn’t want to slow down. So it forced them to think about what they needed to do and how they could do it.

I don’t think Google gets enough credit for their role in fusing that DevOps mindset – bringing teams together, pushing across cross-functional barriers, and eliminating that chain of Dev to Ops to Security that’s just no good for anyone.

Etsy, I think, is one of the most tangible examples of a company that nearly died as a result of becoming paralyzed by the complexity they created in their systems. In a very short period of time, in a matter of months after adopting DevOps principles they significantly improved their situation and stopped the bleeding and within a year they were a role model example for anyone to follow.

Data Center Spotlight: What did they do? How did they overcome those problems using DevOps?

Cris Danulik: The biggest thing they did was to recognize just how bad their productivity was and to force themselves to stop trying and pushing against that wall – take a break, clean up some of the recurring problems, some of their technical debt – and then start decoupling some of their systems.

One of the things that they did was created this unnecessary dependency across a bunch of different teams in order to make just about any piece of functionality happen. One of the things they did was to break that dependency and say that as convenient and as brilliant as it seemed, to make sure that your best database finds were involved in every database change. The result was that they became the hot commodity in the entire organization, and there just weren’t enough of them. They broke that and they said, “You know what? We’re willing to say that we might not always get the best SQL every time for every piece of code, but that’s okay, because we don’t have this dependency that was paralyzing us.”

Data Center Spotlight: Now let’s move down from the largest social media companies, down into what, for most people listening to this, is the real world. And I know you can’t share a whole lot about your customers for reasons of privacy and security, but can you give us an example of a case study that you at Rhythmic have been involved with, of a company that utilized DevOps to improve their operations?

Cris Danulik: Absolutely. One of our customers that I think just fits the promise of DevOps so well was owned by a large dot com that went on a large acquiring binge over the course of several years and folded up a bunch of really great talented developers, married them to their behemoth Enterprise IT team and expected them to be able to work in that. They sold out that company and all those developers, and kept their very traditional Enterprise IT team.

We had a customer that knew their ping points so well over the years – they knew that they had these arbitrary limitations that were preventing them from deploying as often as they wanted, from being able to see exactly how their systems ran in production, and being able to release in a timely fashion. Deployments were an affair for them, an unnecessarily so because they had done everything on their end to make deployments be a breeze.

We came in and took the role of that enterprise IT team and completely changed it into a… The developers are our customer, as opposed to how the old mindset, which was that Operations was in charge, and they were the ones to be pleased. We felt and said that we needed to keep them by giving them the tools that they need to do fast. The results were extraordinary. They were able to, really, in a short time, not only establish themselves in the market that they used to lead, but catch right back up and overtake the market in a span of a few years. The result was that they went from a company that was sort of dwindling into irrelevance into a very large acquisition in a period of less than five years.

Data Center Spotlight: Cris, hearing you tell that story, you and I have spoken quite a bit the past couple of years about how the developer mindset and the operations mindset are two different things – sort of a left brain, right brain – creative versus, “Hey, let’s make this work,” sort of thing. It seems like DevOps might go a long way towards bridging those two mindsets, and helping those two very different qualities that those people have, who are good at those jobs, helped them have a meeting of the minds and work better together.

Cris Danulik: Yeah, that’s why it’s so critical to be working in an integrated fashion and to be working from the outset together, because a lot of times developers will have creative ideas that are as brilliant as they are flawed, and completely terrible to move forward with in production. But if you start up front, knowing that brilliant idea, a lot of times operations can work back and forth, it’s a collaborative effort. It’s not Operations say, “Here’s how you make your idea not stupid,” but it’s a back and forth process. You can come up with something that captures that original great idea, but in a way that you can run it in production effectively.

I think if you look at what companies that are doing DevOps are doing in production today, the way that they can visualize into their application and see detailed metrics and monitor every single little facet of the application…it wasn’t Ops that came up with those ideas and figured out how to do that, it was developers. That mindset is saying that developers deliver us crap and we have to figure out how to run it, doesn’t add value to the business. We need to encourage that creativity, and let them give us better ways to run their apps. The only way to do that is with that collaboration that happens on day one of every project.

Data Center Spotlight: Cris, I saw a case study recently, which I found to be somewhat surprising, in that it involved a large government agency succeeding with DevOps and, of course, my assumption is that a large lumbering government agency would have a hard time succeeding with something that is more, I think, along the realms of an organization that’s small and nimble.

Rhythmic, I know you work with startups, you work with enterprises, you work with government agencies, you work with all sorts of companies. I know all organizations are different, but which of those categories of company or organization does DevOps show the most promise for?

Cris Daniluk: I don’t’ think it necessarily boils down to what type of organization you have. I think any organization can benefit from it. We talked earlier about remanufacturing, and it’s very difficult to find a manufacturing organization or process that can’t be improved through Lean techniques. It’s the same for DevOps. If you avoid that rigid definition and you keep just the core principles that we’re going to build loosely couples systems, we’re going to have tightly coupled teams, and we’re going to release often as we can, so the amount of changes that we’re making is small, that can be used anywhere. That can be used at an enterprise at any size. There’s no reason that you need to feel like you’re too big or too bureaucratic to do that. The key is to be flexible so that you can identify your actual organizational barriers and work around them, so that you can still apply a DevOps mindset.

One of the things that people think about often is, “Well, how can I just be delivering all of my functionality to people piecemeal?” A lot of government agencies have a big problem with that. They need to be able to – also for statutory reasons – communicate out to the public and to their employees, and to their union members, a very well-defined training program to learn about new functionality. They can’t have it show up every couple of weeks, but DevOps doesn’t require that you make the functionality visible to the end user every time you release, it just requires that you’re releasing incrementally, and you might hide through those changes are actually visible to.

There’s a hundred different problems that you can make up that all sound a lot like that one, and there are at least as many answers, if you’re willing to be creative, and you’re willing to adapt the philosophy of DevOps to the specific environment you’re in.

Data Center Spotlight: Let’s shift gears just a little bit and talk about people who are responsible for the management of a technology team, yet they’re not technologists themselves. You’ve got CEOs, CFOs, COOs, all taking on more responsibility for the performance and budgets of their technology teams. What would be the best way for a business executive to address DevOps and the potential it can bring with this technology team? How can that business executive, how can he or she get that conversation started?

Cris Daniluk: I think tying it back to Lean is critical for somebody that’s not familiar with DevOps, because Lean manufacturing is a proven time and time again technique to improve the productivity of a manufacturing plant. That’s important because our business leaders of today have been, fortunately, able to learn that productivity is what matters and efficiency is quite often a distraction from what true productivity really means.

DevOps is all about productivity, and it sacrifices efficiency to get there. It sacrifices efficiency by integrating teams early, by making systems that take longer to develop, but that are easier to maintain. There’s this constant trade-off there in almost every facet of it, of throwing efficiency out the window and saying, “That’s not what matters. What matters is that we have something to show for it.” Any business executive can relate to that and can champion that idea out to a company.

Data Center Spotlight: How do you, at Rhythmic, help organizations implement and run DevOps? How do you help your customers to do that?

Cris Daniluk: The starting point is always putting ourselves in the middle of the process and being engaged on day one with a development team, with any existing operations and security folks, and making sure that we’re seeing as close to the problem as possible. When you come into a company that doesn’t do DevOps today at all, usually the first thing you need to do before you can make any changes is implement amazing monitoring, because you need the confidence that once you’ve changed something you know whether or not you broke something. Failing fast is okay if you fix it fast, and monitoring is essential to that, so that you know a problem before a customer knows a problem.

So we’re typically starting either from scratch, or just by building on top of existing monitoring, and making that be capable of fully visualizing every layer of the infrastructure, from underlying routers, and switches, and servers, all the way to the responsiveness of the application when somebody clicks on a particular button. You have to have that before you start making big changes.

Once we do that, we start looking for little hanging fruit where we can start taking away barriers that keep them from moving as fast as they want. That, typically, revolves around automating key processes. Deployment automation is one of the things that we’re often doing first in a customer, so that the rote process of packaging and orchestrating all of the commands that you have to run across a wide number of servers in a very controlled way to avoid impacting your users is just asking for mistakes to be made, and it’s asking for people to not want to do it, where the automating it onto a button that you push is inviting people to want to push that button and make that button be more awesome every time they do a deployment.

Those are the sort of things that we look for early on, and then we try and build over the course of time the gradual maturity where we make incremental gains as we go. I’ve never seen a completely DevOps, I’ve never seen that DevOps reaches potential, I’ve never seen it be done or complete. It’s a mindset of continuing to improve after you hit those initial early wins.

Data Center Spotlight: Cris, if someone listening wants to avail themselves of your DevOps experience and your infrastructure expertise, what’s the best way for them to get in touch with you and Rhythmic Technologies?

Cris Daniluk: I’d invite anyone that wants to learn more to check our website, rhythmictech.com, that’s R-H-Y-T-H-M-I-C. We’ve got a lot of great white papers and case studies talking about DevOps and some of the other challenges that we’ve helped our customers with. You can also find me on LinkedIn. I am the real Cris Daniluk in northern Virginia.

Data Center spotlight: And you’re the H-less Cris Daniluk, right? C-R-I-S, that’s your first name?

Cris Daniluk: Correct. My mother had to be different.

Data Center Spotlight: Cris, I appreciate your time today. This is a great half hour and, hopefully, we’ll be talking about this again in the future.

Cris Daniluk: I look forward to that. Hopefully in our next podcast you don’t try and make me into a dead bird.

Data Center Spotlight: (laughter) Thanks, Cris.

Cris Daniluk: Take care, Kevin.



Image courtesy of Wikimedia under Creative Commons license.