Podcast: An Amazon Web Services “Realist” on How to Leverage the Cloud Behemoth’s Strengths While Avoiding AWS’ Pitfalls

An Amazon Web Services data center in Oregon
An Amazon Web Services data center in Oregon

Founded as a way for the developer to take advantage of Amazon’s excess IT infrastructure capacity on an as-needed basis, Amazon Web Services (AWS) has pivoted toward the large enterprise user, with GE, Capital One, and others with massive infrastructure needs going nearly full bore with the cloud giant.

The decision of those enterprises to get out of the data center operations business and use the limitless scale and capacity of AWS is understandable. They have the budget to make the shift and the IT staffs to pull it off. It is a classic case of outsourcing to someone who can do it better and more inexpensively than you.

But for smaller users, without the proper staffing and budget, is going all-in with AWS advisable? That’s the topic of a conversation with Cris Daniluk of Rhythmic Technologies. Cris describes himself as an “AWS Realist” who both enthusiastically uses AWS for some applications while avoiding them entirely in other circumstances. His experience provides an interesting perspective that suggests that it isn’t always advisable to fully commit to a single provider.

This audio of this interview can be listened to in the player, and there is now a transcript below for those who prefer to read the interview.




Kevin O’Neill, Data Center Spotlight: Thanks for joining us, it’s Kevin O’Neill with Data Center Spotlight, and the topic we’re talking about is Amazon Web Services. Should Amazon Web Services be part of your service mix? We’re going to find out what the answer is from Cris Daniluk. Cris is a managed cloud expert, the founder and president of Rhythmic Technologies. Cris, thanks for joining us today.

Cris Daniluk, Rhythmic Technologies: Thanks for having me, Kevin.

Data Center Spotlight: Cris, why don’t you tell me a little bit about what you do, and what Rhythmic Technologies do, just to give people a little bit of background about the direction you’re coming from?

Cris Daniluk: Sure. Rhythmic Technologies provides managed cloud services. We actually run infrastructure, but we also provide what we call ops support for our clients. We work directly with development teams and help keep their products up and running either in an Amazon-like cloud, in our data center, in their data center. Wherever they are, wherever they need to be, we work to monitor and manage the platform, keep it secure, and keep the apps working as well as they can, and working with the development team to keep them focused on building the product, and take the load off for scalability, performance monitor, security, the sort of things that developers don’t necessarily love to spend a whole lot of time thinking about.

Data Center Spotlight: How long have you been in business, and how did you come to start Rhythmic Technologies?

Cris Daniluk: We’ve been around for almost nine years now. The business started rather unexpectedly, as an outcome of my honeymoon. We were in Jamaica, about halfway through our ferry, very rainy honeymoon, and I looked at my wife, and I said, I think I want to quit and start a business, and she said, okay, which was a little surprising to me. She says now that she was a little more scared than she admitted to at the time, but we started the company immediately after we got home, with about $3,000 in the bank, most of which bought a few servers, and we were off and running. A lot of the clients that we had in the early days are still our clients today, most of them in fact are still our clients today, and we’ve been really fortunate that as our clients have grown, we’ve grown with them, and we’ve added clients along the way, but we’ve been I think quite lucky to have an excellent set of clients that really are focused on brining interesting products to the market, and like I said earlier, not necessarily wanting to spend a whole lot of time dealing with ops, but they know it’s important.

Data Center Spotlight: Some conversations we’ve had that I thought would be interesting for these purposes, is that you provide a different service mix for just about all of your customers, from the startup to the enterprise, and everything in between, sort of depending what their needs are, and you’re frequently somewhat skeptical, somewhat critical of Amazon Web Services and what they offer, and most people do nothing but rave about Amazon Web Services, so I just thought some of your skepticism is kind of interesting, especially the fact that AWS is part of the mix for a good number of the managed cloud and hybrid solutions you provide to your customers. Is that correct, is that a correct sort of overview of being an AWS skeptic, yet using them a good bit, and could you explain why you do so?

Cris Daniluk: Sure, Amazon has built a pretty incredible platform, that the AWS platform encompasses from originally just providing what they called extra capacity that they had, just really to serve sort of a key ten weeks a year. They did a really good job of making that extra capacity available to people and then realizing this is a whole lot more valuable than they initially thought. They’ve rented that out with a number of complementary services, and you can really build apps that cater to each of those services, and do pretty impressive things without needing to necessarily go out and buy what used to be dedicated equipment to accomplish the same thing. The problem is that to really take advantage of that, you have to completely align your architecture to AWS, and a lot of people are considering whether they should move their existing application into AWS. That’s not necessarily the best course of action. Sometimes it is, sometimes it isn’t, but it’s a very complicated question, and certainly not inevitable. It’s certainly not obvious, and it’s certainly not a guarantee that every application is going to end up in AWS in five years, which is sort of what their marketing department has done a really good job of making a lot of people think.

Clients have to look at whether or not they want to spend time building their app differently, whether they want to reinvest in architecting it to run optimally in AWS, and I also have to look at whether or not it is necessarily cost-effective to do so. AWS serves some problems very well, and other problems not so much. For example, running an enterprise database in AWS can be incredibly expensive, compared to investing in physical hardware and running it in a traditional data center, and that’s partly where hybrid solutions become so valuable, because you can pick and choose the best parts of each solution, and each model, bring them together, and get exactly the right blend that works for your particular application.

Data Center Spotlight: Before we get too deep into the skepticism, I’ve also heard you say some pretty complimentary things about AWS. Who is AWS most useful for?

Cris Daniluk: I think AWS can be useful for pretty much anyone, it really depends on the problem that you’re trying to solve. A big problem is that people think that they have to pick AWS or not AWS, as opposed to just using it where it makes sense. I think that a lot of people that are building newer ground up web applications in the last several years, those tend to lend themselves very well to AWS, and that’s not a coincidence. Amazon is building its own web platform that way, and naturally, the AWS toolset portfolio is tailored first around them, as customer #1, but then their other critical customers are doing similar things. Netflix is putting out a lot of the open source technology that a lot of smaller startups are adopting themselves. Netflix is entirely and exclusively based in AWS, so it’s sort of natural that the technologies that they put out also tend to work pretty well in an AWS setting.

Data Center Spotlight: Okay, you also hear about, in recent months you’ve heard about GE sort of making a much larger commitment to AWS, CapitalOne, some other companies like that. What is so attractive about AWS to some of those companies who obviously are not necessarily starting from the ground up?

Cris Daniluk: A company that large, they can afford to have certain applications, certain components of their system not necessarily be cost-effective and still be okay when you look across an entire landscape. GE has something like 75 data centers nationwide. They’re going to consolidate from that down to four, they’re going to keep four, their most confidential information, and everything else they’re moving to AWS, and at that scale, whether or not an individual application is best served in a data center, or in a cloud environment, it gets lost in the weeds a little bit. I think part of the reason why it’s so attractive for a company like that in general, it’s because it gets them out of the data center business. They’re not colocation by and large, they’re running their own data center, so they’re in the physical security business. They’re in the cooling business, they’re in the power delivery business, and they’re having to support a large number of moving parts inside their organization, but they can simplify, and really just focus on writing code that helps solve business problems, either internal corporate problems, or problems with their customers.

So it’s easy to see the appeal to a large organization like that, but it’s also worth pointing out that more organizations that size are not moving, than are moving.

Data Center Spotlight: A lot more, it seems.

Cris Daniluk: Most organizations are thinking about and moving little pieces here and there, and a lot of the Fortune 500 is doing that, not necessarily going whole hog like GE.

Data Center Spotlight: Would it make sense, this is sort of a general question, would it make sense to keep your legacy, keep doing what you’re doing with your legacy stuff, and new initiatives, to take a long, hard look at AWS, if you’re an enterprise with the kind of resources they have, both financially and personnel?

Cris Daniluk: If you’re a big enterprise, and you’re not thinking about AWS, both for legacy and new build, I think you’re doing something wrong. It’s an emerging part of the market, it’s growing, the public cloud in general is growing quite a bit, and there’s a reason for that. So I think it should be in the thought process, and there is some logic to say, let’s leave our legacy where it is, and let’s move new apps, new build out to AWS. There could be some side effects to that, though, that as you move more and more things out of your data center, what’s left becomes more expensive to maintain, and becomes harder and harder to move. So you make those legacy problems even bigger if you abandon them for this new future model. More importantly, I don’t think you have to think that way. A lot of people think that you have to invest in all cloud, no cloud. We’re going to keep doing it the way we are. We’re going to do it differently, and it doesn’t have to be so dramatic. It can be intelligent and reasoned, and if you use the right tools, and the right processes, you don’t have to carry the burden of having a bunch of different providers, and a bunch of different ways of doing things. If you use the right tools and process, you can sort of normalize the differences between them, and choose the right tool for the problem without a lot of downside.

Data Center Spotlight: All right, so I asked you to make the case for AWS, and you’ve done so effectively and enthusiastically. All right, Cris, you’ve made a good case for AWS, which I asked you to do. Why don’t you tell me a little bit about some of the shortcomings of AWS and the reasoning that you do not universally use them in your managed cloud solutions for your customers?

Cris Daniluk: AWS is an excellent platform to build an application on, but most people that are considering AWS already have an application. The question then is, does it make sense to migrate that application to AWS? There’s a lot of challenges there, not the least of which is that because it’s so very different in design and architecture and how you pay for it, where you’re essentially buying things for pennies a minute, and you’re combining them, different types of products, servers, storage, queues, backup, replication. They have sort of a menu of I think something like 70 different services in their catalog now, all of which are priced as pennies an operation, sometimes even less than pennies, pennies a minute, and that adds up very quickly, understandably, but in a way it’s very difficult for people that have been trained to forecast a budget a completely different way to effectively judge up front. So what often happens is, that people get out to AWS, realize that they’re spending two to three times more than what they thought they would spend, and then they’re trying to figure out how to re-architect their application in a hurry to be more cost-effective. The problem with that is that you shouldn’t be spending your time architecting your application to be more cost-effective. You should be spending your time architecting your application to deliver a new functionality to the market. You should be working on your core problems.

So, the very point of what AWS was built for was to help you not have to do that, to help you focus on your application, and instead, your very first experience with it is, oh my God, I’m going to go out of business if I keep paying this bill every month. I need to fix this. One of the things that we try and do is look at, what is going to be cost-effective to run in AWS, and what is not? I mentioned earlier that enterprise databases aren’t necessarily cost-effective. A lot of distributed systems aren’t, either, a lot of people use Solr or Elasticsearch on their backend, NoSQL databases are very common. Sometimes, depending on the design, this can be cost effective in AWS, but quite often they’re not. Quite often, they can cost up to five times as much to run in AWS, as it would to simply buy physical hardware and colocate it somewhere. That’s a pretty steep price to pay, and a lot of the companies that we’re dealing with aren’t necessarily in a position where they can say, that’s not a big sum of money, I don’t need to worry about that.

The other big problem that AWS has, again, related to architecture, is that your instances are literally just that. They’re an instance running, they’re not a server running in a data center where if something happens to it, you can have a team swarm in on two hours support, get it up and running, and get you back in business. Your instances in AWS are only running until they’re not anymore. So you have to build your application to be able to tolerate any instance, any server going away at any point in time, and if you think about your application, you probably have a couple of servers that you hope never go down, that you know are vital parts of your infrastructure, and AWS is very unforgiving to that type of design, so you have to build your app to be resilient to every single layer, and that’s not as easy as it seems, and again, it gets you back to, how much time are you spending architecting to your backend infrastructure, to your cloud platform, and how much time are you spending building a product that answers your market? So, we try and look at our customers as, what they have now, what direction they need to be going in, and whether something like AWS is actually going to support the direction that they’re going, and be cost-effective? It shouldn’t be a distraction. It shouldn’t be something that drives up costs, it should be something that provides capability and benefit, and lets it move faster and more cost-effectively, and in some cases it does, and that’s when we start looking at hybrid architectures where we’re taking pieces that make sense in a data center and combining them with pieces that really benefit from what AWS does really well, and putting them together into a single solution.

Data Center Spotlight: Can you give me a specific example, a use case of maybe someone, I know there have been instances where you pulled some of what people are doing out of AWS, can you give me a little bit of an overview there, is that a fair question to ask?

Cris Daniluk: Sure, absolutely. We recently completed a migration out of AWS for a client that was running a very large Apache Solr index, which is a search engine. They were spending something along the lines of $50,000 a month to run that in AWS, and we actually brought it into a relatively unique, but interesting, low-cost physical option using essentially commodity servers. We were able to, for about $150,000 in hardware investment, and about $15,000 a year in colocation expenses, run the same application that they were paying $50,000 a month for previously. Not only did we get those cost savings, but the response time of that search cluster went down by about 50%. So, we picked a huge amount of performance as an unexpected bonus, and saved them an enormous amount of money a year, about 70% on that particular piece of that infrastructure.

Data Center Spotlight: That organization that you just mentioned, how does someone figure out whether the applications and the systems they have would actually be among those that work well in AWS? What’s the best way to figure that out?

Cris Daniluk: There are rules of thumb. Unfortunately, Amazon does its best to obscure and blur what those rules are, as they say, once you get it there, then you can figure it out, which is dangerous, because that takes time and you’re giving them a lot of money, while in the process of learning how to most cost-effectively run in their infrastructure, which is really quite brilliant for them, when you think about it. But we’ve got our own rules of thumb that we look at, we try and zero in on where are your storage I/O requirements greatest, and that typically is large databases that sort of sit in the center of your architecture. Do you have a distributed system, and if so, how is it sized today, and how does it perform today, and how will it be sized in a few years? And those are the ones that we really focus on doing a good cost analysis in AWS for. The best thing that you can do is simply try it out, the nice thing about AWS is that there are no commitments for the most part. So you can pilot your application out there, and see how it will really perform, and what it will really cost to get the performance you need?

One of the big problems that clients have is that they size out the server types and storage requirements that they think they’re going to have. They get to AWS, and they realize that, well, five physical servers in my data center doesn’t correlate anywhere near five virtual servers in AWS the way I thought it would. And even to a lesser extent, going from a virtualized environment like VMWare Enterprise out to AWS, there’s still a meaningful performance hit that causes you to go from three servers to five servers, and from servers of the type you thought you needed, to two or three classes up to get the performance that you need. All of those can be addressed through good testing with as close to live data as you can, and we try and get our clients to zero in on those areas of concern, where we expect the costs will be different than what they expect, and to actually do those pilots.

Data Center Spotlight: So the takeaway from this, Cris, is that AWS is a great company, and really, when you look at what they give you, just about unlimited capacity, turn it up at a moment’s notice, depending on what you decide to do contractually, drop it at a moment’s notice, obviously to do that is pretty expensive. It’s just a fantastic company, fantastic service, but it’s not necessarily the perfect thing for everyone, and for a lot of people, it can be the absolute wrong thing. Why is that not more in the conventional wisdom? Why do you really hear nothing much more than cheerleading for AWS, and it’s fairly rare to talk to someone like you who brings up some of the legitimate issues that exist with AWS?

Cris Daniluk: You know, that’s a tricky question. To some extent, you really need to applaud their marketing team. As impressive as their engineering team is, I think their marketing team might be the best in the company. But at the same time, there’s also this industry trend that’s happening right now, of cloud in general. Not just of running your infrastructure in the cloud, but running applications in the cloud, Salesforce, Office 365 are two big examples, and Google Apps for that matter, of people simply saying, I don’t need to run software on my desktop anymore like I thought I did. So, there’s a lot of confusion right now, and a lot of disruption right now, and you know, to be honest, we don’t know exactly what the boundaries are going to be. In five years, a lot of problems that don’t make sense in AWS today, might make sense then, but at the same time, there is more knowledge out there about what does well and does not do well in AWS, and I think people realize, you have to scratch and dig a little deeper to get to it. You have to get past pages 1 through 3 of glowing search results about how great Amazon is, and get into people telling their stories of their migrations. It’s the best way to learn about it. There’s horror stories out there of people who had to back out because they couldn’t afford it, people who lost their jobs, and then there are also stories that are less dramatic, of people who simply learned on the fly, and made it work, that the costs were more expensive, but they were able to figure it out, and they’re sharing their war stories. Then, there are a lot of people who have a perfectly good experience, that everything went well for them, and they achieved their objectives. They saved money.

So, there’s enough of that out there, that people can read and start learning about, and start to get a sense for where and when it does make sense, and also a direction for where and when it might make sense in the future.

Data Center Spotlight: Cris, let’s talk about AWS directionally. The shortcomings that you highlight, are they overcoming them, are they doing things to make it easier for people like you who are creating managed cloud solutions to have AWS as a bigger part of the mix, or just for entire companies to jump in, just purely with AWS for all their infrastructure? Are they doing things to make that more sensible for everybody?

Cris Daniluk: They are to an extent. They’re doing it in a way that benefits them greatly. One of the things I mentioned was something like an enterprise database, or distributed search index that aren’t typically good candidates to run in AWS, assuming that they’re under load, they’re busy, they’re real enterprise systems. Amazon’s answer to that is, well, in the sales meeting is, it’s not as expensive as you think, but in a product direction perspective, they’ve introduced services that do exactly those things. They have an elastic search as a service. They have an enterprise database, they have a couple, actually, as a service that you can rent, where you’re not buying instances and building your own anymore, but rather using theirs. They’re not quite as cost-effective as doing it yourself, but they’re a lot closer, they’re not these orders of magnitude of trying to run it yourself in AWS. What’s interesting about that, though, is now you’re not just using their generic infrastructure services. You’re using products that don’t have any other, there’s no analogous product to some of what they have in the market. You’re locked in, you’re betting your business on their future at that point in time, and I’m not sure that that makes sense for pretty much any business to be that beholden to another company. When you buy software, if the company goes out of business, you typically get to keep the software, and that’s not great, but it’s certainly better than it just not working anymore. With AWS, if you’re locked in there, and their business goes south, you go south with it.

Data Center Spotlight: It seems like with their growth, AWS is almost to a certain extent, become a single point of failure for the internet.

Cris Daniluk: Absolutely, I mean, it’s a front page news article any time they have a problem, and there’s a large cadre of companies that whenever they have a problem that’s caused by AWS, immediately tweet, it’s not us, it’s AWS, what can we do?

Data Center Spotlight: And then people are forgiving of that. Oh well, it’s AWS, who thought that would happen? All right, well, Cris, I’ve enjoyed this conversation, is there anything that would have made sense to cover that would help the listener that we did not cover, as we wrap up?

Cris Daniluk: I think the most important thing for people that are trying to understand how and what they should be doing with AWS is to keep in an open mind, look beyond the surface to get to real information, get beyond the marketing hype, and make informed decisions. Don’t feel like you have to be all in, or not in at all. Learn as much as possible, and make sure that you stay as flexible as possible, so that you can take advantage of where something like AWS or another provider, or who knows what happens in five years, be able to take advantage of it without making yourself constrained by it later.

Data Center Spotlight: Cris Daniluk, that’s some good advice, appreciate you being here with us today to talk about Amazon Web Services, and the best use of Amazon Web Services. If someone wants to get in touch with you, what is the best way to do so?

Cris Daniluk: You can reach me on LinkedIn, and I’d love to hear from you.

Data Center Spotlight: That’s Cris Daniluk, you forgot the H in your first name, C-R-I-S D-A-N-I-L-U-K, correct?

Cris Daniluk: That’s correct, but you’ll have to blame that on my mother.

Data Center Spotlight: Are you the only Cris Daniluk in IT infrastructure in Northern Virginia?

Cris Daniluk:   Yes.

Data Center Spotlight: That’ll make the LinkedIn searches a little bit easier. Cris, thanks for your time today, some good stuff, I appreciate it.

Cris Daniluk: All right, thanks Kevin.