DevOps Chats: Software Architecture for Cloud Native, .NET Core and Open Source

Mitch Ashley: Hi, everyone, this is Mitch Ashley with staging-devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by a very special person, a good friend of mine, Donald Lutz. He’s a principal software architect at many companies. We’ll kinda get into that, but he’s focused on microservices cloud, he also is very involved with Microsoft in his background.

He’s currently involved in several things, but he’s with his own company, Technetronic Systems, and he’s worked with Faction, ViaWest, he actually was one of the original co-founders of BoldTech Systems with me back a while ago.

So, our topic today is microservices going cloud native. Donald, welcome to DevOps Chat.

Donald Lutz: Thank you, Mitch. Thanks for having me on. I really appreciate this.

Ashley: Oh, it’s a great honor of mine, you’re a good friend. And to that point, why don’t you tell us a little bit about yourself, introduce yourself to our audience, tell us about what you do.

Lutz: Yeah, I’ve been sort of a principal software architect and a director of software engineering for different companies like Faction, my own company TSI and I’ve really in the past five or six years focused on cloud native and taking—you know, getting microservices and applying that to building cloud native applications. That’s really kind of what my focus has been. And there is a lot of DevOps that goes in there. There’s also a lot of architecture and design, so it’s kind of a combination of all of the above.

Ashley: Mm-hmm, excellent. Well, just full disclosure, I mentioned before, you and I have known each other a long time. We started working together back in the telecommunications industry and I did a startup called BoldTech Systems, which you were a part of founding and getting going and, you know, you’ve hired me, I’ve hired you. [Laughter] We kinda worked with each other—which, by the way, for folks earlier in their career, that’s a great thing to do. Keep your friends close, because you’ll get to work with them again.

But you’ve had this consistent pattern in your career where I know you were very Microsoft and .NET focused for a while and then kinda moving into cloud and broader technologies. I mean, you’ve worked on Java and other things, too, it’s not like you just did Microsoft, but tell us a little bit about your philosophy on technology. Are you a fanboy Microsoft or how do you kind of view that with everything else that’s going on in the technology world?

Lutz: I mean, I could be viewed as a fanboy of Microsoft. I mean, I helped—I’m running now .NET Core, an open source Meetup for Microsoft. I really like where they’ve taken .NET Core. I mean, I would claim, six or seven years ago, Microsoft realized that they needed to stop being so proprietary and get more into open source.

Ashley: Mm-hmm.

Lutz: So, they really kind of redid all of .NET, and .NET Core is fundamentally, it allows you to build microservices, traditional apps, too. But, you know, they moved to the open source arena because they knew their future was Azure, it wasn’t really Windows in the traditional sense. You know, 75 plus percent of all their things that were under Azure are Linux that run with another product called Service Fabric and, you know, that to me, is a great technology. I have done Java, I’ve done Go, I’ve done a lot of different things, you know? I have a large Microsoft contingent, but they’re all part of the same thing where everybody’s trying to get to cloud native—what does cloud native look like? Whether you’re Microsoft, Pivotal, VMware—everybody’s trying to figure out how to you get there, you know?

And a lot of people have taken steps in between, like, some people are trying to run VMware on the cloud, but I’m seeing a big push where I’ve been talking to a lot of people, I’ve been working with a really large financial institution right now, and they’re actually punting on the in between and they’re going completely cloud native using both multi-cloud and in their definition, is Azure an AWS?

Ashley: Interesting. And I’m aware of who you’re working for. I know we’re not talking about the company name specifically, but give us an idea of the scope and magnitude. Because we’re not talking like, you know, a credit union or a large regional bank.

Lutz: No, they’re one of the largest financial institutions in the world. So, they are looking—they’re moving everything at the top level of their management, you know, CTO level, they decided that they wanted to do microservices, domain driven design. They wanna go all cloud native. It’s either gonna be running in AWS and Azure. They don’t wanna own any hardware any more, they don’t wanna have any of it. For them, it’s an amazing kinda revolution, because they’ve never gone there, and they’re kinda viewing that they wanna run their company in a technological era kind of like a startup, so they’ve really changed their whole view.

So, I was kinda surprised that they were that interested in that and the fact that they looked at the cloud and they said, “We need to go cloud native, because we don’t really need in between steps because then again, we have another shift and lift then in five years,” and we only wanna have one shift and lift, so I was really surprised they even thought of that.

Ashley: Well, it is, and I’m very surprised by that, too. It’s certainly—you know, I don’t talk to every big financial institution in the world, but to make that level of commitment of, we are gonna go cloud native, not like we’re gonna lift and shift what we have and just kinda live with legacy, if you will, and build new things in cloud native, they’re actually taking the major part of their software assets, the systems rely on day to day and they’re shifting that just going to cloud native in one big step—well, several one big steps, apparently. [Laughter]

Lutz: It’s a lot of big steps, but I mean, it was kinda surprising, because you know, they brought in executives who had been more out in Silicon Valley and not traditional financial people. So, they’ve made a different, and they went and found a bunch of us who have been kind of in the cloud native microservices space and brought us on. They kind of viewed it as, they’re trying to build, like a, sort of the Super Bowl winning football team, that if you get the top players, then you can get this to happen.

Ashley: Mm-hmm.

Lutz: That doesn’t mean there still aren’t interesting political battles, since this is a company that’s historically, you know, they still have DB2 and green screens, so, you know—

Ashley: Sure.

Lutz:—it’s not like it’s completely, “Oh, yeah, this is awesome!”

Ashley: Right. [Laughter] It’s not like they started in the Linux area, though—era.

Lutz: Yeah.

Ashley: So, say a little bit about—so, you have a unique perspective of, you’re not the kind of entrepreneur that’s worked in a from scratch startup in the garage every time, repeat, repeat. You’ve done that, but you’ve also worked with a lot of existing organizations. I know ViaWest had been acquired, I think, by Shaw Communications before you joined and they were going through a transition.

Lutz: Yeah. I went there because they were—I was looking for a place that wanted to move to microservices and they had been acquired by Shaw. I knew their CTO, we chatted, so they hired me as a Director of Software Development, and the goal—they were all .NET, so I moved into .NET Core and we started building microservices, which is the whole idea of, there’s a lot of stuff that comes out of major design and bounded context.

So, instead of having, you know, your monolithic application that has customers and data center stuff, we started building services like, “Here’s your customer service.” And it all deals with that. We also brought in some other, DevOps people know about more CQRS and event sourcing. So, the whole idea, each service has its own database that it writes its data to. It then issues commands—so, say I created a customer, I say “create customer.” That causes an event called Customer Created that gets stored in an event store, which almost looks like an audit log. You never delete from it, you only add to it.

So, if you ever have any problems with all these distributed services failing, you can rerun this audit log and get back to where you were in the beginning.

Ashley: Mm-hmm.

Lutz: So, it’s kind of a powerful concept. These concepts have been around for a while, but you’re starting to see them being employed at a larger level. Because, you know, the idea of synchronous communication doesn’t work well in cloud native. In fact, you really just want to expose public APIs, and behind the scenes, you want data streaming, you want eventing. You don’t want any asynchronous communication, because it’s not the right approach. You can’t scale, and you’re gonna be dealing with eventual consistency.

So, you know, once you start dealing with that, once it’s not in one box, you know, eventual consistency is it.

Ashley: Mm-hmm. Where you’re going to kinda stateless applications, right? Microservices—it’s interesting that the trend is, obviously, is build much smaller things, reuse them, create services out of them. But there’s gonna be many, many more mass quantities of them. It’s not like you’re going from—you’re going from one monolith with a lot of functions and APIs to really small pieces of software that are containerized microservices.

But what comes with that is now, all of a sudden, you have the problem of managing that, communicating across and synchronizing when it is asynchronous how you communicate, handling troubleshooting, knowing what all those services are. It sounds like what you’ve done is, you’ve reached back into some maybe techniques and technologies or knowledge of past things and bringing that into this microservices Fabric world. Is that true?

Lutz: It’s true. I mean, I get a lot of Kubernetes and all that, and you know, even though you start doing Kubernetes and a lot of DevOps things, it’s sort of the thing that I find interesting. It’s kinda like, sorta what you on the Phoenix Project, you know, you need to be able to manage all those things and be able to deliver all the time and monitor it and log it. The problem is that that’s not just answered by creating a bunch of scripts, you know what I mean? Like, even though I’ve used Terraform, Puppet, I think there’s a larger scale of solution.

Microsoft just released a new framework called Dapper, and Dapper is basically dealing with the microservices run time. So, it’s a run time that can sit on top of whether the microservices can go talk to Kubernetes, because you need to be able to handle that run time stuff as well as the development issues that go on. Those things are not two separate things, and I still think that’s the—one of the large DevOps challenges is how do you get everybody on board, you know?

And at ViaWest, it was pretty sophisticated, because all our DevOps people were software engineers. So, we just wrote software for most of our problems. We either built it ourselves or used various tools. But then I went to a company called Faction where we had a less mature DevOps practice, and we were building multi-cloud storage, so we had a lot of microservices that allowed us to do Isilon, Dell storage, you know? So, we built a fairly sophisticated microservices platform. We didn’t have the DevOps practice, which resulted in some interesting issues, because the organization was not mature. Everything was manual. So, it’s really hard to roll those out, because you roll them out and then you can’t roll them out as well as you’d like to, because nobody’s catching up.

Ashley: Mm-hmm.

Lutz: They’re like, “Oh, well, how do we do that?” I’m like, “Eh, yeah, that’s an issue.” There’s no tests going on, you couldn’t do a Chaos Monkey. So, that was—in that case, in that case, it was pretty challenging, you know? And also the whole—I’m thinking a lot lately about security and the fact that you have all these cloud native accounts, but you have, you know, you have these very, you have credentials that give you access to RDS and Amazon, and then those aren’t really secured real well, and there’s Vault and there’s multiple ways to do it. And the whole security thing is, I don’t think, well cooked in a lot of ways. I would claim it’s not. [Laughter]

Ashley: Well, application security is now become—becoming, maybe both—you know, one of the top topics. It seems like we’re just sort of, we do things differently, and then we have to go back into and go back and address, like, “Okay, well, how are we doing this now?” Because, you know, it’s not about protecting servers and network devices. We’re exposing so much of the tax service as an application through all these microservices, APIs, et cetera. You gotta kinda rethink it.

And then you’re automating the development environment to do 15, 20, 100 code deploys a day, potentially. Well, that takes automation—that means all those credentials, security keys, certificates, whatever are built into a process that’s all that secure. I think that’s what you’re talking about is, you have to do it, but you have to also go back and re-look at it to say, “Are we doing this securely?”

Lutz: Right. You have to look at things, “Do I have—is data at rest encrypted? Do I check any kinda hash on it? Is the data in flight encrypted? What are the levels of security? Do I let my JWT token float to one service, to another service, because that could be an attack, too.” Because I went to service one and now I go to service two and I’m using the same JWT token, but maybe the claims on that second service, it shouldn’t be allowed to do those things, because the claims that it used from the first service lock it down better, but in service number two, it allows you suddenly, “Hey, I could go look at different accounts that Mitch has, and well, that’s interesting.”

So, then you start thinking about, you know, how—there’s a couple good security books about that whole idea of, you know, JWTs are interesting, but the problem is that if they float, if we haven’t figured out the right way to share those, that’s another thing that causes lots of problems.

Ashley: Mm-hmm. And for me and for our listeners, can you define what a JWT token is?

Lutz: Yeah, it’s a JSON web token. It’s just—

Ashley: JSON, right?

Lutz: Yeah, yeah. It’s just a token that basically you can use with an API. You know, it’s a little—it’s really for APIs. OAuth, you can do both. OAuth is more effective with users, but those are, you’re seeing the industry starting to adopt JWTs between services. So, now people are saying—but the problem is, the concept behind it is a claim. I wanna make a claim that your name is Mitch and you live at this address and you have this phone number. And in this case—oh, by the way, you can look at your Social Security number, but in this case over here, you shouldn’t be able to look at this other person’s Social Security number.

So, you start making all these claims, and then you have to start getting really smart about how do we define all these claims for these very distributed systems.

Ashley: Mm-hmm.

Lutz: Which, historically, distributed systems were everybody—oh, if you had the root password, you’re golden. We’ll just use that and everything will connect to it.

Ashley: Yeah, interesting. It’s really interesting, not being that close to it but following what’s happening with it. I think it’s based on, like, RFC 7519, so it’s based on a spec. It’s about how claims essentially get bedded into what we traditionally think of as a data interface specification between services and applications. Anyway, we’re not a JSON JWT podcast, but interesting sideline.

I wanna get back to talking about microservices in cloud native. So, as you’ve worked with existing applications, and even at Faction with something that maybe didn’t start out in a DevOps world, what do you see as some of the kinda approaches that make sense and maybe things that you or others have tried that turned out not to be the best path or dead ends that you’ve corrected from and now here’s how you do things?

Lutz: The first problem I see is, you make nanoservices. So, instead of—you know, you make a microservice that tracks time, and it’s so small that it’s not very effective. There’s a software architecture discipline called the main driven design, which is about when you build an application and a service, there’s a domain that it covers, whether it’s a customer or a product, and it needs to be the right size. And if it’s the right size, it’s what’s called bounded context. Because the whole idea is, you want that service just to answer that bounded context, which is based off the idea of a single responsibility principle. If it does more than one thing, you have a problem.

So, kinda getting people to understand that is complicated at a business level, because you know, business people historically want to throw everything in whatever they’re building. Like, “Oh, can’t we just, why do we need to have such a small bounded context? Can’t we, like, throw all the addresses and products in here?” I’m like—no, because we need to keep those separated, because we’re trying to, each microservice has to have a very distinct separation of concerns, because if you don’t have that separation of concerns, you end up with a monolith, which is what we’re trying to avoid.

Ashley: Yes, end up back into the spill over again of domains into, this microservice has too many things in it and now one thing’s affecting another, versus let it be a single purpose in our service.

Lutz: You know, and that’s why we actually saw that this large, large financial institution has adopted domain driven design and they’re actually building domain models for all their applications and everything. So, there will be demand models behind them.

So, it’s just not the microservice, they actually can say, “Oh, by the way, this is what this financial instrument looks like, and it actually maps to our true domain model and then will map to a true microservice.”

Ashley: Mm-hmm, okay, so domain driven design, nanoservices, or kind of focused functional microservices that don’t kinda spill over.

Lutz: There’s another thing, too, you need is what historically—Chris Richardson has a book called Microservice Patterns, and it’s the idea of what’s called a microservice chassis. So, once you start building microservices, you either need to make a framework or use a pre-existing framework, whether it’s Spring Boot or some stuff in .NET Core so that when you write the microservices, all the logging is done the same, all the separation and concerns are handled so you deal with all those non-functional requirements so you don’t spend any time developing it.

Ashley: Mm-hmm.

Lutz: But then once you do that, then you have to start saying, “Well, how does this play in the run time world of Kubernetes?” Which gets into something you said earlier—we need to sort of connect the framework with the run time stuff, so that’s kind of a problem you see in Pivotal, Microsoft. They’re all looking at how do we get these really interesting software engineering frameworks that then wrap into run time things that are equivalent, so we end up with, you know, “Oh, they’re the same thing, they’re isomorphic.” They’re not different, they just look different because we view them as different domains.

Ashley: Mm-hmm. Interesting. Well, we have a couple minutes left. I wanted to just take a little bit of a left turn—not quite a squirrel moment, but a little bit of a left turn. I’ve been doing some podcasts here recently with folks, usually developers, DevOps engineers, Ops folks who are talking at the Spinnaker Summit Conference, speaking on Kubernetes and Spinnaker Open Source.

So, we’ve had a great opportunity to have other developers, real hands on technical folks on the podcast. And one of the things that I always encourage folks is, even in the open source community, you contribute not just by writing code, it’s by sharing your knowledge, applying these tools and software and sharing the knowledge and networking and building community with others. That’s vital to having, you know, vibrant software is having a vibrant community about it. And I know you’re very active in your own developer circles and communities in creating those. Tell a little bit about what you’re involved in now and why you do it.

Lutz: I like doing it to share. I mean, I would say about four years ago, there was a gentleman running a group, there’s a group worldwide called ISO, the International Association of Software Architects. And he was trying to get the group growing locally in the Rocky Mountain region—you know, Denver, Wyoming, that kinda area. He hadn’t gotten very far, so I got kind of involved, and we’ve actually grown it over the 4 years that we have 2,000 members and we get about 60 to 100 members coming every meeting, and it’s truly talking about architecture. It could be the Kubernetes architecture, it could be domain driven design. Last month, I talked about how to build microservices that are stateful. Surprisingly, there’s a company called Cloud State. Microsoft and ACCA, they figured out how to build stateful microservices. We won’t get into that.

Ashley: They didn’t bring pitchforks and, you know—

Lutz: They didn’t bring pitchforks. It works pretty well.

Ashley:—torches? [Laughter]

Lutz: But we get different people. We got the guy who’s the original guy at FedEx who created all their stuff that looked like Kubernetes in the ‘60s come talk. We’ve had people from Slack. We get a lot of people to come talk. I’ve talked at the Carnegie Mellon architecture conference, I can’t remember what it’s called. I talk at a lot of Microsoft groups. I run a Meetup for Microsoft. They have a few .NET groups that are dying, so I pitched to them—why don’t we turn that into a .NET Core and open source group?

Ashley: Mm-hmm.

Lutz: And so, I’ve been running that. I’ve grown it—that’s almost to 1,000 members now. It was actually only 200. So, in the three or four months, I’ve actually grown it. I like creating communities of technical people, because when you do that, you actually can solve interesting problems. You also get to hear other people’s experiences.

I can give you an example—one of the guys who get involved with ISO, I was talking in Colorado Springs at an event, I can’t remember where. And I was outside the event and this guy, Tyler, came up to me and started talking to me and I discovered he was really technical. And he said, “What are you doing?” I said, “Oh, I’m talking about microservices and that.” And he says, “Why don’t I come in?”

And we actually—he became the person who asked me the most questions during the event and then he became one of the four members to help us run the Denver Software Architects Group.

Ashley: Mm-hmm. Interesting.

Lutz: And we became really good friends and it was like, you know, just talking to somebody outside an event I was gonna speak at.

Ashley: Very interesting. Well, you know, I think it’s also, all of the things you said, I totally agree with. It’s also a way for someone who wants to know about a topic, maybe, and I like how you’re kinda keeping the groups fresh on what’s happening and new in Microsoft and other worlds of software. It’s a place for folks to go and learn about it and hear what other people—even if you’re very engaged in a new technology already, it’s, what are other people doing and what have they learned from it? What can you share, what can they share with you, and vice versa. So, it’s a healthy thing to do.

Lutz: Yeah, I really enjoy it. I find it more enjoyable—it’s up there with this robotics stuff FIRST I do in terms of raw enjoyment, because it’s so, you’re plugged into so many people where, when you know, when I’m doing consulting or helping build a startup or all of the above, it’s enjoyable. But it has—this has more of a kind of like, not selfless, but it has more of a giving back vibe that I like a lot.

Ashley: Mm-hmm. Very good. Well, hey, we’ve run out of time. You and I could talk for 12 hours and, you know, up ‘til 3 a.m. in the morning. There’s so much fun things that you’ve done. I really appreciate you being on the podcast.

Lutz: Well, thank you for having me on. I really appreciate it. I really enjoyed it.

Ashley: Well, good. Me, too. You’ve listened to another DevOps Chat. I’m very pleased to have my guest, Donald Lutz, on with us today. Donald is a principal software architect, he focuses on microservices, cloud, cloud native, generally Microsoft technologies, but a lot of other things and he’s with Technetronic Systems. He’s also been with Faction, ViaWest, BoldTech Systems, a number of different companies, so he has a lot of software background. And again, our topic today was microservices in cloud native.

Thank you, also, to your, our listeners, for joining us. This is Mitch Ashley and you’ve listened to another DevOps Chat. Thanks, and be careful out there.

— Mitchell Ashley