Scaling Infrastructure to Support 100X Growth w/ Dan Drew

ABOUT THIS EPISODE

Could you grow your product into 200 markets without needing hundreds of people to manage it? One company is on a mission to do just that. We're joined by Dan Drew, CTO at Didja, which created a product that lets customers record local broadcast TV on their mobile phones, tablets, or TVs.

Dan joins the show to discuss:

  • How Didja partners with legacy broadcasters to monetize and leverage their content
  • The #1 reason companies have trouble scaling
  • How to scale your software and your team without blowing out your costs

Want to hear more stories from high growth software companies? Subscribe to Application Modernization on Apple Podcasts, Spotify, or check out our website.

Listening on a desktop & can’t see the links? Just search for Application Modernization in your favorite podcast player.

You are listening to application modernization, a show that spotlights the forward thinking leaders of Highgro software companies. From scaling applications and accelerating time to market to avoiding expensive license and costs, we discuss how you can innovate with new technology and forward thinking processes and save some cash in the process. Let's get into it. Today we're speaking with Dan drew about scaling infrastructure to support rapid growth. Dan Drew, is CTO at did you a company that is created a product called local bTV that let's customers watch and record local broadcast TV on their mobile phone, tablet or TV. In this episode we'll talk about scaling infrastructure and controlling costs. This is an exciting growth story with some great takeaways. Here we go with our guests, Dan drew, hid and welcome to application modernization. Thanks for joining us today. Thanks for having me. So, Dan, tell us a little bit about yourself, your company, did you and the problem you're solving for customers. Sure. Yeah, so a little bit about myself. I've been building software professionally for a little over twenty five years. Started out doing enterprise software at Microsoft large scale. Therefore basically building software that's large companies used to deploying manage computers and software in their enterprises and then currently down at a tiny, tiny, tiny, minuscule and start up and Silicon Valley doing video software and done everything in between. So done video, done enterprise, done security and you know, all kinds of different businesses. Even did a year at Myspace for those of you who like Trivia, and that was interesting. So done all kinds of things and seeing the whole spectrum from large to small, to get to bed and what to do and what not to do. And our company did you here in the valley. So what we're doing, as I mentioned, we're a very small company and what we are aimed at is taking local broadcast television and really trying to take that into sort of the modern age of digital TV. You know. So if you think about Apps like Hulu and sling TV, there really hasn't been a successful attempt to take local broadcast television and give it that kind of application where any user on any device can watch it the living room, the coffee shop, bedroom. Right now you have to have an in town on the top of your house which is connected to your living room TV and that's how you watch over the air television. So that's what we're focused at and that benefits users because they they get a bunch of content that you just don't have in those other apps they're more focused on like the AIS and you treat traditional cable companies. And it also benefits all these local television stations that you know, are a little behind. Why? Frankly, just because of the industry they're...

...in, where they don't have these APPS and they're obviously not in the other leader APPs and they also don't have their own direct apps like the paramount plus APP or thing like that. So you know, they're really looking for an avenue to get their content out there, you know, on new devices, to new users and also do modern things like add insertion and, you know, find more ways to monetize and leverage their content. Right. Yeah, I've I've certainly experienced some of these companies that are trying to solve for this in the past. You know, I bought a Roku and tried to, you know, get my local channels and I think one of them ended up getting deleted from the APP. Tell us a bit about, say, the changing market and what did your is doing? I know you spoke about you know, some big growth plans. Can you share with us, you know, how the market is changing and what you're doing to solve this on a bigger scale? Yeah, well, I mean, I think part of it is just this realization, you know, that broadcast channels need to, you know, find more ways to reach their users right you know, they can't stay tied to an untent, you know, just the traditional ways of viewing broadcast television. So that's where we're trying to help. And then I think also, you know, part of the reason, you know, some of those other solutions have struggled or ran into issues, you know, where they violated either content rules or, you know, rented to legal issues with the AP stores or whatever, and this hasn't changed. Unfortunately. This is evolving, but it hasn't drastically changed. But there's a lot of complexity around content rights, you know, and it can be anything from where you can show it, when you can show it, what device you can show at what time of year, what phase of moon, whether or not as the third summer solstice of February. You know, there's a lot of complexity about content and how you can use it. So some of these APPs have tried to just say screw it and you know, and just put that content out there. And you know, that's where they run into problems, you know, based on just the essentially illegal or is not approves in some way or another. So that's why, you know, we're trying to do everything by the book. We're trying to work with all the major networks and make sure we sort all those things out before we put in any content that we don't have approval for. So it I mean, it does mean that we can't currently put get every channel in every market, but it also means that we are legal and we're not at risk of someone pulling us or suing us or something like that. So it does add some extra challenges for us on the business side, but we have good relationships of all the companies and we're, you know, trying to work through these issues with them to come up with something that's mutually beneficial. Right. And we spoke pride to jumping online around your growth plans. You know, when you came on board you were in, you know, a couple of markets. You're now approaching fifteen markets in the US with plans to get to two hundred markets. So what do you like...

...from infrastructure perspective. You know, what are your top priorities when you consider, you know, growing that significantly. Well, it's a case where, you know, every dollar and every line of code counts, right. You can't scale any company or any business if you're tripping over your own complexity, and so a lot of what we've tried to do is just get all the fat out of the system. That includes software fat. Of You need to make sure that our software is is easy to use internally, not only as our users using our APPS, but internally our software has to be easy to use for us to manage, you know, all those different markets. It has to be scalable in terms of size, but also in terms of human effort. Right. I worked at a lot of companies where, you know, their scale testing involves, you know, three people good enough, and then when you really start scaling, the software becomes unmanageable, you know, and then you end up needing like hundreds of support people, you know, dealing with customer issues. You need lots of operations people. You need a network operation center to, you know, make it things sure stand up. So a lot of what we want to do, especially as a small company. You know that can't burn money on, you know, resources or people or technology. was making sure that our software is easy for our small operations to use and requires the least of out of human intervention so that we can scale as a company. Like, if we had tried to get to the number of markets we have now with what we had when I started, we'd be dead in the water right there was just a lot of human intervention, a lot of human configuration and management involved in, you know, keeping a particular market Allie. So a lot of the focus has been building out our platform to make sure we make this as simple as possible. So what types of technologies, platforms processes have you implemented to help reduce that human effit and make it easier feel operators? A large part of it was moving to continuer based system. So obviously a big trend now regardless, but absolutely critical for us, where we can have things going up and down and, you know, we know when things should mention is we're a hybrid model because we're dealing with local channels. We need a presence of some kind in every market so that we can receive those over the air channels and then also, because we're Geo fenced, you know, the way TV markets work, you can only watch, watch what's in your local market. So we also deliver, you know, the content back locally, right. So we don't send all the content to a central location like, you know, a lot of the other TV APPs do, and then just you read it through cdns and all these other things. We actually set up hardware and services in each market and order to manage that in any sort of scene way we need to abstract it.

So what we did is move from managing machines directly to essentially each market is a cougnudies cluster with essentially managed service that's in Amazon that is able to essentially manage all of these different clusters. But what that allows us to do is, instead of having to have a human go and decide, you know, I'm going to put this channel on that and configure in whatever, we can have a central system say look, all right, we want channel X up, figures out. Okay, I need these containers. That needs to go to that data center and a couple minutes later, you know, that's just up and running with nobody touching it, just going to a UI and click a few buttons. So building that kind of a system a from an internal platform perspective and then using software like Amazon and those services that are available, and cub nities from a contumer management perspective. We're absolutely critical for us as far as being able to scale home. Got It. So Cuban Eddi's is a is a key pot of this platform you speak of as you're innovating and developing this platform. Like, what's our big challenge that you've run into, you know, in the last four years that you know really sit of soot out and it's quite hot to solve. As far as specific instance, I don't. Don't of that would call out any one thing. I mean a lot of what I've seen as far as companies having difficulty scaling is chalk down to just in experience with scaling or not understanding the software and how to scale it, either building your own software or using third party software. So, you know, giving one example is whenever I interview someone who says I've used Cassandra, you know I'll ask them jokingly, Oh yeah, how many times did you have to rebuild that cluster? Because everyone who tries to scale CASSANDRA INEVITABLY GETS IT wrong the first half a dozen times. Right simply from just not understanding how it works in terms of low balancing and scaling, you know, God, it's great, and then you add more data and more data and more Datas like, Oh my God, it's falling down, you know. So it's one of those I don't say it's almost a lost start, but it's certainly a missing art. And especially, as you know, a lot of these startups are built by people that lack that experience, you know, just because of you know, what their experience has been. So, you know, trying to build the software itself, you know, like I mentioned, like focus on operability and monitoring and logging, you know, making it easy to diagnose and trouble shoot. You know, these are things that tend to fall off the table, you know, from a software engineering perspective, you know. And so gets deployed and then problems happen and it's really hard to figure out what happened and it takes a lot of manual effort. You end up doing things like making your developers go on call because they're the only...

...ones who understand the system, which, you know, in my opinion, means you didn't do it right, right, you know. So you you see companies, it's almost like when a you've injured a leg or something in the muscle starts growing a weird ways. You know, you start seeing the company try to compensate, you know, for these issues as far as how the software is built, and you know, you start seeing the sustained engineering team grow and the sales engineering team grow in the support team grow and the operations team in the knock and everything. So I think a lot of it is just I think there needs to be more focuses. You know, as these companies want to scale, they really need to focus on getting it right when they're small, because there is a tendency to say, oh, we're small, who care, is just build it. And this effects not only the software scaling but your costs as well. Right. It's like, Oh, we'll just sign up for this because it's easy, and then, you know, six months later you're like, holy cow, we got a tenzero a month bill. Right, every CTS West, nightmare, right, CTEO and CFO and CEO right in the whole see level, because you know that it directly impact your burn rate as a company. And then that goes into your hiring and what else you can spend money on. You know, a lot of companies tend to cut corners, not realizing it's going to bite them in the ass very, very soon. You know. They think, Oh, you know, what will matter until a week grow to like two hundred people or five hundred people. And I've been at companies where they started struggling with this at fifty, you know. And so you talk to companies where over at twenty people, we're going to double on it, and that's where a lot of startups, you know, are thinking. It's like, okay, well, you need to know, start thinking ahead. You know, this is where you really need to put your big boy pants on and make sure your software scales and your team Scales and you've got the right mentality in there and you're managing your costs, because all you need is one big deal that really makes you grow and really get your usage up and all your costs and all your scaling just can go out the window. So that's why that's been such a high focus of mines and starting the company is making sure that, you know, we can go to two hundred markets and not need a one hundred percent team to manage it. Yeah, and I want to just pull on one of the points you mentioned there around managing the costs. And when we spoke prior to this call, you you said engineer is need to have a business hat. Yeah, and you know what we discussed was that, you know, engineers need to think about the business and think about the costs. Can you tell us how you, you know, maybe enable your engine is to think about the business a bit more and consider costs and how some of the architecture decisions may impact those costs in the future? Yeah, well, I mean it was a lot of reasons engineer should be aware of the business, just from every aspect of software engineering, but especially, you know, in a small company and start up where you you know, you don't have money to burn. There tends to be well either when you're starting. You just sign up for something because it's there, you know, and...

...it's easy, but then you don't really look at what the plan is and what happens. If you know what, where do we expect our numbers to be, say, a year from here? You know? So really encouraging the engineers, you know, as you're evaluating technologies in different ways to solve this problem. You know, the easiest thing is not always the best thing. You know, sometimes it's worth spending a month doing something that's harder, but then we don't have to rewrite and throw it out the window in six months. Right. And so, for example, when I joined, we had a number of contracts where we were already paying for us, you know, a large amount of money monthly, and I did the math, you know, for where I expected our numbers to be, you know, say a year from now, based on how we were trending, and it was going to go up by an order of magnitude. You know, we were potentially, even for one service, going to be paying like a hundred k a month for or why out, you know, just based on what we were throwing at it and what we were using it for. You know, it really had to look at okay, what are we really doing with it? What are the other alternatives? And, you know, in some cases I was able to take those and even though it cost us some engineering time. I think one of them took us like three to four months to swap it out, you know, which is not trivial. But when it came down to the amount of money we saved, that was definitely the right call. And you know that is a business decision, not an engineering decision, right, you know the kind of thing go to the CEO and say, look, we need to help development and do this, otherwise, you know, a burn rate. It's going to be ridiculous and obviously, in my case, my CEO. So heck yeah, into that. So you know, those are definitely, you know, things that you have to keep in mind as an ensuring leader, and you know your engineering team is. Every time you make a technology to subcision, there's a cost behind that. You sometimes it's a monetary cost, sometimes it is a human cost, you know, manage and learn how to use it and things like that. So those things all have to be carefully evaluated again against your business needs as well. Like I mentioned, you know, a couple of services we have we weren't even using ninety percent of their features and it was costing a lot of money. So, you know, from a business need it wasn't really what we needed in them. From the cost perspective it was, you know, bad for us. So obviously needed to reevaluate those things. Yeah, I think that is really interesting because, you know, sometimes there may be a perception that you know, engineering leaders are out there sort of creating new features, driving value building products. But you know there is another end of that stick where you can create value and drive costs down by being smarter about, you know, what you're using and in some of the technologies that are being implemented. I like that story a lot. I want to pull on another thread that you mentioned just before around you know, experience. You know we're in a market right now particularly you know it's some of these innovative technologies like Cubin Eddi's where it's hard to find talent, it's hard to keep talent and you...

...know it's odd to find people with experience right so what are your thoughts around the market right now and what are you doing to empower or train up your team to be able to use some of these modern technologies in your business? Yeah, well, I market wise it is very challenging. I'll generalize a little bit because it's hard to get into details with that, but I mean there is because of the popularity of the industry, there has been a lot of flood of New People into it, you know, and you know there's like the boot camps and you know, all kinds of people trying to get into the software industry for many reasons, and so it does mean you have a lot of variety and their experience level and doesn't mean that those people are good or bad or anything. It just means you do have, I would say, more heavily weighted, for my experience, of hiring on the lower experience level or if they have longer terms of experience, it's generally more limited. You know, you don't see a lot of people that have gone in depth into a lot of things, right. So if you're looking for people that are really sort of in depth and really, you know, have become experts on say, something like Cooper is, you know, that's a very, very, very tiny pool, right, and the different companies also have different way of how they're organizing that, you know. So you have like the whole site reliability engineer of old less every where. You know, some companies just don't even make the engineering to learn. That's like that's an OBSC problem, right, and then you get into these similar cases where the engineers don't really understand how things work when it's deployed and then that affects the software, you know. So I would almost expand engineers need to know. But the business they also need to know about everything. You know, they need to understand how their stuff that's going to run in you know, and how it operates and even if they're not going to manage it directly right. So you know, there's there's that aspect and these are very difficult technologies. You know. That's one thing that hasn't changed in software is that, you know, a lot of technologies coming out, but they're certainly not getting any easier to learn how to use, you know, and you know whether it's just crappy documentation or just very, very complicated to understand and configure correctly. You know. So it's not really something where someone you know can just hop in and, you know, do it well. You know, you really need to spend quite a bit of time becoming an expert in something. So again, finding people that have had that opportunity to get that experience is very, very small pool of people. So that is definitely a challenge, you know, and you know, as a hiring manager trying to use the usual pools of candidates. I mean you can literally throw out a hundred resumes before you have a phone screen with one. You know, depending on what you're looking for. So the it is definitely a challenge right now from, you know, my perspective and my team. You know, I'm always looking for opportunities and, you know, any manager should be doing this for their team,...

...for them to, you know, go outside their wheelhouse and learn new things. And you'll obviously, you know, work with your employees to see what they want to learn. You know, don't just throw them things they have no interest in. But certainly, because we are a small company, in a startup, you know, everyone kind of has to wear many hats. You know, we don't have room for the person who says I want only want to do this and refuse to touch anything else, right, you know. So, for example, you know, my IOS developer has taken on Broku and you know my android developer has taken on our web client and you know, and this has all been great for them as far as learning new technologies and broadening, you know, their experience and doing some operational stuff and learning more about, for example, like CICD, you know, using, you know, Jenkins and other systems like that and becoming more experts on that, getting better idea of the operational side. And obviously we're because we are a small team. You know, every there is no hard wall between operations and Qa and development like you might see it bigger companies. So you know, we're all talking, we're all working together and just my philosophy is, you know, engineers own their software. You know, you don't get to say it's not your problem just because you hand it off to deploy right. So, you know, I absolutely want them to understand that. They need to know how this works and you know, you know you need to do better logging and things like that. I need to make sure operations can knows how to use your software deployed and when to contact you and all s fails and things like that. So there is an ownership aspect to that. That's interesting. Yeah, let's deep dive on that. So you're saying you know, you want your your team to own it. took us through a little bit more detail how they approach that. Well, I think it's all as I mean, there's the just the mindset, you know, of this is not a dead problem versus a QA problem, versus an ops problem verse. You know, if Qa is having trouble testing your software, that's you're a problem. You wrote the software right. If operations is having trouble keeping your service up. That's your problem. You wrote this software, so you know that ownership of you know you wrote it. You need to make sure that you've done all the due diligence and you understand how it's going to be used and you understand not just the engineering requirements but you know, first and foremost, obviously you've understood the end user requirements and the business requirements, but also are their operational requirement, support requirements, QA requirements. You know, for example, you're doing automation, you need to make sure that your APP has, you know, the necessary things in it so it can be driven by an automation framework all right, which is not something an engineer would normally do just by default. Right. So that needs to be a collaborative effort between the QUA team and the engineering team. So there's a lot of different parts of that and depending on which angle you're coming at it, you know it's different part of the problem, but at the heart of it is really the person writing the software and the team writing the software. You know, the support team has no ability to add better lotting, right, the operations team...

...has no ability to add better monitoring and health checks right. You know you're responsible for that, and the more that you can think about that up front instead of realizing you've made a mess of it and then trying to retrofit at all afterwards, you know, the better for you on the company because instead of you having to come back and do the work through times, you've just done it the first time as you were building this software. So, you know, the more engineers can have that mindset and that thought process, you know, the less wasteful it is for the company to build the software and the less wasteful it is for the company to manage and maintain the software. You know from all the other groups that are impacted. Yeah, I love that approach. So, shifting gears, I want to talk to you about my space, and we touched on this prior to the show and I'd love to hear your thoughts on you know how that experience was for you and you know, maybe time it to scaling and the topic of this podcast. was there anything that you learned there that you are now applying in your roll today? Well, I mean I think my space was interesting for me. I mean for it was my first web company, so I done primarily enterprise software until then, you know, running on desktops and servers, and so that was hard of what attracted me. The position is, let me go see what this other side looks like. I'm sure web development is totally different. So the first thing I learned is it's not. It's not even remotely that different. But yeah, I mean, I guess I would say that I actually learned a lot of what I do for scaling and, you know, team management prior to my space. And when I went into my space expecting everything to be different, and then even when I went to smaller companies and expected everything to be different, I found that a lot of the stuff I learned about trying to run an already large group, for going to ready large product ended up applying. It was interesting like some of the stuff that was missing from the process of my space, you know, where like Oh yeah, we don't have to do that, you know, was just skip that, and and yet almost every week they would trip over it, you know, and be a panic and you know, people like rolling back so software and whatever. So it was an interesting, certainly learning experience to see how my skills from my previous enterprise life applied to a different type of company. I Web Company and obviously my space is just grown at the time, so I think they had just grown about four hundred people and I just came from Microsoft, so that was a relatively small company to me. So it was interesting to see how all of that applied. And, you know, all the stuff that you know, smaller companies tend to look at as large company overhead done right is actually, you know, helps companies of every size, you know, streamline and avoid, you know, the problems that you and every start tripping over once you get past you know, very, very small proof of concept you because, I mean, if you're doing a website type of application, you know you can go from having a few small set of test users to having millions of users, you know, overnight, right, because there's no barrier to entry. Not's not like us where you have to...

...be in a certain markets, like anybody. You ask and go to your website and use it right, you know, and so like, for example, I deployed one change which worked great in development while I is at my space and they deployed it and it immediately ground the website to a halt because it, you know, because we're doing like sixteen million requests or something a minute, you know, and you as I mean nobody yelled at me. It's like, yeah, there was no way you could possibly simulate that right, and you know, we fixed it real quickly. But you know, that's the type of scale you have to be planning for because, you know, one day you're like you're just cutting corners and everything's great, and the next thing you got real users and real scale and now you've got to crapload of tucklet debt to deal with. You know, your database is falling over, your websites falling over, you know, and now you have to waste a lot of time and resources that you want to be using to build features and acquire new customers to just keep the site up. Yeah, interesting, right. So having that mentality just helps you, you know, grow the business in general. Yeah, I'm here in a common theme is making sure that it's set up right, you know, the first time in. Try not to cut any corners so that when you are preparing and when you are scaling, you don't right on into any challenges or maybe regret taking those short cuts. Yeah, well, I mean, I will clarify. I mean there's such a thing, as you know, what I call plan tuck that. And you know, when you are trying to move fast, you obviously you don't want to fall into where I'm going to plan everything perfectly for the next three years. You know, that's not really what I'm trying to say. You know, like get it right the first time so you never have to touch this again. You know that's not possible. It's not how you move fast as a company. But you can make intelligent, you know, shortcuts, which is different from just cutting corners without any real for thought. You know, you can choose to not implement something that you don't need yet. You know, you can choose to design something in a way where you know you're going to have to replace it later on and you've, say, abstracted things out in a way that makes that easier. You know. So there's a way to approach it that is better than just blindly, you know, powering ahead and building a mass that you have to completely delete or you end up, you know, patching for the next three years. But you know, again, that just comes down to being mindful and intelligent about your decisions and where you cut and how you chose to approach that. Yeah, okay, yeah, that makes a lot of sense. So what advice do you have for leaders at other high growth software companies? If there's, you know, one piece of advice that you could pass on to others listening today, what would it be? Well, I think pry what I've been repeating. I mean, and I also do, you know, engineering leader and mentoring as well, and this is probably the number one message that kind up trying to get across. As you know, you can't afford to be the...

...engineering leader or the engineering team that just sits around waiting for others to tell you what to do. You need to understand the business, you need to be part of the conversation. We need to understand not just the business and business model but all parts of the business, because everything, at the end of day, comes down to engineering and the software right. So the operations team, the support team, the sales team, all of those people depend on you building the right thing and building it well, you know, so they can operate it, they can support it and they can sell it right. And the more that you know you, as an engineering leader and as an engineering team, understand those things and understand those conversations and are part of those conversations you know, the more value you are bringing to the company and the more the company is going to be successful. Right. So that, to me, is the number one thing, as you really need to have your full business had on. You can't just wait for someone to tell you to implement feature x and Y, because that eventually will just break down. Yeah, understand the business. That's the topic of the session. So one last question for you then. Say, if I'm a leader and I want to help my team seem understand the business more, is there something tactical that I could do tomorrow to really enable my team to where their business had? Yeah, it depends on the size of the company, the Organization of the company. If you have, for example, of here separate product teams or you have like, you know, planning meetings or things like that, that's usually where I encourage, you know, allowing the team to attend those and be part of those and really hear the discussion. You know, a lot of engineering teams suffer from pass down effect, you know, where they just hear everything thirdhand and they don't really understand the conversation and the different tradeoffs and things. So any opportunity and give for your team to be a part of the discussion you know, like if if there's sales meetings, like they don't have to attend everyone, because that's not actually that's use of their time. But giving them the opportunity to and the option to attend those meetings or listening on those discussions, you know, really give them a better appreciation for the thought process and you know, hear how that those decisions of old. You know, hopefully, certainly the more senior team members can contribute to those discussions as well, you know, and represent engineering as far as making those priority decisions. And you know, what do we work on? How long will it take? You know, can we do it? Can we not do it? Things like that. So the more exposure you can give your team to that conversation the better. And even in the past, down where it doesn't make sense for them to directly and our face, you know, with the sales team or whatever, the product team, you know, making sure that they have the forums to you ask those questions and really understand, you know, don't let them just sit in a corner and code and, you...

...know, make them, you know, appreciate the bigger discussion and ask questions and you know, whatever else you can do to get them to understand what's going on around them. I love that tactical advice. Hey, Dan, really appreciate you join in application modernization today. Thanks for sharing your thoughts. Yeah, thanks for having me. Had A great time. Application modernization is sponsored by Red Hat, the world's leading provider of enterprise open source solutions, including high performing Linux, cloud, container and COUBERNETTI's technologies. Thanks for listening to application modernization, a podcast for high growth software companies. Don't forget to subscribe to the show on your favorite podcast player so you never miss an episode, and if you use apple podcasts, do us a favor and leave a quick writing by tapping the stars. Join US on the next episode to learn more about modernizing your infrastructure and applications for growth. Until next time,.

In-Stream Audio Search

NEW

Search across all episodes within this podcast

Episodes (29)