Scaling Applications to Meet Increasing Customer Demand w/ Ahmed Farag

ABOUT THIS EPISODE

In this episode, we profile how FaceGraph, led by visionary founder & Chief Executive Officer, Ahmed Farag, is helping schools, offices and retailers use AI and ML to engage customers in new and powerful ways.

Join Ross Beard and Ahmed Farag as they discuss:

  • Overcoming challenges in scaling Facegraph’s application
  • Adapting quickly to meet new pandemic-related demands
  • Leveraging autoscaling to improve application performance

Check out these resources we mentioned during the podcast:

Application Modernization is a show where Software Leaders discuss the challenges and successes in modernizing applications and infrastructure at high-growth software companies. Learn more on Apple Podcasts, Spotify, and the Shadow-Soft website.

Listening on a desktop & can’t see the links? Just search for Application Modernization in your favorite podcast player.

You are listening to applicationmodernization, a show that spotlights the forward thinking, leaders of Higosoftware companies from scaling applications and accelerating time tomarket, to avoiding expensive license and costs. we discuss how you caninnovate with new technology and forward thinking processes and savesome cash in the process. Let's get into it today, we're talking with AmeFarag about scaling applications to meet increasing customer demand. Ahmadis founder in co of Facegear, a company that helps schools, officers andretailers, use artificial intelligence and machine learning to engagecustomers in new and powerful ways in this episode will talk about three bigscaling challenges that are made in his team ran into, as they grew from threecustomers to five hundred customers. This is a really interesting story withsome great takeaways, I'm so excited to share it with you to day. Here we gowith our guests. Armed Farag, Hi Amad. Welcome to the show. How are you todayI'm doing good? How are you good good today we're going to be speaking to youabout scaling, applications and infrastructure? Can you tell us alittle bit about yourself and what you do? Yeah, so my name is Ametron, am asoftware engineer and I start my career in two thousand and four buildingapplications in my my really love for software or started. When I startedwriting a little bit of you know, Games at the time we had the similar phonesthat had this game snake. So I used to try to build those games and it waskind of interesting how you can build a moving object for me, like I used towrite programs that basically do like calculations and stuff, but just to dographics, that what really got me into...

...the world of programming- and I didthis with mostly Microsoft tools. So I was a Microsoft donat guy from thebeginning and then I work for a few companies Inter Natale, so I'moriginally from Egypt. I worked in Egypt in the Middle East a little bitthen. I worked in the UK, Netherlands and I got married to my wife, who'soriginally Egyptian, but she was born in Chicago and we moved to the US withher family over there, and I worked in most recently in Microsoft andthroughout the the years in two thousand and ten, I founded a companycalled Software Ranger that does software consulting and in two thousanda D and seven two thousand and seventeen I'm sorry we started facegraph and face graft. Is You know my last venture, which is focused on aThenar yes face craft is a very interesting story? Can you talk usthrough what face craft does and what problem you are solving for customers,yeah, so face graph is focused on providing you know a platform that willhelp solutions interact with humans better, and this is very complex, likepeople will think of like what does that mean? Well it. Basically, the bestmaterialization of this idea was in one of the products that face graphproduced called smile me in which does attend as tracking what we've done is.We've taken an open source platform called identity server for it's,basically, an identity system that was created by a group of Engineers, itsopen source on GITTAP, and it does modern authentication it. So it does aoff an open ID which is in the last, I would say, six or seven years is thethe way to do authentication in anything like face book, even Microsoft.So we customized that that platform and we built face graph on top of it, whichallows you to do things like you know, issue and a loft token with facialrecognition, so we sort of merged ai in...

...things the advances then Ai, that makesyou know, machines understand human, better. You know, which is facial,recognition, voice, recognition and so on. We made it a part of the the OOFFequisite and that helps us build a robust system, for example for F. Youknow we call it, you know smile and that helps a customer do attendancetracking. So if you go to an organization today, you step in infront of a machine, it would recognize you automatically, so you don't have topress even in a button that says. Oh, I'm here take my picture. I will justbasically recognize you're here and then I'll do official recognition whenit does that it will recognize who you are, what time you came and if you arein a school dropping off a kid, it will tell you: Oh you have two kids, whichone you're dropping off and so on. So we made that process. Seamlessinteresting. Can you distinguish between two people? That might be twinswell. This is a very that's, a very good question. Actually that's aproblem in facial recognition that hasn't been solved. So, if we'retalking about identical twins, it's very, very diffi cult and when applehas produced face ID first in the market, there were like a couple ofcompanies. I think one of them in Russia that was trying to show how theycan break it as well as when Microsoft did the you know the facial recognitionin the calle windows, hello, when the hello and windows there was some. Soyou know, companies have shown how you can have a twin that basically unlocksthe machine or apple with you know. If your identical see the problem isfacial recognition, that's successful today, which we call practical, meaningit would. Ninety nine percent give you a good match, usually works. It has twocomponents: it has the detection component, which is essentially yougive me a picture. I need to detect where the face in the image, so thatuses machine learning and then the other part of it is just a normalsearch problem. where, once you do the Texan one of the things you do is youtry to extract the features of the face? So where is the location of the eye thesize of the eye, the islets, the nose the mouth to all of those? So the morefeatures you gather, the better your...

...agrin for for matching- will be foridentical twin those are identical. So it's a it's not easy. So in what wecall high high securities scenarios, let's say: unlocking doors opening bankaccounts. We normally require a second factor, authentication, much likehaving you sign in with a using em a password. We would actually send you atext and then you would have to enter a verification code. This way, if yourtwin brother is trying to steal your money, they have to have your fault.That's really interesting and a matter of fact. I am an identical twin, somaybe you could test use us as testers in the future. Anyway, it sounds like you're lookingat lots of data as you're growing. How are you managing all of this data andhow you scaling Your Platform Yeah? I know this is a good question, so I meanjust to bring you know the audience this peon on the sides of Dhata we'redealing with. So currently we have about twenty two sand. You know activeusers. Most of those users will be using our system between six a m in themorning Pacific Time, all the way to around ten a m in the morning specifictime because most of our customers are here in in the: U S, Canada and Mexico,so we're working on three time zones and all of them come in the morning,use our system and then most of them go in the afternoon around. I would sayfor three PM: Pacific all the way to to six or seven right, so we normallyreceive about a million request during that time. So that start, that's ourpeak and our requests are very large because we receive images right, sowe're not really getting just usually in a password. We have a very bigvolume of data that comes in in the morning and you know obviously comes inin the afternoon, and the challenge that we have is we have to do ourresponse under a second, so our promises when you stand in front of themachine you try to get something done. Whether you know today is the tennistracking. It could be also temperature...

...scanning, because we've added thisfeature for Corona, you don't want to be waiting for a minute to get aresponse. So we have this obligation to make this work really fast and then Ado,this the obligation of security and safety, where we have to do a lot ofscraping for data and so on on this spot. So for for scalable, when westarted, we had three customers. Now we have about five hundred or so customers.We work with businesses directly, not consumer, so we run into problems ofscale, and more importantly, is how can we, you know, have responses, goquickly to customers? So S, you know basically scale on responses per second,and then this, the other skill problem is actually storage, because we, someof our customers, need to have their donaster for up to ten years, and a lotof those will keep records as well as images interesting, so, specificallywith storage. How are you solving for this problem yeah? So what were youusing today, since you brought up stories so we're using three clouds?Today, mainly, everything is hosted on Microsoft, Azure cloud, includingstorage, but we still use Google cloud, for you know, push Enif cations to ourabsent devices. That would be fire based and we're using ALBS FOR AI andfor the devos agents. So for storage we're using, we have two types ofstorage, binary files and those will be stored, essentially on asure storage asfiles. But then everything else is starting our database, in that casewe're using Microsoft, cosmos, dva, no sequel database- and you know this justbrings me to you- know what other services were hosting our system on. Soessentially when we started we didn't want to, we were really keen to launchand we built our system very simple. A number of web applications and WebServices and those web services were hosted on Azure AP service. You know wehandle most oft our requests...

...synchronously, so you receive therequest in you. Then you receive the response and we leverage auto scalingand APP service. So when we have, you know, for example, a certain number ofrequests coming that goes up or c Pu or memory contention. We would have itautomatically roll out additional instances of OB service and we wereusing a service premium. Interesting so mean we're hearing a lot of customerswanting to embrace cabinets and move to containers, and it sounds like ifyou've taken a different direction. You know our audience would be reallyinteresting to hear sort of. Why did you go that route? Is there reasons whyyou are not using cubieres yeah? I know that that's a great question and wealways thought about that. We always thought about. When are we going to?You know more specifically run MICROSEC ES instead of having US run thosebigger monolithic services? How can we decoupee our system? So it's morescalable- and you know, like I said in the beginning or our goal was to launch,and it was really easy for us to launch directly. You know having those webservices available. We only have three customers and we've gone from. You knowthree customers to five hundred and specifically, ironically, during corona,you know in March, when things went went down, everybody was not going toschool or work. We were obviously also impacted, but what helped us when wepartner with a company in Hong Kong and we were able to customize the advicethat will help customers go back to work in school, so we were able to dolike temperature measurement, with attendant tracking and so on, so thatactually created an explosive growth that we haven't planned for. So westarted getting problems and, more specifically, because you buildingsystems on the cloud transient problems or really common, you may have adatabase not available for the second that you need it. So that's when werealize realized, that's the time that we should think about, maybe continereour system. So the way we built our services. You know those courseservices that I talked about as well as...

...the non essential services. Most ofthem are built in done at core, so they can run of Linux and windows. So if wedecide to host them on a container that shouldn't be a problem. However, so ourgoal was not to really use containers. Our goal was to break down the services,make them be coupled as much as possible, so we can provide morescalable Y and what we've decided to do is we used the Microsoft as ourfunctions and as your functions gives you that ability it gives you to havethe ability to have microsec, but one of the you know again. This is for theaudience. This is more similar to the Landa architecture. So you can do thisin other cloud platforms as well, but since we've been invested in Microsoftclouds we were focused on using Microsoft Asaftida to implement microservices. But you know the interesting thing that I think Microsoft has donevery well. That really benefited us is they have? You know a solution to oneof the problems that you may face when you use containers- or you know, youbuild micro services, which is essentially you have to think of makingyour services stateless as much as possible. If they are stateful, thenyou have to worry about storage and making sure that you know, as you,services being spun up and down you're, not losing your storage. So one of thethings we looked at is using Azenian with durable entities, and this issomething that they've released myself release. I think about three or fouryears ago. So durable entities gives you, you know few benefits the mostimportant benefit. is they have the ability where you can create thatentity, so that entities essentially a type you can define to any class? Anyyou know type you define our new code can become a construct or the objectsdefined out of that class can be serialized automatically for you. Sothat would be the state and the platform as your function will takecare of storing that state for you and...

...it will re hydrate it for you when yourservice is up. So when your function is running, so it takes care of having twothings breaking down your services into smaller pieces, but it also has theability to make it really easy for a developer who doesn't know so a lotabout micro services to just you know, adding a few attributes in UC sharpcode. You can essentially just enable some of your types to be those entitiesthat will be you know, stored for you and they will be permanently stored inaddress store, so they're not really ephemeral. They are actually stored inactual stories that doesn't go away interesting. So so it sounds like youcan achieve the scale ability. You know that that you're needing withoutnecessarily you know needing to embrace Cuban etis. Yes exactly and so fewexamples that maybe make this clear. We have a lot of things that run on theside that don't require an immediate response to csomers. Those are thethings that we were able to break down, bring them up as services. We've alsoused, obviously, Microsoft Service bus. So we used you know an enterprise Q, sowe switch from receiving services directly and doing responses directlyto having an enterprise bus that will receive you know. The requests fromcustomers in that case would be attendants, request or authenticationrequests, and then we have a number of workers. Those are a ER functions thatwill jump in and pick up those requests in process them. So we do a lot ofthings in processing. You know today other than just authentication, soauthentication that would just come, maybe in the middle of the process.After we've identified the person we've verified that he's really who he claims.He is. You know there are things that we, you know do things like. Oh, let'scan the face and see if they're, using a mask or not, and those are thingsthat have been yeah- I mean it's like you know some of the customers calledthis and said: Hey Look, you have a custom device, we like it, but we weactually want to know. If, if the...

...people who came into the Office are,you are wearing a mask so adding those extra processing on top of the servicesthat we do will definitely mean delays the customers so with the ability to have you knoweverything broken down into ques and then having different types of workers,jump in and do different types of processing, we were able to add thosefunctions to come in and do, for example, use vision, Apis to doanalysis and see, there's a mask or not. So the flow would be if there is nomask, we would send an alert which will be another cue that we will omit to andthat would send text messages and we use tolio for text messages today andemail and Pusitao so breaking down those external services or not veryimmediate important services in attendance into functions. Thosesmaller functions have, you know, help us with scales, so can increase themwhen we need when we have load and then they are completely de coupled. Ifthere is a failure reaching a database, then we retry until the message isdcted and everything is good. So this way we can, you know we can prevent.You know we have a durable sustainable system even in high load scenarios, oneof the biggest problems we faced when we had you know bigger customers. So wework with the number of school districts. Those tend to have a lot ofusers, a large number of users, and then that means a large number ofdevices, so some of them may have like up to eighty or ninety devices spreadout physically to different locations. So it's very hard for an ad man to go,walk to each device and see what's going on if there is a problem, so wecreated the device management and this really falls under kind of it. So allof those devices are connected, the Internet. All of them are reportingback. You know the attendance, but if one device is out of date, so we pushfor more updates constantly through our apis because we do add additionalfunctionality or we do patching and so on. So it's not possible to gophysically to each device and do that.

So what we've done? We have two thingsimplemented: one is I'm online, so each device will pin our server every minutejust to announce that online and this way, if a device is off line for somereason because of power outage, somebody has misplaced the device forit's. I it's broken. You can see that on the Dash Board. The problem withthis is: We have tons of requests coming in just to say I'm online andthat used to bombard ar data bays our services. So that was a good example ofwhere we use the durable function, so we created an entity that entity eachentity instance represents the device and it would store the things so notonly. It starts the last thing like we used to do in our database. Now westore the last things for the last thirty days and all of that is storeoutside the database, so because that information is not important. If welose it, we don't care in customer doesn't care, it's not part of their.You know their data and in also we don't really do back up under store forthis kind of storage. It's more of a temporary storage for us, but it'sstill, you know, stored permanently for the last three days. The good thingabout this is we alleviated the load problem and you know number two,because customer uses are use our devices, usually using wife, it's verycommon that we receive a support. Call that says hey this device is notsending that out. I don't know what's going on with it and we can easily lookat the last three days pings and we tell them hey. Look. We see that devicewas disconnected from this time to this time, so you may have a wife I problemand then they can resolve it easily. It was really hard for us to tell them hey.It might be why I it might be our service was down. So it's really hardto tollest. When we don't have data so now having those entities, it made iteasier for us to help customers find conductivity problems, for exampleright. That's, that's really interesting! So when I'm thinking about that, you knowI'm thinking about. Perhaps yes, some...

...of these problems that might come up.Can you talk us through your instrumentation and what sort of othertools are you using to monitor and manage your platform? Yeah? Soinstrumentation is key. I think it's one of the things we've done well inthe system compared to other parts. Instrumentation is important for oursystem because we have a number of rest apis. We have an identity system andthen we have an attendance tracking system and then we have three differentAPPS. Those are the personal apps that you can download on the APP store thatyou can use as a user to see your attendants, maybe do attendance fromyour phone and all we also have devices. So it's not uncommon that you start arequest. Let's say one device where you get your pictures canned or you use anFID card, and then that goes to the system and gets recorded. And then youknow you get a notification on your phone that says you've been signed inso because in operation can cross all these systems. You know, starting fromyour own device, to maybe sometimes to our device all the way to the servicesand sometimes going to a third party like told you to send a message when wehave a problem, it's very challenging to understand where the problem is sofirst off what we're, using mainly for instrumentation today as a servicewe're using Microsoft as your appenines, so appen sites is the monitoring toolinstrumentation for applications that runs in Microsoft, Azure and it doesintegrate very well with APP services. So with apposed an AP service, it doeshappen, SD that runs with different types of platform. So there's an STKfor Angler. There is an Stk for dot net. There is one for Ph and so on, Java andso on. So it's very easy for us to plug it in, and that would give you thebasic, so that would give you essentially locking for all webrequests that you were receiving. So you can see when the request happen.What I Paris is coming from, what time...

...it happened and so on so what we had todo. On top of it, we had to use the APPIN SIDES API to add a correlation IDthat we've added in every operation that we create. So this way we canprovide this ability to trace any operation from device all the way tothe APIS. So today, when we have a severity, one exception, so basicallyan exception. That's not handled that was raised by the system. Ourengineering team gets notified by email and we get that correlation ID that wecan go and use. You know appiness, which has a queerer language thatallows you to query that correlation ID, and then it would show you all thesedifferent calls that was involved. You know this is a B, a podcast. If we canshow things, I would show you that it would show you a graph that basicallythey call it and to end trouble, shooting that's a feature in appenineswhere you can see that request started from this service which runs on thephone and then, if it's gone to your API at this time, and then it calledall these dependencies. So I can call, for example, a database which is Cosmosdeb in that case, and I then it calls storage. So if failure happen instorage that bubbled up to you know, the colic went to the database thatbubbled up all the way to the attendants and then all the way to thedevice. It appears to be a device problem. But if you know you know, whenyou dig down using that corelation ID, you can see it's a storage problem, sothat really helps us tremendously to be able to respond. You know to problemsand do trouble shooting faster interesting. So what are your thoughtson to some of these other application performance? Monitoring tools, likeyour dinora es or you know I know observability- is now what theirobservability platforms is a lot of the language. I'm hearing like how does sayAPP insites compared to those tools and platforms, yeah now Dine Tras, we it'sgreat. I mean I think there are a number of platforms out there, thatthey do very similar things. They do Montrin, they do instrumentation, Ithink all of them. They will give you...

...the bare minimum for tracing, but youalways find that you, depending on what you're building you always have toembed instrumentation in your day today development. So you always have to addextra things that would allow you to provide better trace ability, and thenyou augment what you get from that platform and make trouble shootingeasier. You know the reason we chose appin sights is because of cost. So youknow it is very cost effective. You know compared to Dynatai, I think forour size that made sense. We definitely look at other tools and we had requestsfrom developers where they wanted to use other easier tools, but because,like I said it was sort of easy for us to integrate it with our stack and then we had good implementation with it. Itwas working just fine, I mean the other thing you may think about is how canyou maintain availability, because they I mean availability instrumentation?You know they kind of work together. If you have an outed, then that might becau like unplaned outage. Obviously, then that might be caused by a problem.Then you want to trouble shoe using your instrumentation tools, monitoringand so on. So what we liked about Athen sites is, does have that ability to soit has availability monitoring, so you can set it to paying your end pointsand give it. You know certain rules that will check on and it does thatevery minute and if, if you have an out it, it would call a phone number, itwould also send an email and send the text. So combining these together inone platform was very important, for us is very user friendly. You can set itup from the Agur portal and there is not really a huge cost for us to do toreceive those alerts. Yeah Yeah, I e. It makes a lot of sense as you're sortof just starting out to use some of those native services with Azure. Soyou know great insights here. I think you know. Leaders at high gro softwarecompanies are getting a lot out of this.

What other advice do you have forleaders that we haven't touched on? Yet I think, if, if you're building one ofthe things that we learned during the last couple of years, if your solutionconsists of hardware and software, you know what helped us a lot is we youknow coming in as a software company. Originally we always think from asoftware perspective and we try to even convert those hardware problems intosoftware as much as possible. So a good example is you know we have thisthermal unit which it's a terminal. It's a ten point, two inch screen thatyou can place in an organization that allows you to do a tennis tracking andin order to do testing for the firm wore that runs on that particulardevice and then do releases for that particular device. It'd be really hardfor us to use, like you know, have units with testers and then also haveunits with the support engineers and then use like manual methods to dotesting as well as support. So what we've done is we created the softwaresimulator that would simulate using the device which basically help us,obviously with testing. So you don't really have to have a device with youeverywhere, especially when offices were closed down, everybody was workingremote. It was really challenging to make those things available. So I meanmy advice: is you know, try to solve things with software as much aspossible, I think consuming or maybe spending or investing time on test.Automation is also very key, so we were really. We had a great test engineerthat does you know the test, automation packages for us and we were literallytesting everything with code. I mean this is a side from unit test, so youknow. Obviously you write your gun. A test is a developer that will help youwith the devos. You know makes the process of validating problems earlyquicker and then you can do releases,...

...but then that doesn't replace thesautomation. So, just to give you a quick example, we win, so phones arereally really advanced these days. They take very, very big nice images and weone of the things we provide to early education. Schools is taking picturesand share them with parents, so sometimes they would take very largepictures and some of these platforms, you know, have the picture a stored,not in the same orientation that it was taking. So I be flipped, so we have ourone of our test. Automation scripts would be, it will automate creatingpictures on devices that are connected to a machine and those are the agentsthat we build. So one of one of the great things I like about Amazon whipservices. They made the ability to create virtual MACimages. So this way I can, I don't really have to buy all these MACmachines to build. You know an is APP or do do test for ISS. I just basicallygo spin up a dedicated host in Amazon, and that would have my test images thatruns Mac. That runs macos, so we would run you know, automation that Scans theimages and just decides. If the image is slipped or not, so we were able toautomate. You know the testing for all of those scenarios just using scripts.That really provides US provides with the agility to be able to do quickfixes and releases yeah yeah. I like that sole things w with softwarewherever possible. So I mean what's the vision for Facci we've heard about yourstory, we've heard about the last eighteen months and a little bit of thepivot to serving needs for your customers around some of thetemperature checking with with coved. What's the plan for the next two orthree years, where you see the company going so the long term vision is to tryto augment devices that existing in organizations- and you know software tohelp them with those front door activities a you know those would besigning in a visitor or doing...

...attendants, or maybe recognizingcustomers like in the commercial space. So if we look at the breakdown of ourcustomers today we have about thirty one percent, our offices. So basicallythat could be. You know a software office. It could be a company that doesaccounting or whatever any office kind of job and then twenty five percent ourschools, and then we have a good portion that our factories and thenthirteen percent are support facilities. So the market we trying to break intoin help is basically commercial market. You know essentially stores retailstores, you know cafes and restaurants, and what we think that can be veryhelpful is a product that we're working on W, which we call a face graph in abox so face graphing. A box is essentially going to be a smallcomputing unit that you can plug into your power in a company, and it wouldhave wife that can connect to your on premise network and it can be attachedto the cameras on the premise. So what it does it basically is a smallprocessing units that will take whatever you allow it to take frominformation. Let's, let's say you have cameras on premise which a lot ofcompanies like retail source, have cameras so I'll, take that camera feedand then it would analyze it for any important information like, for example,frames that has human faces in that case. Obviously, that's what we careabout and then it will only send those frames to your back and instance offace graph and then what that does to you is this going to tell you hey. LookCustomer X, Sam has walked into the office or walked into the retail storeand he likes this drink, so it may pop up in their starbucks at that hey sam.We see you here, you want to get your late, you know so the way we thinkabout this is it we think it's not going to work. If we tell customers say,look in still all these cameras put our devices, that's going to be too much toinvasive, but if we just put it in a...

...box, you just plug it in connects yourexisting hardware and then connect your software tack. That would make moresense. That's really interesting men. We work with a number of sort ofretailers and fast food companies that are, I think, on that track. You knowthey're investing in some of the infrastructure and the technology rightnow with that type of personalization as the vision- I remember you know fiveten years ago in some of you know my marketing classes, we would speak tothat as the vision right. You walk into a store that someone already knows yourname and they already know what you want or end or need. So it's that's,really cool vision and I will certainly be be following a journey awesome. Iman. It's been a absolute pleasure hearing about your journey and hearingsome of these insights thanks for joins today, Hey Ros. I appreciate the timeand thank you for having me application. Modernization is sponsoredby Red Hat the world's leading provider of enterprise, open source solutions,including high performing Linux cloud container and Cuban ttes technologies. Thanks for listening to application,modernization, a podcast for high growth software companies, don't forgetto subscribe to the show on your favorite podcast player. So you nevermiss an episode and if you use apple podcast, do us a favor and leave aquick rating by tapping. The stars join us on the next episode to learnmore about modernizing your infrastructure and applications forgrowth until next time.

In-Stream Audio Search

NEW

Search across all episodes within this podcast

Episodes (13)