Our Head of Managed Services Timothy Fox was recently featured on Amazon Web Services’ This is My Architecture video series highlighting innovative architectural solutions on AWS. In this episode, Tim explains how we, an AWS Premier Consulting Partner built a flexible and scalable IoT solution to power our customer’s video streaming application. The AWS This is My Architecture video showcases how we used a mix of AWS services and data stores, including ELB, EC2, S3, RDS, DynamoDB, and Redshift, and the concept of “pods” to optimize the customer’s environments.
Video Transcription
David: Welcome to AWS This is My Architecture. I’m David and I’m joined here by Tim Fox, Head of Managed Services at CorpInfo-Onica. CorpInfo-Onica is an AWS Premier Consulting Partner. What do you guys do?
Tim: We specialize in helping companies get their stuff to the cloud specifically, on Amazon, and once they get it out there, we can also run it for them; because we’re also an AWS Managed Service Partner.
David: Alight, so a double threat, that’s awesome! You came here to talk today about an IoT device that delivers video to the cloud, what’s going on here?
Tim: We have a customer that has had some challenges in figuring out how to actually get their IoT device to work in a scalable way. So for them the big challenge was that they started off as a small company and as they rapidly grew, they ran into some challenges with how they actually built things.
David: Dangerous of scale, right, the traps and pitfalls? But we solved that in an interesting way. Why don’t you show us what you did?
Tim: When you think about it, you actually have a device that sits out on a wall, and its whole purpose is to be event driven and once an event occurs, it actually records that video. So the challenge is, as you get more and more devices, how do you actually scale that? At what point or what interactions do you have to define the scale points in that environment? The basic part of it is when we actually looked at it, for us we’re looking at what needs to be on a global scale and what can be more segmented to make it a little easier for that device to function and give you scalability. And the way we looked at it, was the concept of dividing those up into what we call parts.
The first part of that is on the global side, you have a service whose whole purpose is to figure out where that device lives or in more basic terms, you have do setup, user setup, and control that information as well as keep the event stream. The second part of that is you have to actually setup the device, when an event occurs, what do you do with it? Now you have a setup that actually controls all of the events with regard to recording the video and then once you actually have that video, doing something else. Once an event actually happens and you get video that’s coming in, the system is going to record the video on our service and when it records that video, it will store it in Amazon Simple Storage Service (S3).
David: Okay, that’s a great place to land data.
Tim: Right, so when you come into the database, your Amazon Elastic Compute Cloud (EC2) instances are actually taking the action to store that information about the events in Amazon’s Relational Database Service (RDS). Along with that, you need to understand how these events are showing and what are the key things that makes their problem unique. Each user can have one device, but if somebody wants to use it; say you have one and I have one and we both want to use it, now how do you get that information to your cell phone or computer? That becomes a different problem, right? Because now I have one video, stored on S3 but I need to be able to give everyone access to it.
David: So how do we do that?
Tim: So part of that is actually building on a global service or having something that understands where those users live. So when you come into the database, you then have of course your EC2 instances that direct that information and store the event stream in Amazon DynamoDB. So what happens is when you actually hit that front end, you’re actually pulling back a stream from DynamoDB of all of the events that are associated with you as a user. So even though you may be looking at somebody else’s device, your event stream still contains all of that information. And along with that event stream, we’re going to direct you to the specific point to go get your video.
David: So like pointers in the database?
Tim: Correct.
David: Okay, great and it’s blazing fast because it’s on Amazon DynamoDB, right?
Tim: That’s the advantage we get. And we can scale that event stream in a variety of different ways. So if you want to look at that information, let’s say, there are different event types that can occur and you can segment those and create specific tables for each of those event types. And maybe it becomes a little counter-intuitive but it’s not necessarily, you think about doing something like that on a referential database like RDS, but for the speed standpoint and the fact that this date is read very rarely, it sounds strange, but you spend more time writing information than you do actually reading it.
David: So DynamoDB is a great choice even though it’s not inherently relational?
Tim: Correct. If I was going to spend more time reading the information, there might be some other ways we architect it, but in terms of how we’re actually allocating and what we’re doing in DynamoDB, it’s heavy on the right side and lower on the read side.
David: So we can dial up the rights and call it all good, right?
Tim: Correct.
David: Awesome.
Tim: Once all of this is done though and you’ve actually collected the information, you have the event, you recorded the video, and you’ve recorded maybe some interactions that the users have with that video, maybe you’ve understood how often they viewed it or anything else, you then want to be able to report on that information. So as a company I need to be able to collect that information and have a method by which I can actually see what occurs. So from that standpoint, you’re then taking the information out of RDS about the users and everything else, you’re also taking the event streams and you’re shoving them into Amazon Redshift, so now you can actually have reporting on it.
David: So advanced user interactions and we’ve got all the metadata, put it in Redshift and magic happens!
Tim: Right.
David: Awesome.
Tim: That’s the whole intent.
David: Well, this is a really great solution and I really find it interesting, how you are able to go relational without being relational. Thanks for coming out Tim.
Tim: No problem.
David: Thanks for watching This is My Architecture.
Contact us to jump start your big data and IoT architecture!
