New Domain 2 - Design for New Solutions

18. Streaming with AWS Kinesis

Hey everyone, and welcome back. Now in today's video we will be discussing the AWS kinases service. AWS Kindnesses is basically a set of services which make it easy to work with setups of training data in AWS. Now we have already seen what streaming data is all about. So what Kindnesses provides is that it provides a variety of services specifically for streaming data, and each of them is tuned for a specific use case. So there are four primary types. One is the kinase data stream, the second is the kinase data fire hose. You have data analytics and a video stream. So the kinase data stream is basically intended to capture, process, and store the data streams in real time. Then there's the kindness. Data. Firehose Now the firehose basically allows us to capture and deliver data to a data store in real time. So the primary aim of the data fire hose is to move the data from point A to point B.

Then you have the kinases data analytics, which basically allows us to analyse the streaming data in real time with SQL or Java code. And the last one is Kinesis videostream, which basically allows us to capture, process, and store the video stream. So, if you have streams coming from a device such as a camera, So the camera generally runs twenty-four seven and if you have a lot of streaming data there, that kind of data can be sent to the kinases video stream and from the video stream you will be able to see the live feed of the device. There are three important entities to remember when it comes to kindness. So this will basically help you understand things faster. The first is the producer, the second is the Streamstore, and the third is the consumer.

So producers basically produce the data and send it to a stream store. So the stream store is basically where the data would be sent and the data would be stored from the stream store. You have the consumers who would be taking the data from here and would be doing some kind of analytics or some kind of storage there. So this stream store that we see over here is what the Kindnesses fit into. So let me quickly show you kindnesses from the AWS console. So I'm in my AWS Management console and let's open up Kindnesses here. So when you click on Get started, you will see that there are four options available over here. So one is the simple Kindness stream, then you have the Kindness firehose, then you have the Kindness analytics shere, and then you have the Kindness video streams. Each of them has its own usecase, but one thing they all have in common is that they deal with streaming data. So this is the high-level overview of the Kindnesses service. I hope this video has been informative for you and I look forward to seeing you in the next video.

19. Overview of Kinesis Data Streams

Hey everyone, and welcome back. In today's video we will be discussing at a very high level the kinase data stream. So the kinase data stream basically allows us to capture, process, and store the data streams here. So let's understand the overall architecture here. So on the left hand side, first you have the inputs. So these inputs are basically the consumers who will be sending the data. Then you have the Kindnesses data stream over here. So the data stream is where the data from the consumers will be stored. Now, do remember that the data will not be stored for an unlimited amount of time. So there is a predefined time for which the data will remain present within the data stream. Now from this point, you will have the consumers. So consumers can be EMR. It can be yourEC, for instance. It can even be the dataanalytics service from Kindnesses and various others. So the consumers will take the data, whichever is being stored in the data stream. It will go ahead and process that data, and then the report will be generated. So this is the overall architecture of the kinase data stream here. So before we go ahead and do analytical on this, let's quickly look into how we can create our first data stream. So I'm in my Kindnesses dashboard,so let's click to get started here.

So there are four types of streams here. So you have the data stream, you have the delivery stream, you have the analytics stream, and you have the video stream. So let's click on "create data stream." So the first thing that you would have to give is the name of the stream. So I'll call it Kplab's Hype and stream it here. Now, if you look into the diagram, this is very similar to what we have been discussing. You have the producers here, and you have the kindnessesdata stream where whatever data which has been generated by the producers will be stored. And then you have the consumers who will consume the data and then the appropriate report or storage will be created.

Now you also have the concept of a shard. Shard is something that we'll be discussing in the next video where we'll be doing the practical session on how you can put the data as well as how you can retrieve the data from the data stream and also understand various concepts here. So basically, in a nutshell, the more shards that you might have from multiple consumers, the more shards that will be needed. So let's quickly do one in terms of the number of shards. And if you notice over here, as soon as we put one, it automatically showed us the amount of capacity that can be achieved with the number of charts. So if I just increased the number of shards,the amount of capacity would also be increased.

So again, if you have a large number of producers who are writing huge amounts of data, then the number of charts that will be required will be greater. Anyways, we'll be discussing that in the next video. So you select the shard right here and you go ahead and create a data stream. So it takes a little amount of time for the data stream to be created. So if you see the status is created, So, once the data stream is created, you can go ahead and insert data into the data stream. You can even go ahead and pull the data from that specific kindness data stream. So let's do one thing. We'll conclude the video here. And in the next video, we'll look into how exactly we can put the data, how exactly can we pull the data, and also understand the concept of shard in practice. So with this, we'll conclude this video. I hope this video has been informative for you and I look forward to seeing you in the next video.

20. Configuring our first stream with Kinesis

Hey everyone, and welcome back to the Knowledge Portal video series. In the earlier lecture, we were speaking about the basics of what streaming data is all about. We also look into how databases cannot work when you talk about streaming data. And there are some different kinds of tools which will be required in order to handle that type of data. So in today's lecture, what we'll do is take an overview of one such tool called AWSKinases, which will be used for handling streaming data. So let's go ahead and understand more about it. AWS Kinasis is a managed platform that allows us to load and analyse streaming data on a real-time basis.

Now this is a definitive kind of analysis, and we will definitely understand more when we do the practical related to AWS Kindnesses. But just remember that there are three major types of features which are available. One is through a product called AWS KindnessStreams, which is basically used to collect and process large streams of data in real time. So this is the basic kind of stream which is available. The second is called a Kindness Firehose, which is basically used for delivering streaming data to a destination like S Three or Redshift. So if you want to store that streaming data in some kind of storage device like S Three or an analytical device like Redshift, this is where you use this. And third is Kinases Analytics, which basically allows us to run certain kinds of analytics in-streaming data with the help of standard SQL queries. So just remember these three and we'll understand more about them in the practical sessions. So again, going back to the slide that we had discussed in the earlier lecture, when you talk about streaming data, there are three entities which are generally involved. One is the producer. The producer will produce the streaming data and that producer will send that streaming data to a stream store. So this is a stream store and then the consumer will retrieve the data from this specific stream store.

This stream store becomes very mission critical middleware because if this stream store goes down, then your entire website's functioning will be affected. So, having this be very stable is very important. This is where AWS Kindnesses comes in handy. If you see, AWS Kindnesses is a managed platform, so you don't really have to worry about this middleware going down. So all of these are managed by AWS. great. So this is the basis of the theoretical part. Let's go ahead to the AWS management console and see how it really works. So this is the management console and if you just type "Kindnesses" you see the description work with real streaming data. So just open this up and here you'll see there are three applications which are available. One is the Kinases Streams, which is basically used for processing the streaming data.

This is something that we will be looking at. Second is Firehose, which basically does a lot of things like capture, transform, and store your data in AWS's three or other locations. And third is the analytics, which is basically used to run SQL queries on the streaming data that really comes. So we will be focusing on the Kinesis streams for now. So let's go to the kinases stream console and here you will see I don't really have any streams. So the first thing that we have to do is to create this middleware. So this middleware is called a "stream store" and this is the reason why the first thing that we'll be doing is creating a tiny stream.

So I'll click over here and it'll ask me for a stream name. So, let me put it this way: KP Labs 'dreams' okay. We'll be talking about shading in the relevant section,but just remember, just put shard as one for the time being and click on Create Kindness stream. Perfect. So the kindness stream is being created. So if you'll see over here, the status is getting created. So it takes some amount of time for the kindness stream to be created. And once it gets created, then we can go ahead and understand how we can push data to the kinases stream and then we can pull data from the consumer. So we'll be looking at how we can do that in the next lecture. However, for the time being, go ahead and create a kinetic stream from your end as well. So if you see the kinase stream is created, the number of strands is one, and the status is active. Perfect. So since our Kinahs stream is created in the next lecture, we'll go ahead and understand and do more practical related to this particular topic. So that concludes this lecture, and I hope to see you in the next one.

21. AWS Kinesis – Practical

Hey everyone, and welcome back to the Knowledge Pool video series. Now, in the earlier lecture, we discussed the basics of AWS kindnesses and we also created our first kindness stream. So in today's lecture, we'll go ahead and look into how we can put some data in this specific stream and also how we can retrieve the data from this specific stream. So, before we do that, let me show you one important thing which will help you understand this much better way.So, we already understood the producer,the stream, and the consumer's way. In a PowerPoint presentation, if you remember, you have a producer, you have a stream store, and you have a consumer over here. Now, there is a very important concept called "shards." So, the basic concept of shard is like we discussed the Uber application. What happens if all of the producers, which are the drivers, send data in a single stream around the world? And then, if everyone is sending data in a single stream, what would really happen is that there would be too much clumsiness that will come.

So, in order to reduce that, what concept is introduced in kindnesses called a sharding concept? So what really happens is that within the kindnessstream that we create, we create a substream. So you can consider this a substream. And each substream can be used for a specific purpose. So you have a shard one. So, shard one can be used for all the data related to the drivers of a specific area. In Bangalore, you have shard two. Shard two can be used in some specific areas, including two in Bangalore. So, if there are ten regions in Bangalore, you can have ten shards associated with that specific region. So, what really happens is that the data is actually in a very organised way and this will really help to segregate things. Now, one important thing to remember here is that each chart that you create has a specific throughput capacity. So it is already mentioned that each chart can ingest up to one MB per second and a thousand records per second and emit up to two MB per second. So there are certain limits related to the throughput of the shards that you create. We had now created a stream of shards as one. So we just wanted to keep things simple.

By default, Amazon will give you a maximum of 500 shards, which you can increase later. Perfect. So let's do one thing. Let's go ahead and implement the practical. So, what I have done for our case is that I have created a user. In fact, the user was already created in our earlier practical session. So the user is called Steven. And if you see over here, Steven has kind access and I have downloaded the security credentials. So this is the access secret key and I have configured the access and secret key on a Linux box. Perfect. So let's go ahead and verify our kindness stream with the AWS CLI. So you will see over here that there is one stream that you will find, which is the KP Labs hyper stream. So this is a stream that we created. Now, in order for us to do things quickly, I have created a small document which I'll be sharing in the forum itself so that everyone can look into the commands. So the first command is kindness liststream.

So this will list the streams that are created. Now in the second command, what we are doing is actually putting some data in the kinesis stream. So the command is "AWS kinases put record." We've put the stream name, which is KP Labstreams. We put the partition key ID and we also put the data, which is hello from Kplab. So let me just copy it and I'll paste it over here. I'll just update kindness streams and let's see. Perfect. So it gave us a response. So as soon as we ingested the data in the kindness cream, it gave us two responses. One is the Shard ID. So this is the Shard ID and the second is the sequence number. Now, since we have only one shard, this is not a big issue, but if we have multiple shards, then remembering Shard ID is very important because if you do not remember in which shard your data is stored, it is difficult to retrieve the data back. Okay, so that is the shard ID. The second is the sequence number. A sequence number is unique to a specific data record that we have ingested in a tiny stream. Just remember that. So let's do one thing. Let's play one more record. I'll say second time. Now in the second record, after this record gets ingested, you will see that the ShardID remains the same since we have only one Shard ID, but the sequence number changes. So this ends at 00:26 and this ends with 890. So the sequence number is unique to a specific record that you ingest in a Kinasis stream. Perfect. So now we have ingested two records in the Kinasis stream. So the next thing that we'll be doing is we'll be focusing on how we can retrieve the data from the kinase stream. So we have already put the data in the kinase stream. Now we'll focus on how to retrieve it. Now, in order to retrieve the data from the akindness stream, you have to get a Shard iterator. Now, shards and iterators are very important. Now in the Shard Iterator command, if you see, we have AWS kindness, we get Shard iterator and we specify the Shard ID over here.

So this is the Shard ID, which is the one that we receive after we ingest the data. You see, we got the shard ID. We pass that Shard ID and we have some arguments, which is the trim horizontal. We'll be discussing this in the relevant section about this followed by the stream name. So I'll copy this command and let's paste it. I'll just press enter perfect.And upon execution of this command,you get a shard iterator. Now, a shard iterator is basically important if you want to fetch the data from a specific shard. So, this is quite important. You cannot get data directly from a stream; instead, you must use a shard iterator. We will be discussing that in the relevant section. But just for the practical purpose, I wanted to show you how things really work.

So, once we get the shard iterator, we can go ahead and fetch the data from a stream. So, I'll paste this command, which is AWS, Get records. So getting records basically means fetching the data and specifying the shard iterator and pasting the shard iterator that we received from the earlier command. Now, remember, this shard iterator is temporary. So a new shard iterator might be generated after a specific amount of time. Once you run the command, you will see that there are two records which were received. This is the first record and this is the second record. It also tells you the sequence number associated with each of the records so that you can actually identify them. So, if you see over here in the earlier command, the sequence number was 00:26. If you'll find over here, you have a 00:26 sequence number. So this was the first record and the data is from Kplabs. And the sequence number associated with the secondrecord that we had stored ends with 890, so you have an 890 over here. Perfect. Now, the last thing that I would like to show is that in the records you do not directly get the plain text data. So basically, this is a base 64.

Now, storing a base 64 is critical, especially if you have a binary stream. So this is where it is important. So, once you get the data, you can make a copy. Let's do an online base 64 decode. So, I'll just open up the first website that comes, I'll paste this base 64, I'll click ondecode, and you'll see you get a decoded data which is hello from Kplabs back. Similarly, I'll copy the base 64 associated with the secondrecord over here, I'll paste it over here, I'll click on decode, and you get the plaintext data back in. So, this is the basis of how you can store data and how you can retrieve the data. Remember, if you want to retrieve the data,you need to get a shard iterator. Without a shard iterator, you will not be able to retrieve the data back. So that's the fundamentals of storing and retrieving data in a science stream. We'll be talking more about this in the relevant sections. However, for this practical lecture, I just wanted to show you how exactly this would really work. So, one thing I would strongly advise you to do is practise all of this once so that it becomes much more effective. Otherwise, it will just remain as a theory.

22. Kinesis Data Firehose

Hey everyone, and welcome back. In today's video we will be discussing the AWS kinases data fire hose. Now, the primary function of the kinases firehose is to load the data streams that are coming from the producers into the AWS data store. And this is why, in the second point, I mentioned that the Kindnesses Fireworks are primarily about delivery from point A to point B. So this can be better understood with this diagram, where during the injection phase, the data can come from a wide variety of producers, there can be multiple producers. Then in the second stage of transformation,the Firehose does have a certain amount of capability to transform the data. So it can change the format. It can compress the data before it sends it to the third stage of delivery. In terms of delivery, it can send data to S3, Redshift, Elasticsearch, and Splunk.

So this is where the real analysis would happen. So basically, Firehose is not for analytics. It can transform the data and deliver it to a specific end point. So if you look into the delivery architecture of Firehose, you have the data sources over here. So this data source will send the data to the Kindnesses Firehose. So there can be a certain amount of data transformation that can happen, and then it can send the transformed records, let's say, to the Elasticsearch service. We can even configure it so that if something does not work as expected, it can fail those records to the S3 buckets so that data cannot be lost. So this is the high-level overview of Firehose. Let's jump into practical and understand how exactly it might work. So I'm on my console. Let's go to the services and we'll go to Kindnesses. All right, so now let's go to theFirehose and let's create our first delivery stream. So let's give it a name. I'll call it Kplash Five Horse. And let's go a bit lower. We'll use the source as direct foot or othersources, so that through the CLI we can directly go ahead and push the data here. So let's go a bit lower. I'll click next. So in the next section here, we can go ahead and apply certain transformations. So here you have the source record,then you have the transformed source records. So here we can go ahead and convert the record format. We can even do various aspects like compression, et cetera. So, if you look closely, the first option is to transform the source record using an AWS lambda.

If you want to do that, you can even convert the record format if you intend to. Anyway, to keep things simple, we will leave these things as default. Let's click next. And the next important part is the destination. We have already discussed that it can send the data to a specific destination. So you have Amazon S3, Amazon Redshift, Elastic Search Service, and Splunk. So let's do one thing. We'll use all three. But before that, let's go ahead and create a new S-3 bucket. Here I'll create a new bucket. I'll call it KP Labs Firehose. And I'll click on Create. All right, so now let's go a bit lower. Now, within the S Three destination, you have to select the bucket where the data will be stored. So I'll select the Kpops firehose and let's click on next. Now, on the next screen here,you can basically do various aspects. If you want to compress the data before it has been stored in S3, that is something you can do. You can even do encryption. You can even enable things like error logging. Anyways, let me just deselect error logging. As of now, in order for Firehose to put data into the bucket, there needs to be a role which it can assume. So you can go ahead and create a new rule or you can choose one. So here it will basically show you the exact policy document.

Here you can click on "Allow" and that's about it. So the Firehose delivery role should be visible within the Im role. You can click next. It will show you the review of whatever options that you have selected. You can go ahead and click on "Create delivery stream." All right, so this is your five hostdelivery streams which are being created now. Similar to the kinases data stream, you can go ahead and push some data to the delivery stream. So if you look into the CLI options here, there are various CLI options. The one that we are interested in is primarily the list delivery stream here. Let me open this up and also put in a record. So these are the two important CLI commands that we are most interested in. Now, before we do that, let's quickly verify whether the status has been created. It has not been as of now. Let's quickly wait a moment here for it to be created. All right, so it has been a few minutes and the Kpops Firehose has been created. Let's click here and it also shows you the delivery stream here. So let's do one thing. Let's click on the first CLI, which is the List Delivery stream, to get the exact command. So the command would be "AWS Firehose list Delivery Stream." So let's try this out. So I'll say, AWS.Firehose. Lit delivery streams I'll also specify the region here. All right, so it shows there are no delivery streams. Let me actually change the region. Because it is oil, it should also be US East. great. So now you see that it is showing that there is one stream which is available, which is KP Labs Five host. So in order to put the data into this delivery stream,the command is quite similar to the kinase data stream. So I have created a sample command here.

So, it is quite simple. So you have a table firehose, you have put in a record,then you have to give a delivery stream name. So in our case, the deliverystream name is KPIs firehose. Then you have the record where the data is attributable one way and then you have the region. So let's copy this up here and let's do one thing. Let's paste it Oh, let me just change the region back. I'll say us east too. All right, so on a successful put up message, it will basically give you the record ID here and it will also state whether the message is encrypted or not. So, once this message is delivered to the kinases firehose, it will route it to the proper destination. So, in our case, the destination is three, and this is a distinct bucket. Do remember that it takes a little amount of time. It may take a few minutes for the data to be delivered to the destination.

So let's quickly wait for a moment here. So it has been a few minutes. Now, if you go to the monitoring tab here, you will see that there is one graph which is upcoming. So you have one record on incoming records. all right? So, in essence, we only added one record, and this is what is currently visible. So this record, since this is a deliverystream, this record would have been transferred to the appropriate destination that we have configured. So in our case, the destination was KP lapse firehose. So let's try it out. So within this specific SD bucket, you have a proper directory structure based on timestamp. So let's open this up, all right? And you have one record that is visible. So let's go ahead and we'll download this specific record. All right, so this is a record that has been downloaded. Let's go ahead and open this up in notepad. So, as you can see in the notepad, you can see the file that you put into the kinase firehose. So that's the high level overview of the kindness firehose. I hope you started to understand what Firehose is all about and how Firehose is different from the data stream. So when you talk about a data stream, it needs to have consumers. It cannot automatically send the data to a specific set of consumers. As opposed to that, Firehose can automatically send data to a specific set of consumers like S3, Redshift, elasticsearch, and even Splunk. And along with that, it can even do certain transformations like compression. It can change the format and a few others.

Study with ExamSnap to prepare for Amazon AWS Certified Solutions Architect - Professional Practice Test Questions and Answers, Study Guide, and a comprehensive Video Training Course. Powered by the popular VCE format, Amazon AWS Certified Solutions Architect - Professional Certification Exam Dumps compiled by the industry experts to make sure that you get verified answers. Our Product team ensures that our exams provide Amazon AWS Certified Solutions Architect - Professional Practice Test Questions & Exam Dumps that are up-to-date.