Data Engineering

20. AWS DMS - Database Migration Services

Let's go again into a high-level overview of another service, which is DMs, or Database Migration Service. It's a quick way to securely migrate databases to AWS. It's resilient and self healing.The source database remains available during the migration, and it's continuous. It supports some kinds of homogeneous migrations, for example,an Oracle database on premise to an Oracle database on AWS, but also some heterogeneous migrations. For example, if you have a SQL Server onpremise and you migrate to Aura, it's continuous data replication and uses CDC or change data capture. And you must create an instance to perform the replication task. So the way it works is that the sourceDB is somewhere, maybe on account or maybe on premise, and you have two instances,and they will be running database migration services. So DMs and then it will know how to move the data from the source database into the target database. So it's a very simple service, as the name indicates. It is a database migration service and a database migration service only. So when we use DMs and when we use glue, again,glue, we already know this, but it's an ETL on Spark. And with Glue ETL, you don't provision or manage resources. And the glue data catalogue helps you make data sources available to other services. Unlike DMs, it's continuous data replication. So glue is batch oriented. Okay? The minimum amount of time, by the way, that you can set up a glue ETL is five minutes. Unlike for DMs, it's continuous, it's real time. DMs do not do any kind of data transformation. It literally takes data from one database and puts it into another. So if you have ETL requirements, you need to do it afterwards. And once the data is in AWS, you can actually use glue to transform it. So it is very common, for example, to use DMs to take a database on premise and replicate it into the cloud, and then use glue ETL to take the data from the database and perform ETL on it before putting it somewhere else. For example, in S Three, two services are complementary. But remember, DMs are for database migrations and are for pure ETL. Okay, well, that's it for this lecture. I hope you like it and I will see you in the next lecture.

21. AWS Step Functions

So, step functions are here to orchestrate and design workflows. They provide you with easy visualizations, and we'll see some in the next slide, and they provide you with advanced error handling and retry mechanisms outside of the code. So the idea is that you would define a workflow, for example, do this, then do that, then retry on failure,then do this, if that works, then do that, if that doesn't work, and so on. And so all of this orchestration is managed bystep functions, and thankfully for the step function service, we get an audit of the history of workflows. So if you have an ETL, for example, that's very complex and needs to orchestrate between 20 different services, you can know how every service reacted, what happened, and get some audits around how things worked. If they didn't work, you can do some debugging. So you also have the ability to wait for an arbitrary amount of time, and a step function on its own. The workflow can have a maximum execution time of one year. So it could be a very very long workflow. So this is all very text-based, but let me just show you some concrete visual examples. So for the first one, we can use stepfunctions to train a machine learning model. For example, here we have the code on the left handside and you're not expected at all to know the code, OK? But on the right hand side, you get a visualisation that anyone can understand. So the start is that a data set is generated, maybe using a lambda function, and then we go into the next step, which is training the model. It applies an algorithm coming from stage maker, and it will train the model. At the end, it will save the model into an SV bucket and maybe do a batch transfer of the model. As a result, the entire orchestration is performed by step function. Each actual step will invoke a different service. For example, lambda, sage maker, s3, and so on. However, in this case, the step function is used to define the workflow for how things should work at a high level. So step function is all about the power of orchestrationservice. Let's look at another example, for example, tuning a machine learning model. So here we generate a training dataset using lambda, then we do hyperparameter tuning using XGBoost, and then we extract the model path. Then we do it again. We save the model for this hyperparameter treating, then we extract the model name, and then we do a batch transform. So you can think about so many different machine learning flows you can do and orchestrate with step functions. Finally, if you have a batch job, for example, this is a cool thing because it has a success and a failure route. So we submit an AWS batch job that will do something. It could be a glue, could be a batch, could be whatever you want. And then if it's successful, we'll notify you. And if there is a failure, it will notify the user. But you can get very, very complicated with your step functions. And this is just a high-level example to explain to you how things work. So going into the exam, remember, any time you need to orchestrate some things or ensure that one thing happens, and then another thing happens, and then another thing happens, then step functions are the perfect candidate for this. Okay, well, I hope that was helpful, and I will see you in the next lecture.

22. Full Data Engineering Pipelines

Okay, so just a very big and long lecture summarising all the services we've seen and trying to understand how they fit with one another. Now, this can be quite heavy for you, but hopefully you understand everything I'm saying right now. And this is just a revision, and hopefully she didn't understand how every service relates to one another. This will shed some light on it. So let's talk about the real time layer. We first have producers producing into a kinesis data stream,and for example, we'll hook kinesis data analytics into it to perform a real-time stream of analytics. We can use Lambda to read from that stream and relax in real time, or we can just have the new destination stream into a kinesis data stream itself. Or we can even send it to Kinesis DataFirehose if you want to ingest the data somewhere. So if it goes into a Kennedy data stream, maybe we'll have an application on EC Two that will read that stream and do some analytics as well. And some machine learning. It could definitely talk to the Amazon SageMaker service to do some real-time ML. If it's in Kennedy Data Firehose format, it could be converted into ORC format to be sent to Amazon S3 and then loaded into Redshift. Or it could be sent in JSON format to Amazon's Elasticsearch service, for example. Also, we don't necessarily have to produce two kinesis data streams. We can also produce two Kinesis Data Firehose and from there, for example, send it in packet format into an Amazon S tray. And as we all know already, we can connect the AmazonKennedy Data Firehose to Amazon Kinesis data analytics, as we saw in the hands-on in the previous lectures. So this just shows you a lot of different combinations we can have with kinesis datastreams, kinesis data analytics, firehose and so on. But hopefully it makes a lot of sense to you and you're starting to finally understand the differences between all these services and how this real-time layer, the near real-time layer, works together. Okay, next we are going to go into the Kinesis Video layer. So we have video producers, and for example, it could be a camera, but it could also be a deep-lens device. And we get sent to a Kinesis video stream. Remember, one stream per device. And so maybe we'll have Recognition, which is a machine learning service that Frank will introduce to you, that reads from that video stream and produces another kinetic data stream that's full of data. And from this we know, we've already seen it. We can have Amazon EC2 Kinesis Data Firehose or Kinesis Data analytics just reading thatstream and doing what we need to do. Alternatively, we could just have Amazon EC2 two instances,reading of the Kennedys' video stream and then talkingto Sage Maker to produce another kinesis data stream that may be read, for example, by Lambda to react in real time to notifications, for example, when something looks wrong in a video. OK, now we have the batch layer. The batch layer is responsible for data collection, transformation, and so on. So, say your data lives on premise in MySequel and we first used Database Migration Service to send it all the way to RDS. This is a one to-one app and there is no data transformation, it just sends and replicates the data from Onpremise to RDS. Then we want to analyse this data. So we'll use Data Pipeline to take the data from RDS and maybe place it in an Amazon three bucket in a CSV format. We could do the exact same thing with DynamoDB going through the pipeline and insert it into JSONformat in your Amazon S3 bucket. Then we need to transform the data with an ETL into a format we like. So we can use Glue ETL to do someETL and also convert it at the very end from the Parquet format into Amazon's Three. And for example, let's say our job creates many, many different files. We could have an AWS Batch task that would clean up your siblings once a day. So this would be a perfect use case for Batch because Batch wouldn't transform data, it would just clean up some files in your Amazon rebucket from time to time. OK, so how do we orchestrate all these things? Well, step functions would be great for that,to just make sure that the pipeline can be reactive and responsive and can track all the executions of the different services. And so, now that we have the data in so many different places, it would be great to have an AWS glued data catalogue to know what the data is, where it is, and what the schema is. So, as such, we'll deploy crawlers into DynamoDB, RDS, and so on to create that data catalogue and keep it updated with the schemas. So that would be the batch layer. And finally, there is the analytics layer. So we have data in Amazon S3 that we want to analyze. So we could use EMR, and Frank will introduce EMR to you, which is Hadoop, Spark, Hive, and so on. We could use Redshift or Redshift Spectrum to do data warehousing, or we could even create a data catalogue to index all of the data in SRE and create the scheme. And so we can use Amazon Athena in a serverless fashion to analyse the data from Astray. And then when we have analysed the data, maybe we want to visualise it. So we'll use QuickSite and Frank will also introduce that to you to do visualisation on top of Redshift or Athena. So that's it for the whole data engineering module. I hope you like it. I hope you learn a lot. And anytime I mentioned machine learning or a service you haven't seen, for example, EMR or QuickSite, Do not worry. Frank is going to introduce everything to you, and I'll leave you in his hands. That was all for me on this course. I hope you liked it. And Frank will see you in the next lecture.

Exploratory Data Analysis

1. Section Intro: Data Analysis

In this next section, I'll take you through the world of exploratory data analysis. Data engineering gets your data where it needs to be,but the algorithms you use for machine learning often require your data to be in a specific format and not to have any missing data in it. We're going to dive into the world of feature engineering, which is an important discipline that isn't often taught. I'll teach you how to deal with missing data through various imputation techniques, how to handle outliers in your data, and how to transform and encode your data into the format machine learning algorithms expect. It's also essential to understand your data before you start training algorithms with it. That's what exploratory data analysis is all about. We're going to cover tools such as SidekicklearnAthena QuickSite MapReduce and Apache Spark can give you insights into your data, even if it's coming from a massive data lake. We'll also cover some data science basics so you can understand concepts like data distributions, trends, and seasonality. This is all stuff you're expected to know on the exam. When we're done, we have a fun hands-on exercise where we will build a search engine for Wikipedia on Elastic MapReduce, which requires quite a bit of preprocessing of our data. Let's dive in and learn how to understand and prepare our data prior to training with it.

2. Python in Data Science and Machine Learning

Let's dive into the slides again. We're going to start with the exploratory data analysis portion of the exam. And you can't really talk about the world of exploratory data analysis without talking about Python. Python is becoming the language of choice for machine learning and data science. However, the test will never expect you to actually write Python code or even understand Python code that's put in front of you. So we're in kind of a weird spot here and how we teach this. Basically, you just need to know how Python is used in this field and some of the various components that are used within it, how they fit together and what they're used for. So we're going to keep things kind of high level here and talk about kind of the main concepts that we have instead. So, for example, we could take a look at this specific snippet of code from a Jupyter notebook and dissect it on a syntactic level. I mean, the first line is saying that we want to inline graphs and charts that are created by the Matplotlib package and that we're going to use the NumPy and Pandas packages. And then we use the Read CSV function within Pandas to actually load up a comma separated value file called Past Tires CSV into something called a data frame. And then we call head on that data frame to display the first five rows of it. So let's peel that apart. Some of the key things that we talked about were Matplotlib, NumPy, Pandas, and data frames. So let's at least talk about those main concepts in higher level packages that we might use. Start with Pandas. That's probably the most important one in the field of exploratory data analysis or data preparation more widely. Pandas is a Python library for slicing and dicing your data and manipulating it. So it's a great way to sort of explore your data, see what's in it, what sort of values you have in the different columns, and kind of manipulate it and deal with outliers and things like that. Now, when we talk about pan, as we talk about data frames a lot, a data frame is just an atable and that allows you to manipulate rows and columns of your data, kind of like an Excel spreadsheet, except you're using Python code instead of Excel and, well, it can do fancier stuff too. You also have the concept of series. In Pandas, a series is just a one-dimensional structure, like a single row from a data frame. You might extract a row from a dataframe into a series because you're going from two dimensions to one dimension there. And it's also important to talk about NumPy. Pandas interoperates with NumPy. You can take a NumPyarray and import that into Pandas, and take an adata frame from Pandas and export that into NumPy. NumPy is just a lower level library that also deals with arrays of data in Python. And often, that's the format that things need to be in before you pass your data off into an actual machine learning algorithm. So it's very common to import your data with Pandas,manipulate it, explore it, clean it up a little bit,and then export that to a NumPy array, which then,in turn, gets fed to a machine learning algorithm. Let's look at a couple of examples here. So in this first one, we're going to extract the first five rows of a data frame from the previous slide. That's what that colon five does there? And we're going to extract just two columns from that data frame. The years of experience column and the higher column So what we end up with is a new data frame that just contains those two columns and the first five rows of it. Pretty simple stuff. So that's a very basic example of how you might use Pandas to sort of extract the data you care about from a larger data set. In this other example, we're doing something a little bit more interesting. We're using something called "value counts" to sum up how many of each unique value within a column exists. So in this particular example, we're extracting the level of education column from our data frame and calling value counts on it, which tells us that within that data frame, we have seven bachelor of science,four PhDs, and two masters candidates within that data frame. So you can see how this can be very useful in understanding your data. You can do things like create new columns based on other columns or check for missing data and deal with that missing data. Again, The details of the syntax in Python are not important. You just need to know that this exists. Now, often you'll be working with a subset of your data because a single PC can only handle so much. So while you're sort of developing and tuning your algorithms, you might use Pandas, for example, to just toextract a random sample of your data to work with while you're in that development stage. There's a lot more to learn here with Pandas, but it's not really important on the exam what the details are. Let's also talk about Matplotlib. It's a basic way to visualise data that might come from a pandas data frame or an umpirese. Let's look at an example here. This is what's called a box. and a whisker plot. It's useful for visualising data distributions and outliers. So in this little snippet of code here, we've set up a uniform distribution of random data. between + and -40 And We've Also Added in Some Outliers Above plus 100and-100 as well as this Box and whisker plot that shows you that mean value in the middlewhere that red line is and The Lines The box itself represents the inner quartile of data, and those outliers are outside the outer quartiles of the data. There. So it's a very simple way of seeing how your data is distributed and how many outliers you might have to deal with in that data. In another example, we have a histogram here. Hopefully you're familiar with what an histogram is, if you're not. Basically, you can think of each of these columns here as a range of values. So when you're dealing with continuous numerical data,we can bucket that into different ranges. We might also refer to that as binning later on. For example, I don't really know what this specificline here might be, but let's say that it represents everything between 22,020 and 3000, for example. And that line represents how many data points lie within that particular range. And we have a nice bell curve here. That's a very typical data distribution in the real world, where things tend to cluster around a specific range of values and quickly fall off as you get to more outliers from that mean value. There's also something called seaborn you should know about. Seaborn is basically a map plot lib on steroids. So here's a few examples: Here's a seaborne box plot. It's basically just like the box and whisker plot in the previous slide from that plot lib. But it's a lot prettier, a lot nicer to look at, and it has more flexibility. You can see here we're looking at several box and whisker plots from different dimensions of our data set here. It also has something called a heat map. And heat maps are something you should know about for sure. In a heat map, the colours of these cells represent the average values at each point there. So in this case, we're visualising miles per gallon for a given engine displacement. And the number of cylinders and the actual colour of each cell here represents how many values fall within each one of those categories,those combinations of categories there. So it's a good way of getting a more intuitive feel as to how often different data points appear in different combinations of your categories. Also, something in seaborn that's often used is called the pair plot. This lets you visualise plots of every possible combination of attributes all at once. So in this example, we're looking at several different metrics around miles per gallon and the number of cylinders of a car. So by eyeballing this series of charts here, you can look for ones that have the best correlation and get some insight as to which attributes of your data actually correlate the most with each other,which can be a useful thing to know. Another thing you see in Seaborn is often called the joint plot. This just combines a scatter plot with histograms on each axis. So you can visualise the distribution of your data as a whole and along individual attributes all in one display here, which is kind of nice. So that's what seaborn is, pretty charts and graphs, and we can't talk about the world of Pandas machine learning without talking about ScikitLearn. ScikitLearn is basically a Python library for machine learning models. And we're getting a little bit outside of the world of preparing your data and exploring your data into actually doing machine learning on it. But I want you to understand how it all fits together as well. However, there is some relevance as you might be using ScikitLearn to experiment on a subset of your data like we talked about. And as you're exploring your data, you might try out different algorithms on it as you go and see how it responds to different changes that you might make to that data. So the good thing about Sci, Katelearn, is that its code always looks pretty familiar, and I need to back up here a little bit. I'm hoping that you have some prior experience in machine learning or some exposure to machine learning before.If not, I have to kind of question why you're taking this exam. You can't really expect to just become an expert in machine learning by watching a course. I mean, you need to have a little bit of background here, guys. But for those of you who are coming into this new regardless and aren't listening to my warning,I'll give you some high-level concepts here with machine learning as we go through here. So, again, the nice thing about ScikitLearn is that you can always use a kind of code that looks the same. So in this particular example, we're importing something called a random forest classifier. And a random forest is just a collection of decisiontrees where each tree has some random variation to it and they all vote on the final result. A decision tree is just a cascade of decision points. So, for example, you might say if the person has a Bachelor of Science degree and has more than ten years of experience and did an internship when they were in college, then they're likely to be hired. So you have this sort of cascade of decision points that lead to a final classification at the end. In that example, hired or not hired And a random forest is just a collection of those trees. So we have a random forest classifier that we're creating, and we're just saying that it will have ten trees within it. And we're going to assign that to a classifier called CLF. So that classifier could be any of a number of different algorithms. They're all going to work pretty much the same way. We're going to call it a day. And you'll see that it takes two parameters, x and Y. Typically, x will refer to the attributes of your data. So the things that you're always going to know about your data coming in and Y will be the labels. That's the thing that you're trying to predict. So, for example, going back to the hiring example, maybe X would contain things like years of experience and years of college. In how many internships do they do things like that, and why would that be the label that you're interested in? In this case, are they hired or not hired? So again, X refers to the attributes and y to the labels. Furthermore, this will usually come from what's called a training data set. As a result, the best practise is to divide your data into two sets. When you're training a training set and a test set, you set aside a certain percentage of your data as a test set so that you can take data that your algorithm has not been trained on and evaluate how well it can predict labels from that test set that you set aside. Okay, so that's the basic idea of a training test. Again, I really hope you already know that already.If you don't and you're brand new to machine learning, you're going to have an uphill battle passing this exam, quite honestly. I'll do what I can to get you through it, though. After that, we can just call predict the fitted model, and in that case, we can just pass in an array of features. This actually corresponds to an employed ten-year veteran or an unemployed ten-year veteran and prints out the predicted result, the predicted label. In this case, the first is one, indicating that we expect it to be hired, and the second is zero, indicating that we do not expect it to be hired. So you'll see that all ScikitLearncode looks pretty much the same. It's very easy to use. The real decision is which algorithm to use and which one gives you the best results. Also, to step back again, we're talking about a classification problem here. In this case, we have two classes that we're trying to predict will be hired or not hired. Other examples might be handwriting recognition. Which letter did the person write down or what number did they write down? In all these cases, we have a discrete set of classifications that something can be that we're trying to predict. The other type of problem is a regression problem, where we're trying to predict a specific numerical value. So, for example, what height do I predict for a given weight based on previous data about the relationship between a person's height and weight? In a regression problem, we have a basically infinite range of possible solutions because it's numerical data that we're dealing with. Now you will kind of see that classification problems tend to be more ubiquitous in the world of machine learning these days. That's because they're a good fit for deeplearning and there are actually ways to take regression problems and model them as classification problems as well that we'll look at later. Getting back to ScikitLearn itself, it also includes some preprocessing capabilities that are relevant to the world of data preparation and analysing your data. So, for example, you could use the preprocessing module of SciKitLearn to scale all of our feature data into a normal distribution. A lot of algorithms expect that as their input, including neural networks. And I would love to go into even more depth here, but we're going to talk about the various machine learning techniques themselves in the modelling section of the course. So for now, I just want you to understand the role that Python plays in exploratory data analysis.

ExamSnap's Amazon AWS Certified Machine Learning - Specialty Practice Test Questions and Exam Dumps, study guide, and video training course are complicated in premium bundle. The Exam Updated are monitored by Industry Leading IT Trainers with over 15 years of experience, Amazon AWS Certified Machine Learning - Specialty Exam Dumps and Practice Test Questions cover all the Exam Objectives to make sure you pass your exam easily.