With the exponential growth of data volumes in the last decade, Apache Hadoop has undisputedly played a vital role in solving the problem of performing analytics on huge data sets. However, together with all benefits and perks, using Hadoop brings along quite a few challenges, as well. An on-premises Hadoop cluster, together with all the applications and utilities you need on it, is pretty complex thing to configure, run and maintain. Even though you can add nodes to it and scale up as your workload increases – that doesn’t happen in a wink of an eye and “compute” and “storage” are still stuck together to a great extent, plus you need to pay for it all in advance.
Why Implement AWS EMR?
AWS EMR is a cloud platform which makes it easy to create and manage fully configured, elastic clusters of EC2 instance running Hadoop and other applications in the Hadoop ecosystem. Create as many clusters as you need in a matter of minutes, enable analytics on large data sets – business critical data, clickstreams, logs, etc. Let your data science teams spin up the clusters they need in order to experiment and create value for your business.
Reduce complexity
As a managed service, EMR takes care of the infrastructure requirements, so you can focus on your core business. You only need to take the decision how many nodes you need in your cluster, what is your preferred Hadoop distribution, and pick the application you want pre-installed, and EMR will take care of creating the cluster for you.
Optimize costs
Run your clusters only when you need them and pay only what you have used. Take advantage of EC2 Spot instances to further reduce the cost. Find and terminate any idle instances so you do not pay for resources you are not using.
Gain flexibility and scalability
Create clusters of the required size and capacity in only minutes, experiment and pick the instance types that make most sense for your workloads. Thanks to EMR Managed Scaling, your clusters can be dynamically scaled in or out.
Secure big data workloads
Take advantage of all built-in security features of the AWS platform – encrypt your data at rest and in transit, use IAM to securely control the access to the AWS resources used, and EC2 security groups to limit the inbound and outbound traffic to your cluster’s nodes. All security setups can be added to security configurations and then re-used as templates whenever you create new clusters.
Get high-availability and reliability
Launch your clusters in as many Availability Zones as you like in any AWS region. A disaster in one region can be easily worked around by spinning up the same clusters in a different region as this happens within minutes and your workloads will not be blocked.
Integratе seamlessly
As a fully managed AWS service, you can easily integrate your EMR clusters with other AWS services like S3, Kinesis, Redshift and DynamoDB in order to enable data movement and analytics across a wider range of services on the AWS platform.
10x Increase in Analytics Team Productivity with an AWS Data Lake Implementation
10x
more productive analytics team
0
manual effort needed to produce unified and consolidated reports
0
infrastructure maintenance needed
A North American Health group was struggling with consolidating their accounting reporting as the group consists of a number of companies and clinics each of them using different accounting software solutions. The group was also looking at a centralized repository for storing and reporting on their EMR data (Electronic Medical Records).
Approach to AWS EMR Implementation
- Identify all stakeholders.
- Conduct a series of exploratory workshops to get acquainted with the end-to-end environment – identify data volumes, producers, consumers, analytics requirements, etc.
- Create a classification of the teams and processes which would benefit from persistent EMR clusters and such which can use transient EMR clusters.
- Create a high-level design of the solution, making sure it integrates well with existing environments, while taking into consideration the possibility of future cloud migrations.
- Create an end-to-end implementation plan, including scope, timelines, milestones, and deliverables.
- Define the data ingestion strategy for each data producing source system.
- If this is your first cloud project – our team will help you establish all necessary, cloud-based infrastructure and security mechanisms.
- In case of migration from an on-prem cluster – perform shadow test to identify the right size and configuration of the EMR clusters, so you get on par or better performance at lower cost, compared to your on-prem solution.
- Automate provisioning of EMR clusters and create security configurations to easily apply the required security mechanisms to each new cluster.
- Implement data pipelines to ingest data frow any identified source.
- Implement or migrate data transformation and analytics workloads.
- Configure CI/CD pipelines to automate, testing and deployment.
- Deliver detailed technical documentation which will allow your team to operate efficiently the new environment.
- Conduct knowledge transfer and training sessions, making sure all technical and business users are well-acquainted with the delivered solution, its features, and capabilities.
What we do
Adastra can help you plan and implement a scalable and secure solution that would best fit your organization analytics requirements. We’ll help you build environment that will allow your teams to get the resources and insights when they need them, at a fraction of the complexity and cost of an on-prem solution, reducing the administration and maintenance complexity to a great extent.
1
Assessment
Identify what are your user personas, current end-to-end environment, and requirements. Based on the findings, Adastra will plan the right approach to sizing and building the environment that will cover your organization needs.
2
EMR implementation
Our experienced team of professionals will make sure you get a scalable, secure and performant solution at a lower cost, compared to on-prem clusters. We will build the data ingestion and transformation patterns and processes, implement and/or migrate the analytics workloads for you, and put in place the necessary CI/CD processes and security mechanisms.
3
Knowledge transfer
We will make sure your team is fully capable and comfortable working with the implemented end-to-end solution, including the ability to easily terminate, adjust and spin up new clusters. Optionally, you can benefit from Adastra’s Managed Services where we run the EMR clusters for you and maintain and evolve all applications and analytics workloads on them.
FAQ
Apache Hadoop is an open-source framework, which efficiently processes and stores large datasets (in the GB to PB scale). Hadoop takes advantage of using a cluster of commodity hardware to massively parallelize the processing workloads. Hadoop consist of four main modules:
- Hadoop Distributed File System – a distributed file system, residing on the cluster, which provides large data throughput and fault tolerance.
- Yet Another Resource Negotiation (YARN) – resource manager.
- MapReduce – a framework which helps programs perform parallel computation on data.
- Hadoop Common – common Java libraries that can be used across all modules.
Some of the most popular applications which store, process, analyze, manage big data, and run in Hadoop are Spark, Presto, Hive, HBase, etc.
A Hadoop cluster is a group of commodity hardware connected together. This cluster runs open-source software and provides distributed and fault-tolerant compute and storage features. A Hadoop cluster implements a Master – Slave architecture. Usually, a high-end machine acts as a Master Node and hosts various storage and processing management services for the entire clusters, whereas the Slave Nodes are responsible for storing the data and performing the actual computations on it.
Running Hadoop in AWS (using EMR) has quite a few advantages, compared to running Hadoop on an on-premises cluster:
- Easy to use – you can launch an EMR cluster in minutes and do not need to worry about all the configuration and administration overhead
- Cost – with EMR you pay only for what you use as you pay an hourly rate for the instances in your cluster.
- Elasticity – you can easily provision as many compute instances as you like to cope with any unpredicted workload and then scale back in.