In this Skillsoft Aspire course, explore the use of Combiners to make MapReduce applications more efficient by minimizing data transfers. Start by learning about the need for Combiners to optimize the execution of a MapReduce application by minimizing data transfers within a cluster. Recall the steps to process data in a MapReduce application, and look at using a Combiner to perform partial reduction of data output from the Mapper. Then create a new project to calculate average automobile prices using Maven for a MapReduce application. Next, develop the Mapper and Reducer to calculate the average price for automobile makes in the input data set. Create a driver program for the MapReduce application, run it, and check output to get the average price per automobile. Learn how to code up a Combiner for a MapReduce application, fix the bug in the application so it can be used to correctly calculate the average price, then run the fixed application to verify that the prices are being calculated correctly. The concluding exercise concerns optimizing MapReduce with Combiners.
Getting Started with Hadoop: MapReduce Applications With Combiners
Course Overview
recognize the need for combiners to optimize the execution of a MapReduce application by minimizing data transfers within a cluster
recall the steps involved in processing data in a MapReduce application
describe the working of a Combiner in performing a partial reduction of the data that is output from the Mapper
configure a Combiner to optimize a MapReduce application that calculates an average value
use Maven to create a new project for a MapReduce application and plan out the Map and Reduce phases by examining the auto prices dataset
develop the Mapper and Reducer for the application that will calculate the average price for each make of automobile in the input dataset
create the driver program for the MapReduce application
run the MapReduce application and check the output to get the average price for each automobile make
code up a Combiner for the MapReduce application and configure the Driver to use it for a partial reduction on the Mapper nodes of the cluster
fix the bug in the previous application by defining a type that represents both the aggregate price and count of automobiles that can be used to correctly calculate the average price
compare the output of the modified application with the previous buggy version and verify that the average prices for the vehicles are being calculated correctly
identify the shortcomings of regular MapReduce operations which are addressed by Combiners, and how Combiners differ from Reducers