Skip to content
This repository has been archived by the owner on Apr 14, 2023. It is now read-only.

Latest commit

 

History

History
42 lines (31 loc) · 1.6 KB

README.md

File metadata and controls

42 lines (31 loc) · 1.6 KB

MapReduce

A simple implementation of MapReduce distributed system and programming model.

Introduction

MapReduce is a programming model and distributed system devised by Google in the early 2000s to do parallel large scale computation of large data sets.

A large input is partitioned into multiple input splits that are sent into worker machines containing the map function. The map function reads the input and each input entry is associated with a unique key.

All (key, entry) data are saved locally as intermediate data and sent over in the network to worker machines containing the reduce function. Entries with the same key are usually combined in the reduce function.

An example of count of URL access frequency from the MapReduce paper:

The map function processes logs of web page requests and outputs ⟨URL, 1⟩. The reduce function adds together all values for the same URL and emits a ⟨URL, total count⟩ pair.

References

Checklist

User Program

  • Input Split
  • Run Workers Locally
  • Tasks Progress Tracking
  • Wake Up from Master
  • Input Split with Custom File Format Reader

Master Program

  • Workers Status Periodical Update
  • Map Tasks Execution
  • Reduce Tasks Execution
  • User Program Wake Up

Worker Program

  • Map
  • Reduce
  • Task Abortion on Master Failure

Distributed File Storage Use

  • Run Workers Remotelly (Linux)
  • Modify Program to use distributed storage instead of local storage