Question

Business Case

Your company needs to conduct an analysis of a large dataset that is 10 times bigger than the size of a hard drive of your most powerful computer. This dataset contains data about ATM transactions. The security team has provided a smaller dataset of suspicious transactions. Your task is to identify transactions similar to suspicious ones.

Directions

explain which parallel computing techniques would be appropriate for this case and why. Explain which component of Hadoop ecosystem may be used in this case, and why.

Example of parallel computing could be (distributed calculations vs. MapReduce) or others

key principles of Apache Hadoop environment (Hive, Spark, Pig, etc.) or others

Answer 1

For this case, Map Reduce parallel computing technique would be appropriate for the following reasons -

MapReduce is a processing technique and a program model for distributed computing based on java.

MapReduce has mainly two tasks, map and reduce.

Map takes a set of data where individual elements are broken into tuples.

Reduce take output from map as input and combine those tuples into a smaller set of tuples.