Spring Integration | Demo

As explained in previous post Spring Integration application can run inside any spring application container. It may be a web application (loading context via web.xml) or a standalone application (loading context via ClassPathXmlApplicationContext.class).

We will start with a quick file watcher example in which we will implement a file adaptor which will process incoming files asynchronously.

We will run the application as a Java application using main() function and ClassPathXmlApplicationContext.class

Our project will have following structure:
 

Hadoop | MapReduce API

MapReduce is a massively scalable, parallel processing framework that works in tandem with HDFS. MapReduce and Hadoop enable compute to execute at the location of the data, rather than moving data to the compute location; data storage and computation coexist on the same physical nodes in the cluster.

Hadoop MapReduce is an open source implementation of MapReduce

MapReduce (older API implementation with Hadoop 0.20 - just for conceptual reference)

Pig | High Level MapReduce

Pig is a high-level platform/scripting-language for creating/running MapReduce programs used with Hadoop. The language used in this platform is called Pig Latin. Pig Latin allows user to write complex MapReduce transformations using a simple scripting language. This language defines a set of transformations on a data set such as aggregate, join and sort conceptually similar to SQL for RDBMS systems.

Pig have rich set of functions, Pig Latin can be also be extended using UDF (User Defined Functions) which can be written in Java, Python, JavaScript, Ruby or Groovy as per custom requirements and then call directly from the language. Detailed list of Pig built in functions can be found here.

Pig is complete in that you can do all the required data manipulations in Apache Hadoop with Pig. It simplifies the complex operations like joins and filters etc. for performing query operations in hadoop.

Wide usage of Pig includes:

• Extract Transform Load (ETL) e.g. Processing large amounts of log data, clean bad entries, join with other data-sets.
• Research of "raw" Information e.g. User Audit Logs where Schema maybe unknown or inconsistent