Cloud Native Landscape

Useful Links

• Cloud Native Landscape - CNCF
• Cloud Native Landscape - GitHub

ZooKeeper | A Reliable, Scalable Distributed Coordination

In previous posts we learnt about various big data projects/systems, all of these systems are distributed and clustered in nature. For distribution and cluster management, all of them needs one or another low level API. ZooKeeper can be seen as one of those low level APIs which can be used to build a distributed co-ordination system.

ZooKeeper is a highly reliable, scalable, distributed coordination system. As per ZooKeeper wiki 

"ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchical name space of data registers".

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization and group services. It provides a very simple interface to a centralized coordination service. The service itself is distributed and highly reliable.

Distributed applications use Zookeeper to store and mediate updates to important configuration information. Many top level big data projects like Hadoop, Kafka, HBase, Accumulo, Solr uses ZooKeeper as a distributed co-ordination system. Extensive list of projects powered by ZooKeeper can be found here.

As ZooKeeper wiki says it coordinates using a shared hierarchical data registers, in ZooKeeper terms these registers are known as ZNODEs.

ZooKeeper comes with bunch of "out of the box" benifits like:

• Fast
• ZooKeeper is fast with workloads where reads to the data are more than writes. The ideal read/write ratio is about 10:1.
• Reliable
• ZooKeeper is replicated over a set of servers known as ensemble. All the servers are visible to each other. The ZK service is available hence there is no single point of failure.
• Simple
• ZooKeeper follows a simple data model and maintains a standard hierarchical name space, similar to files and directories on a file system.

ZooKeeper Ensemble

Spark | Lightning Fast Cluster Computing

Apache Spark is an open source cluster computing platform/framework which brings fast, in-memory data processing to Hadoop. Spark's expressive development APIs allow data workers to efficiently execute streaming, machine learning or SQL workloads that require fast iterative access to datasets.

It extends well known MapReduce model to further efficiently support various types of computations, including interactive queries and stream processing. Speed is the key in processing large datasets.

If we have large amounts of data that requires low latency processing that a typical MapReduce system cannot provide, Spark is the right choice, it performs at speeds up to 100 times faster than Map Reduce for iterative algorithms or interactive data mining as it provides in-memory cluster computing for lightning fast speed.

Apache Spark consists of Spark Core and a set of libraries. The core is the distributed execution engine and the Java, Scala, and Python APIs offer a platform for distributed ETL application development.

Spark was originally developed in the AMPLab at University of California, Berkeley and later donated to Apache Foundation.

Note that Generally Spark is used on the top of HDFS. At a high level we can say we may use Spark Core in conjuction with HDFS.

Spark combines SQL, streaming and complex analytics together in the same application to handle multiple data processing scenarios. It can access wide range of data sources such as HDFS, Cassandra, HBase or S3.

Extensive list of users and the projects powered by Spark can be found here.

At a high level Spark addresses following use cases:

• Streaming Data
• Apache Spark's key use case is its ability to process streaming data. With so much data being processed on a daily basis, it has become essential for organizations to be able to stream and analyze it all in real time.
• Machine Learning
• Spark has useful implementation of machine learning capabilities including wide variety of machine learning algorithms like classification, recommendation, clustering, pattern-mining and so on.
• Interactive Analysis
• Initially Hadoop MapReduce was developed to handle batch processing and SQL-on-Hadoop engines such as Hive or Pig are extremely slow for interactive analysis, where as Spark provides very fast queries to support interactive analysis using its in-memory capabilities. In other words we can say Spark is a batch analytics system that can pretends as an interactive analytics system because of operating on in-memory RDD's and the caching hence possible.

Spark | Use case reference

Big Data Ecosystem

Useful Links

• Apache Hadoop Ecosystem
• Big Data Ecosystem Table

Hive | Input & Output Formats

In previous post we learnt about setting up and runnning Hive on our distributed Hadoop cluster. In this post we will learn about various Hive input and output formats.

Key Formats

• TEXTFILE
• AVRO
• RCFILE
• SEQUENCEFILE
• PARQUET

We will use same Hadoop cluster and Hive setup done in previous post.

Usage | Hands On

• TEXTFILE
• Separated readable text file e.g. text file with tab or comma separated fields. This is the default format for Hive (depending on hive.default.fileformat configuration).
• Syntax:  STORED AS TEXTFILE 
• Usually human readable text.
• CREATE TABLE

  hive> USE user_db;
  OK
  Time taken: 0.044 seconds
  hive> CREATE TABLE IF NOT EXISTS users_txt (uid String, login String, full_name String, email String, country String) COMMENT 'User details' ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE;
  OK
  Time taken: 0.384 seconds
  

• LOAD DATA

  hive> LOAD DATA LOCAL INPATH '/tmp/users.csv' OVERWRITE INTO TABLE user_db.users_txt;
  Loading data to table user_db.users_txt
  Table user_db.users_txt stats: [numFiles=1, numRows=0, totalSize=7860, rawDataSize=0]
  OK
  Time taken: 1.138 seconds
  

• READ RECORDS

  hive> SELECT * FROM users_txt LIMIT 10;
  OK
  755777ae-3d5f-415e-ac33-5d24db748e09 rjones0 Randy rjones0@archive.org RU
  a4dae376-970e-4548-908e-cbe6bff88550 mmitchell1 Martin mhamilton1@stumbleupon.com FI
  f4781787-c731-4db6-add2-13ab91de22a0 pharvey2 Peter pkim2@com.com FR
  d35df636-a7c8-4c50-aa57-e99db4cbdb1a gjames3 Gary gtorres3@bbb.org LT
  d26c04a3-ca28-4d2e-84cf-0104ad2acb92 rburton4 Russell rwest4@youtube.com YE
  6a487cfb-5177-4cc2-bdbd-4bc4751b9592 pharris5 Patrick ptaylor5@cnn.com NO
  3671d7f7-2a75-41dc-be84-609106e5bdfa kcrawford6 Keith ksmith6@weibo.com PT
  beae01c4-3ee6-4c59-b0d6-60c5811367f2 jedwards7 Juan joliver7@fc2.com PH
  899dc8a4-5a8f-44cf-ac23-ae8c3729836c slynch8 Samuel smcdonald8@princeton.edu VN
  f274e93d-378c-4377-a9c7-7c235a36b72a mgray9 Martin mrodriguez9@constantcontact.com IE
  Time taken: 0.696 seconds, Fetched: 10 row(s)
  

Cassandra | Setup

We learnt about Cassandra in previous post. We will setup and run client on an Cassandra cluster(fully distributed) here.

Installation

For installation we will use three nodes. We will install fully distributed Cassandra cluster. Here we are using following details for installation (for complete setup):

• Installation base directory:

• /home/anishsneh/installs

• Installation user name:

• anishsneh

• Hostnames: 

• server01 (first node, say with ip address 172.16.70.131)
• server02 (second node, say with ip address 172.16.70.132)
• server03 (third node, say with ip address 172.16.70.133)

Note that in Cassandra there is NO SINGLE POINT OF FAILURE, hence all the nodes are equal and there is no MASTER or SLAVE.

• Install Cassandra
• Download Apache Cassandra binary from Apache Website.
• Extract downloaded package to /home/anishsneh/installs, such that we have:

  [anishsneh@server01 installs]$ ls -ltr apache-cassandra-2.1.5/
  total 360
  -rw-r--r--. 1 anishsneh anishsneh   2117 Apr 27 07:33 NOTICE.txt
  -rw-r--r--. 1 anishsneh anishsneh  64431 Apr 27 07:33 NEWS.txt
  -rw-r--r--. 1 anishsneh anishsneh  11609 Apr 27 07:33 LICENSE.txt
  -rw-r--r--. 1 anishsneh anishsneh 245971 Apr 27 07:33 CHANGES.txt
  drwxr-xr-x. 2 anishsneh anishsneh   4096 May 17 15:37 interface
  drwxr-xr-x. 4 anishsneh anishsneh   4096 May 17 15:37 javadoc
  drwxr-xr-x. 3 anishsneh anishsneh   4096 May 17 15:37 lib
  drwxr-xr-x. 3 anishsneh anishsneh   4096 May 17 15:37 pylib
  drwxr-xr-x. 4 anishsneh anishsneh   4096 May 17 15:37 tools
  drwxr-xr-x. 2 anishsneh anishsneh   4096 May 17 15:37 bin
  drwxrwxr-x. 2 anishsneh anishsneh   4096 May 17 15:51 logs
  drwxrwxr-x. 5 anishsneh anishsneh   4096 May 17 15:51 data
  drwxr-xr-x. 3 anishsneh anishsneh   4096 May 17 16:46 conf
  

• Repeat above steps for all the three nodes.

Cassandra | Quick Dive

In the previous post we learnt about the basics of Cassandra and CAP theorem, in this post we will have a closer look at Cassandra data model and working of Cassandra.

Data Model

Cassandra is can be defined as a hybrid between a key-value and a column-oriented database. In Cassandra world the a data model can be seen as a map which is distributed across the cluster. In other words a table in Cassandra is a distributed multi-dimensional map indexed by a key.

Cassandra Data Model

Cassandra | Internals

In the previous post we learnt about Cassandra data model and replication concepts, in this post we will look the Cassandra architecture and read/write internals.

Architecture | Highlights

• Cassandra was designed after considering all the system/hardware failures that do occur in real world.
• Peer-to-peer, distributed system in which all nodes are alike hence reults in read/write anywhere design.
• Data is transparently partitioned among all nodes in the cluster.
• Custom data replication is provided out of the box to ensure fault tolerance.
• In Cassandra cluster each node communicates with other through the GOSSIP protocol, which exchanges information across the cluster every second.
• A commit log is used on each node to capture write activity. Data durability is assured.
• At the same time data also written to an in-memory structure (memtable) and then to disk once the memory structure is full (an SStable).
• A row in a column family is indexed by its key. Other columns may be indexed as well, we need indexes to quickly search from cassandra. Note that in Cassandra indexes are virtually another tables.
• Consistency can be choosen between strong and eventual (from all to any node responding) depending on the need. It can be done on a per-request basis, and for both reads and writes.
• Provides data compression out of the box. It uses Google's Snappy data compression algorithm, compresses data on a per column family level. There are not known performance penalty in compression.

Flume | Setup

We learnt about Flume in previous post. We will setup and run Flume agent with Avro source and a Java based client here.

Installation

For agent installation we will use one the three nodes setup earlier for agent JVM and all three HDFS nodes for sink (as setup described in earlier post ).
We will setup single agent :

Note that here we are using following details for installation (for complete setup):

     - Installation base directory:  

• /home/anishsneh/installs

     - Installation user name:

• anishsneh

     - Hostnames: 

• server01

Steps to install Flume NG agent:

1. Install Flume - we will use Apache Flume 1.5.0.1 (with Hadoop2)
   • Download apache-flume-1.5.0.1-bin.tar.gz from Flume Website, note that we are using Hadoop 2 for sink
   • Extract downloaded package to anishsneh@server01:/home/anishsneh/installs, such that we have:

          [anishsneh@server01 installs]$ ls -ltr apache-flume-1.5.0.1-bin
          total 128
          -rw-r--r--.  1 anishsneh anishsneh  1779 Mar 28 15:15 README
          -rw-r--r--.  1 anishsneh anishsneh  6172 Mar 28 15:15 DEVNOTES
          -rw-r--r--.  1 anishsneh anishsneh 22517 May  6 16:29 LICENSE
          -rw-r--r--.  1 anishsneh anishsneh 61591 Jun 10 13:56 CHANGELOG
          -rw-r--r--.  1 anishsneh anishsneh   249 Jun 10 14:08 NOTICE
          -rw-r--r--.  1 anishsneh anishsneh  1591 Jun 10 14:08 RELEASE-NOTES
          drwxr-xr-x. 10 anishsneh anishsneh  4096 Jun 10 15:10 docs
          drwxrwxr-x.  2 anishsneh anishsneh  4096 Sep 17 14:59 lib
          drwxrwxr-x.  2 anishsneh anishsneh  4096 Sep 17 14:59 tools
          drwxr-xr-x.  2 anishsneh anishsneh  4096 Sep 17 14:59 bin
          drwxr-xr-x.  2 anishsneh anishsneh  4096 Sep 17 14:59 conf
         

   • Create hdfs://server01:9000/data/flume directory on HDFS and change its permissions to 777 on server01 (for this demo)

          [anishsneh@server01 installs]$ hadoop fs -mkdir /data/flume
         

          [anishsneh@server01 installs]$ hadoop fs -chmod 777 /data/flume
         

Kafka | Setup

We learnt about Kafka in previous post. We will setup and run three node Kafka cluster (fully distributed) here.

Installation

For installation we will use three CentOS VMs which we configured in earlier post. We will setup three node Kafka cluster.

Note that here we are using following details for installation (for complete Kafka setup):

     - Installation base directory:  

• /home/anishsneh/installs

     - Installation user name:

• anishsneh

     - Hostnames: 

• server01 (broker 1)
• server02 (broker 2)
• server03 (broker 3)

Steps to install Kafka:

1. Install Kafka We will use Kafka 0.8.1.1 (kafka_2.10-0.8.1.1.tgz) built using Scala 2.10
   • Download kafka_2.10-0.8.1.1.tgz from Apache Kafka webpage, note that we are using Kafka 0.8.1.1 which is compiled using Scala 2.10
   • Extract downloaded package to /home/anishsneh/installs, such that we have:

     [anishsneh@server01 installs]$ ls -ltr kafka_2.10-0.8.1.1
     total 28
     -rw-rw-r--. 1 anishsneh anishsneh   162 Apr 22 11:37 NOTICE
     -rw-rw-r--. 1 anishsneh anishsneh 11358 Apr 22 11:37 LICENSE
     drwxr-xr-x. 2 anishsneh anishsneh  4096 Apr 22 12:26 libs
     drwxr-xr-x. 2 anishsneh anishsneh  4096 Apr 22 12:26 config
     drwxr-xr-x. 3 anishsneh anishsneh  4096 Apr 22 12:26 bin
     

   • Repeat above steps for all the three hosts.

HBase | Phoenix

We setup and started HBase cluster previous post. We will write a JDBC based client using Apache Phoenix here.

Apache Phoenix

Apache Phoenix is a JDBC skin on HBase client which turns HBase into a SQL supported database. The driving force behind Phoenix development was to use a well-under stood language like SQL to make it easier for people to use HBase instead of learning another proprietary API. It was originally it was developed by salesforce.com as as a Java/JDBC layer enabling developers to run SQL queries on Apache HBase and later it was open sourced and moved under Apache umbrella.
As per Apache documentation "Apache Phoenix is a SQL skin over HBase delivered as a client-embedded JDBC driver targeting low latency queries over HBase data. Apache Phoenix takes your SQL query, compiles it into a series of HBase scans and orchestrates the running of those scans to produce regular JDBC result sets."
It is entirely written in Java and provides a client-embeddable JDBC driver; It has its own query engine, co—processors and meta-data. Phoenix is used internally by salesforce.com for low latency queries in the order of milliseconds for simple queries or seconds when tens of millions of rows are processed, according to the project's description.
The Phoenix query engine transforms SQL query to HBase scans, executes using co-processors and produces JDBC result sets. Under the hood it compiles queries into native HBase calls (there is NO map-reduce involved)
Note that Phoenix JDBC is developed only for HBase and restricted to HBase ONLY

HBase | Setup

We learnt about HBase in previous post. We will setup and run client on an HBase cluster(fully distributed) here.

Installation

For installation we will use three node Hadoop - YARN cluster (as setup/described in earlier post ).
We will setup three node HBase cluster:

Note that here we are using following details for installation (for complete setup):

     - Installation base directory:  

• /home/anishsneh/installs

     - Installation user name:

• anishsneh

     - Hostnames: 

• server01 (master+slave)
• server02 (only slave)
• server03 (only slave)

Steps to install HBase (on the top of Hadoop 2 cluster):

1. Install HBase - we will use HBase 0.98.4 (Hadoop2)
   • Download hbase-0.98.4-hadoop2-bin.tar.gz from HBase Website, note that we are using Hadoop 2 version of HBase binary
   • Extract downloaded package to /home/anishsneh/installs, such that we have:

     [anishsneh@server01 installs]$ ls -ltr hbase-0.98.4-hadoop2
     total 172
     -rw-r--r--.  1 anishsneh anishsneh    897 Jun  6 10:33 NOTICE.txt
     -rw-r--r--.  1 anishsneh anishsneh  11358 Jun  6 10:33 LICENSE.txt
     -rw-r--r--.  1 anishsneh anishsneh   1377 Jul 14 18:23 README.txt
     drwxr-xr-x.  2 anishsneh anishsneh   4096 Jul 14 18:23 conf
     drwxr-xr-x.  4 anishsneh anishsneh   4096 Jul 14 18:23 bin
     -rw-r--r--.  1 anishsneh anishsneh 134544 Jul 14 18:27 CHANGES.txt
     drwxr-xr-x.  7 anishsneh anishsneh   4096 Jul 14 19:37 hbase-webapps
     drwxr-xr-x. 29 anishsneh anishsneh   4096 Jul 14 19:45 docs
     drwxrwxr-x.  3 anishsneh anishsneh   4096 Sep  7 14:47 lib
     

   • Repeat above steps for all the three hosts.
   • Create hdfs://server01:9000/data/hbase directory on HDFS and change its permissions to 777 (for this demo)
   • Create /home/anishsneh/installs/tmp/hbase directory on LFS in all of the three servers (i.e. server01, server02, server03)

Spring Integration | Demo

As explained in previous post Spring Integration application can run inside any spring application container. It may be a web application (loading context via web.xml) or a standalone application (loading context via ClassPathXmlApplicationContext.class).

We will start with a quick file watcher example in which we will implement a file adaptor which will process incoming files asynchronously.

We will run the application as a Java application using main() function and ClassPathXmlApplicationContext.class

Our project will have following structure:
 

Hadoop | MapReduce API

MapReduce is a massively scalable, parallel processing framework that works in tandem with HDFS. MapReduce and Hadoop enable compute to execute at the location of the data, rather than moving data to the compute location; data storage and computation coexist on the same physical nodes in the cluster.

Hadoop MapReduce is an open source implementation of MapReduce

MapReduce (older API implementation with Hadoop 0.20 - just for conceptual reference)

Pig | High Level MapReduce

Pig is a high-level platform/scripting-language for creating/running MapReduce programs used with Hadoop. The language used in this platform is called Pig Latin. Pig Latin allows user to write complex MapReduce transformations using a simple scripting language. This language defines a set of transformations on a data set such as aggregate, join and sort conceptually similar to SQL for RDBMS systems.

Pig have rich set of functions, Pig Latin can be also be extended using UDF (User Defined Functions) which can be written in Java, Python, JavaScript, Ruby or Groovy as per custom requirements and then call directly from the language. Detailed list of Pig built in functions can be found here.

Pig is complete in that you can do all the required data manipulations in Apache Hadoop with Pig. It simplifies the complex operations like joins and filters etc. for performing query operations in hadoop.

Wide usage of Pig includes:

• Extract Transform Load (ETL) e.g. Processing large amounts of log data, clean bad entries, join with other data-sets.
• Research of "raw" Information e.g. User Audit Logs where Schema maybe unknown or inconsistent

Hadoop2 vs Hadoop1

Hadoop2 was a complete overhaul of Hadoop1, in Hadoop2 ASF introduced MapReduce 2.0 (MRv2) or Apache Hadoop YARN. It is a sub-project of Hadoop at the Apache Software Foundation introduced in Hadoop 2.0 that separates the resource management and processing components.

The main differences are categorized below:

Daemons

Daemons	Hadoop 1	Hadoop 2
HDFS	NameNode SecondaryNamenode DataNode	NameNode CheckpointNode (similar to former SecondaryNamenode) DataNode
Processing	MR1 JobTracker TaskTracker	MR2 (YARN) ResourceManager [one per cluster] NodeManager [many per cluster, one per node] Application Master [many per cluster]

 

Hadoop | MapReduce

In the previous post, we looked at the first component of Hadoop framework i.e. HDFS and its key features. Now we will see the concept of MapReduce frameworks.

Before we start with MapReduce, we need to understand a very important core concept of Hadoop (HDFS+MapReduce) framework known as Data Locality.

• Data locality concept can be described as "bringing the compute to the data." In other words, whenever we use a MapReduce program on a particular part of HDFS data, we want to run that program on the node, or machine, that actually stores this data in HDFS. Doing so allows processes to be run much faster since it prevents us having to move large amounts of data around.
• Hadoop tends to runs map tasks on nodes where the data is present locally to avoid network and inter-node communication latency as much as possible. As the input data is split into pieces and provided to map tasks, hence it is preferred that all the data given to respective map task is available on a single node.

Since HDFS only guarantees data having size equal to its block size (64M) to be present on one node, it is advised/advocated to have the split size equal to the HDFS block size so that the map task can take advantage of this data localization.

MapReduce is a framework/programming-model that allows developers to write programs that process massive amounts of unstructured data in parallel across a distributed cluster of processors or stand-alone computers.
A MapReduce program is composed of a map() function that performs filtering and sorting (such as sorting students by first name into queues, one queue for each name) and a reduce() function that performs a summary operation.

A typical Hadoop usage pattern involves three stages:

• Loading data into HDFS
• MapReduce operations.
• Retrieving results from HDFS.

Let's start with Apache Hadoop MapReduce concepts.

Hive | Setup & Hands On

We learnt about Hive in previous post. We will setup and run Hive (with MySQL based metastore) here. Note that we will use one of the machines used in previous post hence JAVA_HOME and HADOOP_HOME related variables are assumed to be set in ~/.bashrc

Installation

For installation we will use master node (note that here in this demo will run HiveQL using Hive commandline console only, hence we will just install on master node). We will use following details for installation:

• Installation base directory:
• /home/anishsneh/installs
• Installation user name:
• anishsneh
• Hadoop details (will use same Hadoop Cluster configured in previous post) with address:
•  hdfs://server01:9000
• We will use MySQL based metastore with following details (note that we need to install/configure a MySQL server with following details):
• Server name: server01
• Server URL: jdbc:mysql://localhost:3306
• Database name: hive
• MySQL username: hiveuser
• MySQL password: Welcome1hive

Let's install and configure metastore database and Hive:

• Install Metastore Database
• Install MySQL server on server01 (using yum or any convenient method)
• Use root password as "Welcome1root"
• Configure Metastore Database
• Connect and configure MySQL metastore as follows:
  

  [anishsneh@server01 installs]$ mysql -uroot -pWelcome1root
  Warning: Using a password on the command line interface can be insecure.
  Welcome to the MySQL monitor.  Commands end with ; or \g.
  Your MySQL connection id is 5
  Server version: 5.6.17 MySQL Community Server (GPL)
  
  Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.
  
  Oracle is a registered trademark of Oracle Corporation and/or its
  affiliates. Other names may be trademarks of their respective
  owners.
  
  Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
  
  mysql> create database hive;
  Query OK, 1 row affected (0.26 sec)
  
  mysql> create user 'hiveuser'@'%' IDENTIFIED BY 'Welcome1hive';
  Query OK, 0 rows affected (0.07 sec)
  
  mysql> GRANT all on *.* to 'hiveuser'@localhost identified by 'Welcome1hive';
  Query OK, 0 rows affected (0.00 sec)
  
  mysql> flush privileges;
  Query OK, 0 rows affected (0.00 sec)
  

HBase | Distributed, Scalable, NoSQL

HBase is an open source, non-relational, column-oriented distributed database modeled after Google’s BigTable and is written in Java, it runs on top of HDFS. It can be seen as a distributed, multidimensional, sorted map with sparse nature. It provides realtime random read/write access to data stored in HDFS
HBase falls under "NoSQL" umbrella. Technically speaking, HBase is really more a "Data Store" than "Data Base" because it lacks many of the features we find in an RDBMS, such as typed columns, secondary indexes, triggers, and advanced query languages, etc.
HBase is not a direct replacement for a classic SQL Database, although recently its performance has improved. HBase has many features which supports efficient scaling. HBase clusters expand by adding RegionServers that are hosted on commodity class servers (running on the top of Hadoop). If a cluster expands from 10 to 20 RegionServers, for example, it doubles both in terms of storage and as well as processing capacity
Mainly HBase is used to power websites/products e.g. StumbleUpon and Facebook's Messages Storing data that’s also used as a sink or a source to analytical jobs (usually MapReduce)
Before we select HBase in our application, we need to keep following things in mind:

• We need to make sure that we have enough data. If we have hundreds of millions or billions of rows, then HBase is a good candidate. If we only have a few thousand/million rows, then using a traditional RDBMS might be a better choice due to the fact that all of our data might wind up on a single node (or two) and the rest of the cluster may be sitting idle.
• We need to make sure we can live without all the extra features that an RDBMS provides (e.g. typed columns, secondary indexes, transactions, advanced query languages etc.) An application built against an RDBMS cannot be "ported" to HBase by simply changing a JDBC driver, for example. Consider moving from an RDBMS to HBase as a complete redesign as opposed to a port.
• We need to make sure we have enough hardware; Even HDFS doesn’t do well with anything less than 5 DataNodes. HBase can run quite well stand-alone on a laptop - but this should be considered a development configuration only.

HBase itself is a vast implementation, we will see the basic concepts here. We will also setup a fully distributed HBase cluster running on the top of HDFS (three nodes) and run fundamental CRUD operations.

Key Features

• Strongly consistent and random reads/writes. 
• Horizontal scalability and Automatic sharding. 
• Automatic RegionServer failover. 
• Hadoop/HDFS Integration with MapReduce.
• Native Java Client API (HTable). 
• Support for Filters, Operational Management. 
• Multiple clients like its native Java library. Thrift, and REST. 
• Few third party clients are (each of them has it's own advantages) :
• Apache Phoenix (JDBC layer for HBase)
• Stumbleupon Asynchbase (asynchronous, non-blocking)
• Kundera(JPA 1.0 ORM library)

Hive | High Level MapReduce

Apache Hive is a data warehouse solution built on the top of HDFS to facilitate querying and managing large datasets preset in HDFS and compatible file systems such as Amazon S3 filesystem.

It is a query engine wrapper built on top of Map Reduce, it is considered as default data warehousing tool of Hadoop Ecosystem. It provides HiveQL, which is very similar to SQL. Hive hides the Hadoop complexity is from end users.

HiveQL is a SQL like language which supports JDBC drivers and interactive SQL queries for large volumes of data in HDFS.

Hive provides external interfaces like command line (CLI) and web UI, and application programming interfaces (API) like JDBC and ODBC.

It was originally developed by Facebook to manager their large volume datasets and later contributed to Apache. When we think of Hadoop data warehousing, Hive becomes an important contituent of Hadoop Ecosystem.

HiveQL prvides support for adhoc queries, schema on read and transparently converting queries to map/reduce (as per underlying job infrastructure e.g. Hadoop MapRed, Spark or Tez). Hive compiles queries into mapreduce jobs and run them into Hadoop cluster.

Hive is considered as a standard for interactive SQL queries over petabytes of data in Hadoop. It easily integrates with other data center technologies via standard JDBC interface. It needs a metastore which stores the metadata for Hive tables and partitions in a relational database, and provides clients access to this information via the metastore service API.

Note that Hive does NOT provide low latency or real-time queries, in Hive even small queries may take minutes. It is basically designed for scalability and ease-of-use rather than low latency responses.

Key Features

• HiveQL based SQL like interactive queries with JDBC support.
• HiveQL supports FILTERS, JOINS, ORDER BY, GROUP BY clauses out of the box.
• HiveQL allows traditional map/reduce programmers to plug in custom mappers and reducers when it is difficult to write respective queries/logic in HiveQL.
• Different storage types support such as plain text, RCFile, HBase, ORC, and others.
• Metastore or Metadata storage in an RDBMS, significantly reducing the time to perform semantic checks during query execution.
• Transparently converts queries to map/reduce jobs and run them into Hadoop cluster.
• Supports schema on read.
• Supports user defined functions (UDF - written in Java and referenced by a HiveQL query) to handle use-cases not supported by built-in functions.
• Supports complex data types such as STRUCT, MAP, ARRAY.
• Supports indexing to provide acceleration and fast retrievals, index type including compaction and Bitmap index