Cassandra | Quick Dive

In the previous post we learnt about the basics of Cassandra and CAP theorem, in this post we will have a closer look at Cassandra data model and working of Cassandra.

Data Model

Cassandra is can be defined as a hybrid between a key-value and a column-oriented database. In Cassandra world the a data model can be seen as a map which is distributed across the cluster. In other words a table in Cassandra is a distributed multi-dimensional map indexed by a key.

Cassandra Data Model

Cassandra | Internals

In the previous post we learnt about Cassandra data model and replication concepts, in this post we will look the Cassandra architecture and read/write internals.

Architecture | Highlights

• Cassandra was designed after considering all the system/hardware failures that do occur in real world.
• Peer-to-peer, distributed system in which all nodes are alike hence reults in read/write anywhere design.
• Data is transparently partitioned among all nodes in the cluster.
• Custom data replication is provided out of the box to ensure fault tolerance.
• In Cassandra cluster each node communicates with other through the GOSSIP protocol, which exchanges information across the cluster every second.
• A commit log is used on each node to capture write activity. Data durability is assured.
• At the same time data also written to an in-memory structure (memtable) and then to disk once the memory structure is full (an SStable).
• A row in a column family is indexed by its key. Other columns may be indexed as well, we need indexes to quickly search from cassandra. Note that in Cassandra indexes are virtually another tables.
• Consistency can be choosen between strong and eventual (from all to any node responding) depending on the need. It can be done on a per-request basis, and for both reads and writes.
• Provides data compression out of the box. It uses Google's Snappy data compression algorithm, compresses data on a per column family level. There are not known performance penalty in compression.

Flume | Setup

We learnt about Flume in previous post. We will setup and run Flume agent with Avro source and a Java based client here.

Installation

For agent installation we will use one the three nodes setup earlier for agent JVM and all three HDFS nodes for sink (as setup described in earlier post ).
We will setup single agent :

Note that here we are using following details for installation (for complete setup):

     - Installation base directory:  

• /home/anishsneh/installs

     - Installation user name:

• anishsneh

     - Hostnames: 

• server01

Steps to install Flume NG agent:

1. Install Flume - we will use Apache Flume 1.5.0.1 (with Hadoop2)
   • Download apache-flume-1.5.0.1-bin.tar.gz from Flume Website, note that we are using Hadoop 2 for sink
   • Extract downloaded package to anishsneh@server01:/home/anishsneh/installs, such that we have:

          [anishsneh@server01 installs]$ ls -ltr apache-flume-1.5.0.1-bin
          total 128
          -rw-r--r--.  1 anishsneh anishsneh  1779 Mar 28 15:15 README
          -rw-r--r--.  1 anishsneh anishsneh  6172 Mar 28 15:15 DEVNOTES
          -rw-r--r--.  1 anishsneh anishsneh 22517 May  6 16:29 LICENSE
          -rw-r--r--.  1 anishsneh anishsneh 61591 Jun 10 13:56 CHANGELOG
          -rw-r--r--.  1 anishsneh anishsneh   249 Jun 10 14:08 NOTICE
          -rw-r--r--.  1 anishsneh anishsneh  1591 Jun 10 14:08 RELEASE-NOTES
          drwxr-xr-x. 10 anishsneh anishsneh  4096 Jun 10 15:10 docs
          drwxrwxr-x.  2 anishsneh anishsneh  4096 Sep 17 14:59 lib
          drwxrwxr-x.  2 anishsneh anishsneh  4096 Sep 17 14:59 tools
          drwxr-xr-x.  2 anishsneh anishsneh  4096 Sep 17 14:59 bin
          drwxr-xr-x.  2 anishsneh anishsneh  4096 Sep 17 14:59 conf
         

   • Create hdfs://server01:9000/data/flume directory on HDFS and change its permissions to 777 on server01 (for this demo)

          [anishsneh@server01 installs]$ hadoop fs -mkdir /data/flume
         

          [anishsneh@server01 installs]$ hadoop fs -chmod 777 /data/flume