Thursday, July 10, 2014

Spring Integration | Integration Patterns

Enterpise Integration Strategies

In present era no system in the world can be self sufficient to meet all business needs of its own. It needs to interact to number of interconnected systems to achieve business goals . For this interaction it needs to share data/information among those system, mainly we can divide this Interaction or integration strategies in the following categories :
  • Shared File Systems
    Two or more applications share a common file system; one may write to it while the other may poll the file system to read it. The sender and receiver are decoupled in this case. This solution certainly works, but has drawbacks like performance, reliability, and dependency on the File system. 
  • Shared Database
    In this strategy, applications share the data from a single database. One application writes data to the table while the other simply reads from table. The drawback is that this setup forces applications to use a unified schema throughout the organization. Shared databases also pose a few other issues, such as network lag and lock contention. 
  • Remote Procedure Calls
    If data or a process needs to be shared across an organization, one way to expose this functionality is through a remote service, for example an EJB or a SOAP/REST service allows functionality to be exposed to the rest of the enterprise. Using a service, the implementation is encapsulated, allowing the different applications to change the underlying implementation without affecting any integration solution as long as the service interface has not changed. The thing to remember about a service is that the integration is synchronous: both the client and the service must be available for integration to occur, and they must know about each other.  
  • Messaging
    This strategy mainly encourages and supports sender—receiver decoupling.,A sender application sends a piece of data enclosed in a message to a messaging middleman and forgets about it. A consumer consumes the message whenever it can and begins its own workflow. One of the advantages of using Messaging as the medium is that the sender and receiver are decoupled completely. These styles are disparate because no one solution will work all the time. To build a generic solution, middleware thinkers thought of a system based on these I patterns, typically called an Enterprise Service Bus (ESB). An ESB is the ultimate middle-man; it knows how to talk all languages, over all protocols, and mediate messages being passed. JEE and Spring played a key role in simplifying the enterprise programming model. JEE standardized and made commodities of solutions to common enterprise problems like database access, remote procedure invocation, transactions, authentication, directory services, etc. EAI solutions have no direct support in JEE, aside from the basics: RFC, messaging. There are many patterns for Enterprise Application Integration, and just as many protocols that need to be dealt with.

SEDA (Staged Event-Driven Architecture) - An Integration Pattern

Staged event-driven architecture (SEDA) is an approach to build a system that can support massive concurrency without incurring many of the issues involved with using a traditional thread and event based approach. The basic premise is to break the application logic into a series of stages connected by event queues. Each stage may be conditioned to handle increased load by increasing the number of threads for that stage (increasing concurrency).

Kafka | A Messaging Rethought

As per ASF "Apache Kafka is publish-subscribe messaging rethought as a distributed commit log". It is an open-source message broker written in Scala. It provides a unified, high-throughput, low-latency platform for handling real-time data feeds. The design is heavily influenced by transactions logs hence defined as "messaging rethought as a distributed commit log".

Kafka was originally developed by Linkedln, open sourced in early 2011 and later was contributed to ASF. Kafka is considered as The Next Generation Distributed Messaging System.

Kafka - High Level View


Hadoop | HDFS

Apache Hadoop is an open-source software framework for storage and large-scale distributed computing on the top of commodity hardware. It basically provides us two things:
  • A distributed filesystem called HDFS (Hadoop Distributed File System)
  • A framework and API for building and running MapReduce jobs
HDFS is a file system similarly to a regular Unix filesystem with distributed data storage across several machines. It is not intended as a replacement to a regular filesystem, but rather as a filesystem running on the top of regular low level file system (it is like a layer for large distributed systems to use). It has in built mechanisms to handle machine outages, and is optimized for throughput rather than latency.

Key Features

  • HDFS stores files in blocks typically at least 64 MB in size, much larger than the 4-32 KB seen in most filesystems.
  • HDFS is optimized for throughput over latency; it is very efficient at streaming read requests for large files but poor at seek requests for many small ones.
  • HDFS is optimized for workloads that are generally of the write-once and read-many type.
  • Each storage node runs a process called a DataNode that manages the blocks on that host, and these are coordinated by a master NameNode process running on a separate host.
  • Instead of handling disk failures by having physical redundancies in disk arrays or similar strategies, HDFS uses replication (NameNode takes care of replication, failover)
  • NameNode constantly monitors reports sent by each DataNode to ensure that failures have not dropped any block below the desired replication factor. If this does happen, it schedules the addition of another copy within the cluster.