Cascading is a software

abstraction layer In computing, an abstraction layer or abstraction level is a way of hiding the working details of a subsystem. Examples of software models that use layers of abstraction include the OSI model for network protocols, OpenGL, and other graphics libra ...

for Apache Hadoop and

Apache Flink Apache Flink is an open-source, unified stream-processing and batch-processing framework developed by the Apache Software Foundation. The core of Apache Flink is a distributed streaming data-flow engine written in Java and Scala. Flink execu ...

. Cascading is used to create and execute complex data processing workflows on a Hadoop cluster using any JVM-based language (

Java Java (; id, Jawa, ; jv, ꦗꦮ; su, ) is one of the Greater Sunda Islands in Indonesia. It is bordered by the Indian Ocean to the south and the Java Sea to the north. With a population of 151.6 million people, Java is the world's mo ...

JRuby JRuby is an implementation of the Ruby programming language atop the Java Virtual Machine, written largely in Java. It is free software released under a three-way EPL/ GPL/ LGPL license. JRuby is tightly integrated with Java to allow the embeddi ...

Clojure Clojure (, like ''closure'') is a dynamic and functional dialect of the Lisp programming language on the Java platform. Like other Lisp dialects, Clojure treats code as data and has a Lisp macro system. The current development process is ...

, etc.), hiding the underlying complexity of

MapReduce MapReduce is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster. A MapReduce program is composed of a ''map'' procedure, which performs filteri ...

jobs. It is open source and available under the Apache License. Commercial support is available from Driven, Inc. Cascading was originally authored by Chris Wensel, who later founded Concurrent, Inc, which has been re-branded as Driven. Cascading is being actively developed by the community and a number of add-on modules are available.

Architecture

To use Cascading, Apache Hadoop must also be installed, and the Hadoop job .jar must contain the Cascading .jars. Cascading consists of a data processing API, integration API, process planner and process scheduler. Cascading leverages the scalability of Hadoop but abstracts standard data processing operations away from underlying map and reduce tasks. Developers use Cascading to create a .jar file that describes the required processes. It follows a ‘source-pipe-sink’ paradigm, where data is captured from sources, follows reusable ‘pipes’ that perform data analysis processes, where the results are stored in output files or ‘sinks’. Pipes are created independent from the data they will process. Once tied to data sources and sinks, it is called a ‘flow’. These flows can be grouped into a ‘cascade’, and the process scheduler will ensure a given flow does not execute until all its dependencies are satisfied. Pipes and flows can be reused and reordered to support different business needs. Developers write the code in a JVM-based language and do not need to learn MapReduce. The resulting program can be regression tested and integrated with external applications like any other Java application. Cascading is most often used for ad targeting, log file analysis, bioinformatics, machine learning,

predictive analytics Predictive analytics encompasses a variety of statistical techniques from data mining, predictive modeling, and machine learning that analyze current and historical facts to make predictions about future or otherwise unknown events. In busin ...

, web content mining, and extract, transform and load (ETL) applications.

Uses of Cascading

Cascading was cited as one of the top five most powerful Hadoop projects by SD Times in 2011, as a major open source project relevant to bioinformatics and is included in Hadoop: A Definitive Guide, by Tom White. The project has also been cited in presentations, conference proceedings and Hadoop user group meetings as a useful tool for working with Hadoop and with

Apache Spark Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance. Originally developed at the University of Califor ...

* MultiTool on

Amazon Web Services Amazon Web Services, Inc. (AWS) is a subsidiary of Amazon.com, Amazon that provides Software as a service, on-demand cloud computing computing platform, platforms and Application programming interface, APIs to individuals, companies, and gover ...

was developed using Cascading. * LogAnalyzer for

Amazon CloudFront Amazon CloudFront is a content delivery network (CDN) operated by Amazon Web Services. Content delivery networks provide a globally-distributed network of proxy servers that cache content, such as web videos or other bulky media, more local ...

was developed using Cascading. * BackType - social analytics platform * Etsy - marketplace * FlightCaster - predicting flight delays * Ion Flux - analyzing DNA sequence data * RapLeaf - personalization and recommendation systems * Razorfish - digital advertising

Domain-Specific Languages Built on Cascading

* PyCascading - by Twitter, available on GitHub * Cascading.jruby - developed by Gregoire Marabout, available on GitHub * Cascalog - authored by Nathan Marz, available on GitHub * ScaldingScalding
/ref> - A Scala API for Cascading. Makes it easier to transition Cascading/Scalding code to Spark. By Twitter, available on GitHub

References

{{Reflist, 30em

External links

Official website
Free software programmed in Java (programming language) Free system software Cloud infrastructure Hadoop