In
software engineering
Software engineering is a systematic engineering approach to software development.
A software engineer is a person who applies the principles of software engineering to design, develop, maintain, test, and evaluate computer software. The term '' ...
, a pipeline consists of a chain of processing elements (
processes,
threads,
coroutine
Coroutines are computer program components that generalize subroutines for non-preemptive multitasking, by allowing execution to be suspended and resumed. Coroutines are well-suited for implementing familiar program components such as cooperative ...
s,
functions, ''etc.''), arranged so that the output of each element is the input of the next; the name is by analogy to a physical
pipeline
Pipeline may refer to:
Electronics, computers and computing
* Pipeline (computing), a chain of data-processing stages or a CPU optimization found on
** Instruction pipelining, a technique for implementing instruction-level parallelism within a s ...
. Usually some amount of
buffering is provided between consecutive elements. The information that flows in these pipelines is often a
stream
A stream is a continuous body of water, body of surface water Current (stream), flowing within the stream bed, bed and bank (geography), banks of a channel (geography), channel. Depending on its location or certain characteristics, a stream ...
of
records
A record, recording or records may refer to:
An item or collection of data Computing
* Record (computer science), a data structure
** Record, or row (database), a set of fields in a database related to one entity
** Boot sector or boot record, ...
,
bytes
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable unit ...
, or
bits, and the elements of a pipeline may be called
filters
Filter, filtering or filters may refer to:
Science and technology
Computing
* Filter (higher-order function), in functional programming
* Filter (software), a computer program to process a data stream
* Filter (video), a software component tha ...
; this is also called the pipes and filters
design pattern
A design pattern is the re-usable form of a solution to a design problem. The idea was introduced by the architect Christopher Alexander and has been adapted for various other disciplines, particularly software engineering. The " Gang of Four" b ...
. Connecting elements into a pipeline is analogous to
function composition
In mathematics, function composition is an operation that takes two functions and , and produces a function such that . In this operation, the function is applied to the result of applying the function to . That is, the functions and ...
.
Narrowly speaking, a pipeline is linear and one-directional, though sometimes the term is applied to more general flows. For example, a primarily one-directional pipeline may have some communication in the other direction, known as a ''
return channel In communications systems, the return channel (also reverse channel or return link) is the transmission link from a user terminal to the central hub. Return links are often, but not always, slower than the corresponding forward links. Examples whe ...
'' or ''backchannel,'' as in
the lexer hack In computer programming, the lexer hack is a common solution to the problems in parsing ANSI C, due to the reference grammar being context-sensitive. In C, classifying a sequence of characters as a variable name or a type name requires contextual i ...
, or a pipeline may be fully bi-directional. Flows with one-directional tree and
directed acyclic graph
In mathematics, particularly graph theory, and computer science, a directed acyclic graph (DAG) is a directed graph with no directed cycles. That is, it consists of vertices and edges (also called ''arcs''), with each edge directed from one ve ...
topologies behave similarly to (linear) pipelines – the lack of cycles makes them simple – and thus may be loosely referred to as "pipelines".
Implementation
Pipelines are often implemented in a
multitasking OS, by launching all elements at the same time as processes, and automatically servicing the data read requests by each process with the data written by the upstream process – this can be called a ''multiprocessed pipeline.'' In this way, the
CPU will be naturally switched among the processes by the
scheduler
A schedule or a timetable, as a basic time-management tool, consists of a list of times at which possible tasks, events, or actions are intended to take place, or of a sequence of events in the chronological order in which such things are i ...
so as to minimize its idle time. In other common models, elements are implemented as lightweight threads or as coroutines to reduce the OS overhead often involved with processes. Depending upon the OS, threads may be scheduled directly by the OS or by a thread manager. Coroutines are always scheduled by a coroutine manager of some form.
Usually, read and write requests are blocking operations, which means that the execution of the source process, upon writing, is suspended until all data could be written to the destination process, and, likewise, the execution of the destination process, upon reading, is suspended until at least some of the requested data could be obtained from the source process. This cannot lead to a
deadlock
In concurrent computing, deadlock is any situation in which no member of some group of entities can proceed because each waits for another member, including itself, to take action, such as sending a message or, more commonly, releasing a lo ...
, where both processes would wait indefinitely for each other to respond, since at least one of the two processes will soon thereafter have its request serviced by the operating system, and continue to run.
For performance, most operating systems implementing pipes use pipe
buffers, which allow the source process to provide more data than the destination process is currently able or willing to receive. Under most Unices and Unix-like operating systems, a special command is also available which implements a pipe buffer of potentially much larger and configurable size, typically called "buffer". This command can be useful if the destination process is significantly slower than the source process, but it is anyway desired that the source process can complete its task as soon as possible. E.g., if the source process consists of a command which reads an audio track from a
CD and the destination process consists of a command which compresses the
waveform
In electronics, acoustics, and related fields, the waveform of a signal is the shape of its graph as a function of time, independent of its time and magnitude scales and of any displacement in time.David Crecraft, David Gorham, ''Electronic ...
audio data to a format like
MP3
MP3 (formally MPEG-1 Audio Layer III or MPEG-2 Audio Layer III) is a coding format for digital audio developed largely by the Fraunhofer Society in Germany, with support from other digital scientists in the United States and elsewhere. Origin ...
. In this case, buffering the entire track in a pipe buffer would allow the CD drive to spin down more quickly, and enable the user to remove the CD from the drive before the encoding process has finished.
Such a buffer command can be implemented using
system call
In computing, a system call (commonly abbreviated to syscall) is the programmatic way in which a computer program requests a service from the operating system on which it is executed. This may include hardware-related services (for example, acc ...
s for reading and writing data. Wasteful
busy waiting
In computer science and software engineering, busy-waiting, busy-looping or spinning is a technique in which a process repeatedly checks to see if a condition is true, such as whether keyboard input or a lock is available. Spinning can also be used ...
can be avoided by using facilities such as
poll
Poll, polled, or polling may refer to:
Figurative head counts
* Poll, a formal election
** Election verification exit poll, a survey taken to verify election counts
** Polling, voting to make decisions or determine opinions
** Polling places o ...
or
select or
multithreading.
Some notable examples of pipeline software systems include:
*
RaftLib
RaftLib is a portable parallel processing system that aims to provide extreme performance while increasing programmer productivity. It enables a programmer to assemble a massively parallel program (both local and distributed) using simple iostream ...
– C/C++ Apache 2.0 License
VM/CMS and z/OS
CMS Pipelines {{Infobox programming language
, name = Pipelines
, logo = image:pipjarg1.jpeg
, paradigm = Dataflow programming
, year = 1986
, developer = IBM
, designer = John P. Hartmann ( IBM)
, latest_release_version = 1.1.12/0012
, latest_release_date = ...
is a port of the pipeline idea to
VM/CMS
VM (often: VM/CMS) is a family of IBM virtual machine operating systems used on IBM mainframes System/370, System/390, zSeries, System z and compatible systems, including the Hercules emulator for personal computers.
The following versions ...
and
z/OS
z/OS is a 64-bit operating system for IBM z/Architecture mainframes, introduced by IBM in October 2000. It derives from and is the successor to OS/390, which in turn was preceded by a string of MVS versions.Starting with the earliest:
* O ...
systems. It supports much more complex pipeline structures than Unix shells, with steps taking multiple input streams and producing multiple output streams. (Such functionality is supported by the Unix kernel, but few programs use it as it makes for complicated syntax and blocking modes, although some shells do support it via arbitrary
file descriptor
In Unix and Unix-like computer operating systems, a file descriptor (FD, less frequently fildes) is a process-unique identifier (handle) for a file or other input/output resource, such as a pipe or network socket.
File descriptors typically have ...
assignment).
Traditional application programs on IBM mainframe operating systems have no standard input and output streams to allow redirection or piping. Instead of spawning processes with external programs, CMS Pipelines features a lightweight dispatcher to concurrently execute instances of built-in programs to run the pipeline. More than 200 built-in programs that implement typical UNIX utilities and interface to devices and operating system services. In addition to the built-in programs, CMS Pipelines defines a framework to allow user-written
REXX
Rexx (Restructured Extended Executor) is a programming language that can be interpreted or compiled. It was developed at IBM by Mike Cowlishaw. It is a structured, high-level programming language designed for ease of learning and reading. ...
programs with input and output streams that can be used in the pipeline.
Data on IBM mainframes typically resides in a
Record-oriented filesystem
In computer science, a record-oriented filesystem is a file system where data is stored as collections of records. This is in contrast to a byte-oriented filesystem, where the data is treated as an unformatted stream of bytes. There are severa ...
and connected I/O devices operate in record mode rather than stream mode. As a consequence, data in CMS Pipelines is handled in record mode. For text files, a record holds one line of text. In general, CMS Pipelines does not buffer the data but passes records of data in a lock-step fashion from one program to the next. This ensures a deterministic flow of data through a network of interconnected pipelines.
Object pipelines
Beside byte stream-based pipelines, there are also object pipelines. In an object pipeline, processing elements output objects instead of text.
Windows PowerShell
PowerShell is a task automation and configuration management program from Microsoft, consisting of a command-line shell and the associated scripting language. Initially a Windows component only, known as Windows PowerShell, it was made open-sourc ...
includes an internal object pipeline that transfers
.NET objects between functions within the PowerShell runtime.
Channels
Channel, channels, channeling, etc., may refer to:
Geography
* Channel (geography), in physical geography, a landform consisting of the outline (banks) of the path of a narrow body of water.
Australia
* Channel Country, region of outback Austral ...
, found in the
Limbo programming language are other examples of this metaphor.
Pipelines in GUIs
Graphical environments such as
RISC OS
RISC OS is a computer operating system originally designed by Acorn Computers Ltd in Cambridge, England. First released in 1987, it was designed to run on the ARM chipset, which Acorn had designed concurrently for use in its new line of Archim ...
and
ROX Desktop
The ROX Desktop is a graphical desktop environment for the X Window System. It is based on the ROX-Filer which is a drag and drop spatial file manager. It is free software released under the GNU General Public License. The environment was ins ...
also make use of pipelines. Rather than providing a save
dialog box
The dialog box (also called dialogue box (non-U.S. English), message box or simply dialog) is a graphical control element in the form of a small window that communicates information to the user and prompts them for a response.
Dialog boxes ar ...
containing a
file manager
A file manager or file browser is a computer program that provides a user interface to manage files and folders. The most common operations performed on files or groups of files include creating, opening (e.g. viewing, playing, editing or pr ...
to let the user specify where a
program
Program, programme, programmer, or programming may refer to:
Business and management
* Program management, the process of managing several related projects
* Time management
* Program, a part of planning
Arts and entertainment Audio
* Progra ...
should write data, RISC OS and ROX provide a save dialog box containing an
icon
An icon () is a religious work of art, most commonly a painting, in the cultures of the Eastern Orthodox, Oriental Orthodox, and Catholic churches. They are not simply artworks; "an icon is a sacred image used in religious devotion". The most ...
(and a field to specify the name). The destination is specified by
dragging and dropping the icon. The user can drop the icon anywhere an already-saved file could be dropped, including onto icons of other programs. If the icon is dropped onto a program's icon, it's loaded and the contents that would otherwise have been saved are passed in on the new program's standard input stream.
For instance, a user browsing the
world-wide web
The World Wide Web (WWW), commonly known as the Web, is an information system enabling documents and other web resources to be accessed over the Internet.
Documents and downloadable media are made available to the network through web s ...
might come across a .gz compressed image which they want to edit and re-upload. Using GUI pipelines, they could drag the link to their de-archiving program, drag the icon representing the extracted contents to their
image editor
Image editing encompasses the processes of altering images, whether they are digital photographs, traditional photo-chemical photographs, or illustrations. Traditional analog image editing is known as photo retouching, using tools such as a ...
, edit it, open the save as dialog, and drag its icon to their uploading software.
Conceptually, this method could be used with a conventional save dialog box, but this would require the user's programs to have an obvious and easily accessible location in the filesystem that can be navigated to. In practice, this is often not the case, so GUI pipelines are rare.
Other considerations
The name "pipeline" comes from a rough analogy with physical plumbing in that a pipeline usually allows information to flow in only one direction, like water often flows in a pipe.
Pipes and
filters
Filter, filtering or filters may refer to:
Science and technology
Computing
* Filter (higher-order function), in functional programming
* Filter (software), a computer program to process a data stream
* Filter (video), a software component tha ...
can be viewed as a form of
functional programming
In computer science, functional programming is a programming paradigm where programs are constructed by Function application, applying and Function composition (computer science), composing Function (computer science), functions. It is a declar ...
, using byte streams as data objects; more specifically, they can be seen as a particular form of
monad
Monad may refer to:
Philosophy
* Monad (philosophy), a term meaning "unit"
**Monism, the concept of "one essence" in the metaphysical and theological theory
** Monad (Gnosticism), the most primal aspect of God in Gnosticism
* ''Great Monad'', a ...
for
I/O.
"Monadic I/O and UNIX shell programming"
The concept of pipeline is also central to the Cocoon web development framework
A framework is a generic term commonly referring to an essential supporting structure which other things are built on top of.
Framework may refer to:
Computing
* Application framework, used to implement the structure of an application for an op ...
or to any XProc
XProc is a W3C Recommendation to define an XML transformation language to define XML Pipelines.
Below is an example abbreviated XProc file:
This is a pipeline that consists of two atomic steps, XInclude and V ...
(the W3C Standards) implementations, where it allows a source stream to be modified before eventual display.
This pattern encourages the use of text streams as the input and output of programs. This reliance on text has to be accounted when creating graphic
Graphics () are visual images or designs on some surface, such as a wall, canvas, screen, paper, or stone, to inform, illustrate, or entertain. In contemporary usage, it includes a pictorial representation of data, as in design and manufacture, ...
shells to text programs.
See also
* Anonymous pipe
In computer science, an anonymous pipe is a simplex FIFO communication channel that may be used for one-way interprocess communication (IPC). An implementation is often integrated into the operating system's file IO subsystem. Typically a parent ...
* Component-based software engineering
Component-based software engineering (CBSE), also called component-based development (CBD), is a branch of software engineering that emphasizes the separation of concerns with respect to the wide-ranging functionality available throughout a give ...
* Flow-based programming
In computer programming, flow-based programming (FBP) is a programming paradigm that defines application software, applications as networks of "black box" process (computer science), processes, which exchange data across predefined connections by ...
* GStreamer
GStreamer is a pipeline-based multimedia framework that links together a wide variety of media processing systems to complete complex workflows. For instance, GStreamer can be used to build a system that reads files in one format, processes them, ...
for a multimedia framework built on plugin pipelines
* Graphics pipeline
In computer graphics, a computer graphics pipeline, rendering pipeline or simply graphics pipeline, is a conceptual model that describes what steps a graphics system needs to perform to Rendering (computer graphics), render a ...
* Iteratees
* Named pipe
In computing, a named pipe (also known as a FIFO for its behavior) is an extension to the traditional pipe concept on Unix and Unix-like systems, and is one of the methods of inter-process communication (IPC). The concept is also found in OS/2 and ...
, an operating system construct intermediate to anonymous pipe and file.
* Pipeline (computing)
In computing, a pipeline, also known as a data pipeline, is a set of data processing elements connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time- ...
for other computer-related versions of the concept.
* Kahn process networks
A Kahn process network (KPN, or process network) is a distributed ''model of computation'' in which a group of deterministic sequential processes communicate through unbounded first in, first out channels. The model requires that reading from a c ...
to extend the pipeline concept to a more generic directed graph structure
* Pipeline (Unix)
In Unix-like computer operating systems, a pipeline is a mechanism for inter-process communication using message passing. A pipeline is a set of processes chained together by their standard streams, so that the output text of each process (''stdou ...
for details specific to Unix
Unix (; trademarked as UNIX) is a family of multitasking, multiuser computer operating systems that derive from the original AT&T Unix, whose development started in 1969 at the Bell Labs research center by Ken Thompson, Dennis Ritchie, and ot ...
* Plumber
A plumber is a tradesperson who specializes in installing and maintaining systems used for potable (drinking) water, and for sewage and drainage in plumbing systems. – "intelligent pipes" developed as part of Plan 9
* Producer–consumer problem – for implementation aspects of software pipelines
* Software design pattern
In software engineering, a software design pattern is a general, reusable solution to a commonly occurring problem within a given context in software design. It is not a finished design that can be transformed directly into source or machine code ...
* Stream processing
In computer science, stream processing (also known as event stream processing, data stream processing, or distributed stream processing) is a programming paradigm which views data streams, or sequences of events in time, as the central input and ou ...
* XML pipeline
In software, an XML pipeline is formed when XML (Extensible Markup Language) processes, especially XML transformations and XML validations, are connected.
For instance, given two transformations T1 and T2, the two can be connected so that an inpu ...
for processing of XML
Extensible Markup Language (XML) is a markup language and file format for storing, transmitting, and reconstructing arbitrary data. It defines a set of rules for encoding documents in a format that is both human-readable and machine-readable ...
files
Notes
External links
Pipeline Processing.
{{DEFAULTSORT:Pipeline (Software)
Software design patterns
Programming paradigms
Inter-process communication
sv:Vertikalstreck#Datavetenskap