![Pipeline](https://upload.wikimedia.org/wikipedia/commons/f/f6/Pipeline.svg)
In
Unix-like
A Unix-like (sometimes referred to as UN*X or *nix) operating system is one that behaves in a manner similar to a Unix system, although not necessarily conforming to or being certified to any version of the Single UNIX Specification. A Unix-li ...
computer
operating system
An operating system (OS) is system software that manages computer hardware, software resources, and provides common services for computer programs.
Time-sharing operating systems schedule tasks for efficient use of the system and may also in ...
s, a pipeline is a mechanism for
inter-process communication
In computer science, inter-process communication or interprocess communication (IPC) refers specifically to the mechanisms an operating system provides to allow the processes to manage shared data. Typically, applications can use IPC, categori ...
using message passing. A pipeline is a set of
process
A process is a series or set of activities that interact to produce a result; it may occur once-only or be recurrent or periodic.
Things called a process include:
Business and management
*Business process, activities that produce a specific se ...
es chained together by their
standard streams
In computer programming, standard streams are interconnected input and output communication channels between a computer program and its environment when it begins execution. The three input/output (I/O) connections are called standard input (stdin ...
, so that the output text of each process (''
stdout
In computer programming, standard streams are interconnected input and output communication channels between a computer program and its environment when it begins execution. The three input/output (I/O) connections are called standard input (stdin ...
'') is passed directly as input (''
stdin
In computer programming, standard streams are interconnected input and output communication channels between a computer program and its environment when it begins execution. The three input/output (I/O) connections are called standard input (stdin ...
'') to the next one. The second process is started as the first process is still executing, and they are executed
concurrently.
The concept of pipelines was championed by
Douglas McIlroy
Malcolm Douglas McIlroy (born 1932) is a mathematician, engineer, and programmer. As of 2019 he is an Adjunct Professor of Computer Science at Dartmouth College.
McIlroy is best known for having originally proposed Unix pipelines and developed se ...
at
Unix
Unix (; trademarked as UNIX) is a family of multitasking, multiuser computer operating systems that derive from the original AT&T Unix, whose development started in 1969 at the Bell Labs research center by Ken Thompson, Dennis Ritchie, and ot ...
's ancestral home of
Bell Labs
Nokia Bell Labs, originally named Bell Telephone Laboratories (1925–1984),
then AT&T Bell Laboratories (1984–1996)
and Bell Labs Innovations (1996–2007),
is an American industrial research and scientific development company owned by mult ...
, during the development of Unix, shaping its
toolbox philosophy. It is named by analogy to a physical
pipeline
Pipeline may refer to:
Electronics, computers and computing
* Pipeline (computing), a chain of data-processing stages or a CPU optimization found on
** Instruction pipelining, a technique for implementing instruction-level parallelism within a s ...
. A key feature of these pipelines is their "hiding of internals" (Ritchie & Thompson, 1974). This in turn allows for more clarity and simplicity in the system.
This article is about
anonymous pipe
In computer science, an anonymous pipe is a simplex FIFO communication channel that may be used for one-way interprocess communication (IPC). An implementation is often integrated into the operating system's file IO subsystem. Typically a parent ...
s, where data written by one process is buffered by the operating system until it is read by the next process, and this uni-directional channel disappears when the processes are completed. This differs from
named pipe
In computing, a named pipe (also known as a FIFO for its behavior) is an extension to the traditional pipe concept on Unix and Unix-like systems, and is one of the methods of inter-process communication (IPC). The concept is also found in OS/2 and ...
s, where messages are passed to or from a pipe that is named by making it a file, and remains after the processes are completed. The standard
shell
Shell may refer to:
Architecture and design
* Shell (structure), a thin structure
** Concrete shell, a thin shell of concrete, usually with no interior columns or exterior buttresses
** Thin-shell structure
Science Biology
* Seashell, a hard ou ...
syntax for
anonymous pipe
In computer science, an anonymous pipe is a simplex FIFO communication channel that may be used for one-way interprocess communication (IPC). An implementation is often integrated into the operating system's file IO subsystem. Typically a parent ...
s is to list multiple commands, separated by vertical bars ("pipes" in common Unix verbiage):
command1 , command2 , command3
For example, to list files in the current directory (), retain only the lines of output containing the string (), and view the result in a scrolling page (), a user types the following into the command line of a terminal:
ls -l , grep key , less
The command
ls -l
is executed as a process, the output (stdout) of which is piped to the input (stdin) of the process for
grep key
; and likewise for the process for
less
. Each
process
A process is a series or set of activities that interact to produce a result; it may occur once-only or be recurrent or periodic.
Things called a process include:
Business and management
*Business process, activities that produce a specific se ...
takes input from the previous process and produces output for the next process via ''
standard streams
In computer programming, standard streams are interconnected input and output communication channels between a computer program and its environment when it begins execution. The three input/output (I/O) connections are called standard input (stdin ...
''. Each
,
tells the shell to connect the standard output of the command on the left to the standard input of the command on the right by an
inter-process communication
In computer science, inter-process communication or interprocess communication (IPC) refers specifically to the mechanisms an operating system provides to allow the processes to manage shared data. Typically, applications can use IPC, categori ...
mechanism called an
(anonymous) pipe, implemented in the operating system. Pipes are unidirectional; data flows through the pipeline from left to right.
Example
Below is an example of a pipeline that implements a kind of
spell checker In software, a spell checker (or spelling checker or spell check) is a software feature that checks for misspellings in a text. Spell-checking features are often embedded in software or services, such as a word processor, email client, electronic di ...
for the
web
Web most often refers to:
* Spider web, a silken structure created by the animal
* World Wide Web or the Web, an Internet-based hypertext system
Web, WEB, or the Web may also refer to:
Computing
* WEB, a literate programming system created by ...
resource indicated by a
URL
A Uniform Resource Locator (URL), colloquially termed as a web address, is a reference to a web resource that specifies its location on a computer network and a mechanism for retrieving it. A URL is a specific type of Uniform Resource Identifie ...
. An explanation of what it does follows.
curl "https://en.wikipedia.org/wiki/Pipeline_(Unix)" ,
sed 's/ a-zA-Z /g' ,
tr 'A-Z ' 'a-z\n' ,
grep ' -z ,
sort -u ,
comm -23 - <(sort /usr/share/dict/words) ,
less
#
curl
cURL (pronounced like "curl", UK: , US: ) is a computer software project providing a library (libcurl) and command-line tool (curl) for transferring data using various network protocols. The name stands for "Client URL".
History
cURL was fi ...
obtains the
HTML
The HyperText Markup Language or HTML is the standard markup language for documents designed to be displayed in a web browser. It can be assisted by technologies such as Cascading Style Sheets (CSS) and scripting languages such as JavaScri ...
contents of a web page (could use
wget
on some systems).
#
sed
sed ("stream editor") is a Unix utility that parses and transforms text, using a simple, compact programming language. It was developed from 1973 to 1974 by Lee E. McMahon of Bell Labs,
and is available today for most operating systems.
sed wa ...
replaces all characters (from the web page's content) that are not spaces or letters, with spaces. (
Newline
Newline (frequently called line ending, end of line (EOL), next line (NEL) or line break) is a control character or sequence of control characters in character encoding specifications such as ASCII, EBCDIC, Unicode, etc. This character, or a ...
s are preserved.)
#
tr
changes all of the uppercase letters into lowercase and converts the spaces in the lines of text to newlines (each 'word' is now on a separate line).
#
grep
grep is a command-line utility for searching plain-text data sets for lines that match a regular expression. Its name comes from the ed command ''g/re/p'' (''globally search for a regular expression and print matching lines''), which has the sam ...
includes only lines that contain at least one lowercase
alphabetical
Alphabetical order is a system whereby character strings are placed in order based on the position of the characters in the conventional ordering of an alphabet. It is one of the methods of collation. In mathematics, a lexicographical order is t ...
character (removing any blank lines).
#
sort
Sort may refer to:
* Sorting, any process of arranging items in sequence or in sets
** Sorting algorithm, any algorithm for arranging elements in lists
** Sort (Unix), a Unix utility which sorts the lines of a file
** Sort (C++), a function in the ...
sorts the list of 'words' into alphabetical order, and the
-u
switch removes duplicates.
#
comm
The command in the Unix family of computer operating systems is a utility that is used to compare two files for common and distinct lines. is specified in the POSIX standard. It has been widely available on Unix-like operating systems since ...
finds lines in common between two files,
-23
suppresses lines unique to the second file, and those that are common to both, leaving only those that are found only in the first file named. The
-
in place of a filename causes
comm
to use its standard input (from the pipe line in this case).
sort /usr/share/dict/words
sorts the contents of the
words
file alphabetically, as
comm
expects, and
<( ... )
outputs the results to a temporary file (via
process substitution
In computing, process substitution is a form of inter-process communication that allows the input or output of a command to appear as a file. The command is substituted in-line, where a file name would normally occur, by the command shell. This all ...
), which
comm
reads. The result is a list of words (lines) that are not found in /usr/share/dict/words.
#
less
Less or LESS may refer to: fewer than,: not as much.
Computing
* less (Unix), a Unix utility program
* Less (stylesheet language), a dynamic stylesheet language
* Large-Scale Scrum (LeSS), a product development framework that extends Scrum
Other ...
allows the user to page through the results.
Pipelines in command line interfaces
All widely used Unix shells have a special syntax construct for the creation of pipelines. In all usage one writes the commands in sequence, separated by the
ASCII
ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding standard for electronic communication. ASCII codes represent text in computers, telecommunications equipment, and other devices. Because of ...
vertical bar
The vertical bar, , is a glyph with various uses in mathematics, computing, and typography. It has many names, often related to particular meanings: Sheffer stroke (in logic), pipe, bar, or (literally the word "or"), vbar, and others.
Usage
...
character
,
(which, for this reason, is often called "pipe character"). The shell starts the processes and arranges for the necessary connections between their standard streams (including some amount of
buffer
Buffer may refer to:
Science
* Buffer gas, an inert or nonflammable gas
* Buffer solution, a solution used to prevent changes in pH
* Buffering agent, the weak acid or base in a buffer solution
* Lysis buffer, in cell biology
* Metal ion buffer
* ...
storage).
Error stream
By default, the
standard error stream
In computer programming, standard streams are interconnected input and output communication channels between a computer program and its environment when it begins execution. The three input/output (I/O) connections are called standard input (stdin ...
s ("
stderr
In computer programming, standard streams are interconnected input and output communication channels between a computer program and its environment when it begins execution. The three input/output (I/O) connections are called standard input (stdin ...
") of the processes in a pipeline are not passed on through the pipe; instead, they are merged and directed to the
console
Console may refer to:
Computing and video games
* System console, a physical device to operate a computer
** Virtual console, a user interface for multiple computer consoles on one device
** Command-line interface, a method of interacting with ...
. However, many shells have additional syntax for changing this behavior. In the
csh shell, for instance, using
, &
instead of
,
signifies that the standard error stream should also be merged with the standard output and fed to the next process. The
Bash
Bash or BASH may refer to:
Arts and entertainment
* ''Bash!'' (Rockapella album), 1992
* ''Bash!'' (Dave Bailey album), 1961
* '' Bash: Latter-Day Plays'', a dramatic triptych
* ''BASH!'' (role-playing game), a 2005 superhero game
* "Bash" ('' ...
shell can also merge standard error with
, &
since version 4.0 or using
2>&1
, as well as redirect it to a different file.
Pipemill
In the most commonly used simple pipelines the shell connects a series of sub-processes via pipes, and executes external commands within each sub-process. Thus the shell itself is doing no direct processing of the data flowing through the pipeline.
However, it's possible for the shell to perform processing directly, using a so-called mill or pipemill (since a
while
command is used to "mill" over the results from the initial command). This construct generally looks something like:
command , while read -r var1 var2 ...; do
# process each line, using variables as parsed into var1, var2, etc
# (note that this may be a subshell: var1, var2 etc will not be available
# after the while loop terminates; some shells, such as zsh and newer
# versions of Korn shell, process the commands to the left of the pipe
# operator in a subshell)
done
Such pipemill may not perform as intended if the body of the loop includes commands, such as
cat
and
ssh
, that read from
stdin
In computer programming, standard streams are interconnected input and output communication channels between a computer program and its environment when it begins execution. The three input/output (I/O) connections are called standard input (stdin ...
: on the loop's first iteration, such a program (let's call it ''the drain'') will read the remaining output from
command
, and the loop will then terminate (with results depending on the specifics of the drain). There are a couple of possible ways to avoid this behavior. First, some drains support an option to disable reading from
stdin
(e.g.
ssh -n
). Alternatively, if the drain does not ''need'' to read any input from
stdin
to do something useful, it can be given
< /dev/null
as input.
As all components of a pipe are run in parallel, a shell typically forks a subprocess (a subshell) to handle its contents, making it impossible to propagate variable changes to the outside shell environment. To remedy this issue, the "pipemill" can instead be fed from a
here document
In computing, a here document (here-document, here-text, heredoc, hereis, here-string or here-script) is a file literal or input stream literal: it is a section of a source code file that is treated as if it were a separate file. The term is also ...
containing a
command substitution
In computing, command substitution is a facility that allows a command to be run and its output to be pasted back on the command line as arguments to another command. Command substitution first appeared in the Bourne shell, introduced with Version ...
, which waits for the pipeline to finish running before milling through the contents. Alternatively, a
named pipe
In computing, a named pipe (also known as a FIFO for its behavior) is an extension to the traditional pipe concept on Unix and Unix-like systems, and is one of the methods of inter-process communication (IPC). The concept is also found in OS/2 and ...
or a
process substitution
In computing, process substitution is a form of inter-process communication that allows the input or output of a command to appear as a file. The command is substituted in-line, where a file name would normally occur, by the command shell. This all ...
can be used for parallel execution.
GNU bash
Bash is a Unix shell and command language written by Brian Fox for the GNU Project as a free software replacement for the Bourne shell. First released in 1989, it has been used as the default login shell for most Linux distributions. Bash was ...
also has a option to disable forking for the last pipe component.
Creating pipelines programmatically
Pipelines can be created under program control. The Unix
pipe()
system call
In computing, a system call (commonly abbreviated to syscall) is the programmatic way in which a computer program requests a service from the operating system on which it is executed. This may include hardware-related services (for example, acc ...
asks the operating system to construct a new
anonymous pipe
In computer science, an anonymous pipe is a simplex FIFO communication channel that may be used for one-way interprocess communication (IPC). An implementation is often integrated into the operating system's file IO subsystem. Typically a parent ...
object. This results in two new, opened file descriptors in the process: the read-only end of the pipe, and the write-only end. The pipe ends appear to be normal, anonymous
file descriptor
In Unix and Unix-like computer operating systems, a file descriptor (FD, less frequently fildes) is a process-unique identifier (handle) for a file or other input/output resource, such as a pipe or network socket.
File descriptors typically have ...
s, except that they have no ability to seek.
To avoid
deadlock
In concurrent computing, deadlock is any situation in which no member of some group of entities can proceed because each waits for another member, including itself, to take action, such as sending a message or, more commonly, releasing a lo ...
and exploit parallelism, the Unix process with one or more new pipes will then, generally, call
fork()
to create new processes. Each process will then close the end(s) of the pipe that it will not be using before producing or consuming any data. Alternatively, a process might create new
threads and use the pipe to communicate between them.
''
Named pipe
In computing, a named pipe (also known as a FIFO for its behavior) is an extension to the traditional pipe concept on Unix and Unix-like systems, and is one of the methods of inter-process communication (IPC). The concept is also found in OS/2 and ...
s'' may also be created using
mkfifo()
or
mknod
In Unix-like operating systems, a device file or special file is an interface to a device driver that appears in a file system as if it were an ordinary file. There are also special files in DOS, OS/2, and Windows. These special files allow an ...
()
and then presented as the input or output file to programs as they are invoked. They allow multi-path pipes to be created, and are especially effective when combined with standard error redirection, or with
tee
A tee is a stand used in sport to support and elevate a stationary ball prior to striking with a foot, club or bat. Tees are used extensively in golf, tee-ball, baseball, American football, and rugby.
Etymology
The word tee is derived from the ...
.
Implementation
In most Unix-like systems, all processes of a pipeline are started at the same time, with their streams appropriately connected, and managed by the
scheduler
A schedule or a timetable, as a basic time-management tool, consists of a list of times at which possible task (project management), tasks, events, or actions are intended to take place, or of a sequence of events in the chronological order ...
together with all other processes running on the machine. An important aspect of this, setting Unix pipes apart from other pipe implementations, is the concept of
buffering: for example a sending program may produce 5000
bytes
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable unit ...
per
second
The second (symbol: s) is the unit of time in the International System of Units (SI), historically defined as of a day – this factor derived from the division of the day first into 24 hours, then to 60 minutes and finally to 60 seconds ...
, and a receiving program may only be able to accept 100 bytes per second, but no data is lost. Instead, the output of the sending program is held in the buffer. When the receiving program is ready to read data, the next program in the pipeline reads from the buffer. If the buffer is filled, the sending program is stopped (blocked) until at least some data is removed from the buffer by the receiver. In Linux, the size of the buffer is 65,536 bytes (64KiB). An open source third-party filter calle
bfris available to provide larger buffers if required.
Network pipes
Tools like
netcat
netcat (often abbreviated to nc) is a computer networking utility for reading from and writing to network connections using TCP or UDP. The command is designed to be a dependable back-end that can be used directly or easily driven by other p ...
and
socat
netcat (often abbreviated to nc) is a computer networking utility for reading from and writing to network connections using Transmission Control Protocol, TCP or User Datagram Protocol, UDP. The command (computing), command is designed to be a ...
can connect pipes to TCP/IP
sockets.
History
The pipeline concept was invented by
Douglas McIlroy
Malcolm Douglas McIlroy (born 1932) is a mathematician, engineer, and programmer. As of 2019 he is an Adjunct Professor of Computer Science at Dartmouth College.
McIlroy is best known for having originally proposed Unix pipelines and developed se ...
and first described in the
man pages
A man page (short for manual page) is a form of software documentation usually found on a Unix or Unix-like operating system. Topics covered include computer programs (including library and system calls), formal standards and conventions, and ev ...
of
Version 3 Unix
The term "Research Unix" refers to early versions of the Unix operating system for DEC PDP-7, PDP-11, VAX and Interdata 7/32 and 8/32 computers, developed in the Bell Labs Computing Sciences Research Center (CSRC).
History
The term ''Research ...
.
McIlroy noticed that much of the time
command shells
In computing, a shell is a computer program that exposes an operating system's services to a human user or other programs. In general, operating system shells use either a command-line interface (CLI) or graphical user interface (GUI), depending ...
passed the output file from one program as input to another.
His ideas were implemented in 1973 when ("in one feverish night", wrote McIlroy)
Ken Thompson
Kenneth Lane Thompson (born February 4, 1943) is an American pioneer of computer science. Thompson worked at Bell Labs for most of his career where he designed and implemented the original Unix operating system. He also invented the B programmi ...
added the
pipe()
system call and pipes to the shell and several utilities in Version 3 Unix. "The next day", McIlroy continued, "saw an unforgettable orgy of one-liners as everybody joined in the excitement of plumbing." McIlroy also credits Thompson with the
,
notation, which greatly simplified the description of pipe syntax in
Version 4.
Although developed independently, Unix pipes are related to, and were preceded by, the 'communication files' developed by Ken Lochner in the 1960s for the
Dartmouth Time Sharing System
The Dartmouth Time-Sharing System (DTSS) is a discontinued operating system first developed at Dartmouth College between 1963 and 1964. It was the first successful large-scale time-sharing system to be implemented, and was also the system for wh ...
.
In
Tony Hoare's communicating sequential processes
In computer science, communicating sequential processes (CSP) is a formal language for describing patterns of interaction in concurrent systems. It is a member of the family of mathematical theories of concurrency known as process algebras, or pro ...
(CSP) McIlroy's pipes are further developed.
[https://swtch.com/~rsc/thread/ Bell Labs and CSP Threads (Russ Cox)]
The robot in the icon for
Apple
An apple is an edible fruit produced by an apple tree (''Malus domestica''). Apple fruit tree, trees are agriculture, cultivated worldwide and are the most widely grown species in the genus ''Malus''. The tree originated in Central Asia, wh ...
's
Automator, which also uses a pipeline concept to chain repetitive commands together, holds a pipe in homage to the original Unix concept.
Other operating systems
This feature of
Unix
Unix (; trademarked as UNIX) is a family of multitasking, multiuser computer operating systems that derive from the original AT&T Unix, whose development started in 1969 at the Bell Labs research center by Ken Thompson, Dennis Ritchie, and ot ...
was borrowed by other operating systems, such as
MS-DOS
MS-DOS ( ; acronym for Microsoft Disk Operating System, also known as Microsoft DOS) is an operating system for x86-based personal computers mostly developed by Microsoft. Collectively, MS-DOS, its rebranding as IBM PC DOS, and a few ope ...
and the
CMS Pipelines {{Infobox programming language
, name = Pipelines
, logo = image:pipjarg1.jpeg
, paradigm = Dataflow programming
, year = 1986
, developer = IBM
, designer = John P. Hartmann ( IBM)
, latest_release_version = 1.1.12/0012
, latest_release_date = ...
package on
VM/CMS
VM (often: VM/CMS) is a family of IBM virtual machine operating systems used on IBM mainframes System/370, System/390, zSeries, System z and compatible systems, including the Hercules emulator for personal computers.
The following versions ...
and
MVS
Multiple Virtual Storage, more commonly called MVS, was the most commonly used operating system on the System/370 and System/390 IBM mainframe computers. IBM developed MVS, along with OS/VS1 and SVS, as a successor to OS/360. It is unrelated ...
, and eventually came to be designated the
pipes and filters design pattern of
software engineering
Software engineering is a systematic engineering approach to software development.
A software engineer is a person who applies the principles of software engineering to design, develop, maintain, test, and evaluate computer software. The term '' ...
.
See also
*
Everything is a file
Everything is a file is an idea that Unix, and its derivatives handle input/output to and from resources such as documents, hard-drives, modems, keyboards, printers and even some inter-process and network communications as simple streams of byte ...
– describes one of the defining features of Unix; pipelines act on "files" in the Unix sense
*
Anonymous pipe
In computer science, an anonymous pipe is a simplex FIFO communication channel that may be used for one-way interprocess communication (IPC). An implementation is often integrated into the operating system's file IO subsystem. Typically a parent ...
– a FIFO structure used for interprocess communication
*
GStreamer
GStreamer is a pipeline-based multimedia framework that links together a wide variety of media processing systems to complete complex workflows. For instance, GStreamer can be used to build a system that reads files in one format, processes them, ...
– a pipeline-based multimedia framework
*
CMS Pipelines {{Infobox programming language
, name = Pipelines
, logo = image:pipjarg1.jpeg
, paradigm = Dataflow programming
, year = 1986
, developer = IBM
, designer = John P. Hartmann ( IBM)
, latest_release_version = 1.1.12/0012
, latest_release_date = ...
*
Iteratee In functional programming, an iteratee is a composable abstraction for incrementally processing sequentially presented chunks of input data in a purely functional fashion. With iteratees, it is possible to lazily transform how a resource will emi ...
*
Named pipe
In computing, a named pipe (also known as a FIFO for its behavior) is an extension to the traditional pipe concept on Unix and Unix-like systems, and is one of the methods of inter-process communication (IPC). The concept is also found in OS/2 and ...
– persistent pipes used for interprocess communication
*
Process substitution
In computing, process substitution is a form of inter-process communication that allows the input or output of a command to appear as a file. The command is substituted in-line, where a file name would normally occur, by the command shell. This all ...
— shell syntax for connecting multiple pipes to a process
*
GNU parallel
GNU parallel is a command-line driven utility for Linux and other Unix-like operating systems which allows the user to execute shell scripts or commands in parallel. GNU parallel is free software, written by Ole Tange in Perl. It is available un ...
*
Pipeline (computing)
In computing, a pipeline, also known as a data pipeline, is a set of data processing elements connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time- ...
– other computer-related pipelines
*
Redirection (computing)
In computing, redirection is a form of interprocess communication, and is a function common to most command-line interpreters, including the various Unix shells that can redirect standard streams to user-specified locations.
In Unix-like opera ...
*
Tee (command)
In computing, tee is a command in command-line interpreters ( shells) using standard streams which reads standard input and writes it to both standard output and one or more files, effectively duplicating its input. It is primarily used in conjun ...
– a general command for tapping data from a pipeline
*
XML pipeline
In software, an XML pipeline is formed when XML (Extensible Markup Language) processes, especially XML transformations and XML validations, are connected.
For instance, given two transformations T1 and T2, the two can be connected so that an inpu ...
– for processing of XML files
*
xargs
xargs (short for "extended arguments" ) is a command on Unix and most Unix-like operating systems used to build and execute commands from standard input. It converts input from standard input into arguments to a command.
Some commands such as gr ...
References
*
Sal Soghoian
Sal Soghoian is a user automation expert, software developer, author and musician. He joined Apple Inc. in January 1997 to serve as the Product Manager of Automation Technologies. These technologies include AppleScript, Services, the Terminal, A ...
on
MacBreak
''MacBreak'' was an internet television show hosted by Leo Laporte, Kendra Arimoto, Alex Lindsay, iJustine and Emery Wells from TWiT.tv and the Pixel Corps. The podcast was dedicated to Apple's Macintosh computers and other Apple products such ...
Episode 3 "Enter the Automatrix"
External links
History of Unix pipe notation*
Doug McIlroy’s original 1964 memo proposing the concept of a pipe for the first time
*{{man, sh, pipe, SUS, create an interprocess channel
by The Linux Information Project (LINFO)
Unix Pipes – powerful and elegant programming paradigm (Softpanorama)''Ad Hoc Data Analysis From The Unix Command Line'' at Wikibooks– Shows how to use pipelines composed of simple filters to do complex data analysis.
Use And Abuse Of Pipes With Audio Data– Gives an introduction to using and abusing pipes with netcat, nettee and fifos to play audio across a network.
stackoverflow.com– A Q&A about bash pipeline handling.
Inter-process communication
Unix
sv:Vertikalstreck#Datavetenskap