BatchPipes
   HOME

TheInfoList



OR:

On
IBM mainframe IBM mainframes are large computer systems produced by IBM since 1952. During the 1960s and 1970s, IBM dominated the computer market with the 7000 series and the later System/360, followed by the System/370. Current mainframe computers in IBM' ...
s, BatchPipes is a batch job processing utility which runs under the
MVS/ESA Multiple Virtual Storage, more commonly called MVS, is the most commonly used operating system on the System/370, System/390 and IBM Z IBM mainframe computers. IBM developed MVS, along with OS/VS1 and SVS, as a successor to OS/360. It is unr ...
operating system and later versions—
OS/390 OS/390 is an IBM operating system for the System/390 IBM mainframe computers. Overview OS/390 was introduced in late 1995 in an effort to simplify the packaging and ordering for the key, entitled elements needed to complete a fully function ...
and
z/OS z/OS is a 64-bit operating system for IBM z/Architecture mainframes, introduced by IBM in October 2000. It derives from and is the successor to OS/390, which in turn was preceded by a string of MVS versions.Starting with the earliest: ...
.


Core function

In traditional processing, if data records are written out to sequential ( QSAM and BSAM)
data set A data set (or dataset) is a collection of data. In the case of tabular data, a data set corresponds to one or more table (database), database tables, where every column (database), column of a table represents a particular Variable (computer sci ...
on disk or tape, they cannot be read concurrently by another job. The "writer" and "reader" cannot run at the same time. This is termed ''file-level interlock'' or ''data-set-level interlock''. With BatchPipes an installation can arrange for the data to be "piped" between the two jobs. The advantage is that the jobs can run concurrently and it is possible, and very usual, to avoid the time to write the data to
secondary storage Computer data storage or digital data storage is a technology consisting of computer components and Data storage, recording media that are used to retain digital data. It is a core function and fundamental component of computers. The cent ...
and to read it back. The combination of these two characteristics, if used judiciously, leads to a reduction in the combined elapsed time of the two jobs, as measured from the start of the writer job to the end of the reader job. BatchPipes maintains a ''short'' queue of records being passed between the writer and the reader. The writer adds records to the back of the queue and the reader takes them from the front. This is deemed ''record-level interlock'' and allows the reader and the writer to run concurrently. A
sort Sort may refer to: * Sorting, any process of arranging items in sequence or in sets ** Sorting algorithm, any algorithm for ordering a list of elements ** Mainframe sort merge, sort utility for IBM mainframe systems ** Sort (Unix), which sorts the ...
is a special case: ''all'' the input records must be read before the first output record can be written. Hence there can be no overlap between the input and output phases of a sort. But the input phase ''can'' be overlapped with the previous job's output phase. Similarly, the output phase of sort can be overlapped with a downstream job that reads the sorted data.


Advanced pipe topologies

More complex topologies than "one reader one writer" are possible. * "Two readers one writer" is a good example of an attempt to balance reader's speed against a writer's speed. Because the queue is short a faster writer will often be forced to wait for a slower reader to take records off the queue before the writer can continue processing. Using two readers helps to utilize writers capabilities. * "One job as a reader from one pipe and a writer to another" is often seen where this job edits the records. While traditional batch streams often contain such jobs, this kind of processing can be introduced using, for example
IBM International Business Machines Corporation (using the trademark IBM), nicknamed Big Blue, is an American Multinational corporation, multinational technology company headquartered in Armonk, New York, and present in over 175 countries. It is ...
's DFSORT product or BatchPipeWorks (part of BatchPipes).


Criticism

One of the key implementation considerations is scheduling the reader and writer jobs to run together. In practical batch schedules this might not be feasible. Furthermore, if ''any'' job in the pipeline fails, recovery actions will be wider than just recovering this single job. For these reasons some installations have found it difficult to implement BatchPipes.


BatchPipePlex

BatchPipes can use the IBM mainframe
Coupling Facility In IBM System/390 and IBM Z mainframe computers, a Coupling Facility or CF is a piece of computer hardware or virtual machine that coordinates multiple processors. A Parallel Sysplex relies on one or more Coupling Facilities (CFs). A coupling ...
to pipe data between different members of a Parallel Sysplex, using the BatchPipePlex facility.


BatchPipeWorks

BatchPipes includes a set of pipeline stages based on
IBM International Business Machines Corporation (using the trademark IBM), nicknamed Big Blue, is an American Multinational corporation, multinational technology company headquartered in Armonk, New York, and present in over 175 countries. It is ...
's
CMS Pipelines CMS Pipelines is a feature of the VM/CMS operating system that allows the user to create and use a pipeline. The programs in a pipeline operate on a sequential stream of records. A program writes records that are read by the next program in the pip ...
product developed for the VM/ESA operating system. These stages provide additional processing, without the need for additional batch jobs in the pipeline.


History

BatchPipes Version 1 was developed in the late 1980s and early 1990s simply as a technique to speed up
MVS/ESA Multiple Virtual Storage, more commonly called MVS, is the most commonly used operating system on the System/370, System/390 and IBM Z IBM mainframe computers. IBM developed MVS, along with OS/VS1 and SVS, as a successor to OS/360. It is unr ...
batch processing. In 1997 the functionality of BatchPipes was integrated into a larger IBM product - SmartBatch (which incorporated two BMC Corporation product features: DataAccelerator and BatchAccelerator). However SmartBatch was discontinued in April 2000. APT International, based in
Monaco Monaco, officially the Principality of Monaco, is a Sovereign state, sovereign city-state and European microstates, microstate on the French Riviera a few kilometres west of the Regions of Italy, Italian region of Liguria, in Western Europe, ...
, produced a competitive product trademarked as WARP. A few months after the launch of this product, IBM renamed their
OS/2 OS/2 is a Proprietary software, proprietary computer operating system for x86 and PowerPC based personal computers. It was created and initially developed jointly by IBM and Microsoft, under the leadership of IBM software designer Ed Iacobucci, ...
product
OS/2 Warp 4 OS/2 is a Proprietary software, proprietary computer operating system for x86 and PowerPC based personal computers. It was created and initially developed jointly by IBM and Microsoft, under the leadership of IBM software designer Ed Iacobucci, ...
, conflicting with the marketing of the performance product that was the only competitor to BatchPipes. This resulted in 7 years of litigation at
Tribunal de grande instance de Paris The Tribunal de Paris (, ), located at the Judicial Campus of Paris in Batignolles, is the largest court in France by caseload. It replaced the capital's former () and () under an amalgamation of jurisdictions that came into effect on 1 Januar ...
1994-2000 [16.5.1997,77 Rev. Droite de propriété Industrielle(1997),46 - Sté PANEK v. Sté IBM Corp. http://cyber.law.harvard.edu/ilaw/Jurisdiction/Geller_Full.html Subsequently, BatchPipes Version 2 was released, incorporating BatchPipes Version 1 and some additional features from SmartBatch: BatchPipePlex and BatchPipeWorks. BatchPipes Version 2 is still a marketed IBM product.


See also

* Pipeline (Unix), an original concept *
CMS Pipelines CMS Pipelines is a feature of the VM/CMS operating system that allows the user to create and use a pipeline. The programs in a pipeline operate on a sequential stream of records. A program writes records that are read by the next program in the pip ...
*
ESA/390 IBM Enterprise Systems Architecture is an instruction set architecture introduced by IBM as Enterprise Systems Architecture/370 (ESA/370) in 1988. It is based on the IBM System/370-XA architecture. It extended the dual-address-space mechanis ...
, a mainframe architecture


References

{{DEFAULTSORT:Batchpipes Job scheduling IBM mainframe software