Gearman is an open-source application framework designed to distribute appropriate computer tasks to multiple computers, so large tasks can be done more quickly. In some cases, load balancing rather than raw speed may be the main goal; a Web server, for instance, could use Gearman to send tasks for which it is not optimized to another computer (which may be running on a different

architecture Architecture is the art and technique of designing and building, as distinguished from the skills associated with construction. It is both the process and the product of sketching, conceiving, planning, designing, and constructing building ...

, using another

operating system An operating system (OS) is system software that manages computer hardware, software resources, and provides common services for computer programs. Time-sharing operating systems schedule tasks for efficient use of the system and may also in ...

, or loaded with a computer language better suited to a particular operation). It was originally written in

Perl Perl is a family of two high-level, general-purpose, interpreted, dynamic programming languages. "Perl" refers to Perl 5, but from 2000 to 2019 it also referred to its redesigned "sister language", Perl 6, before the latter's name was offici ...

Brad Fitzpatrick Bradley Joseph Fitzpatrick (born February 5, 1980) is an American programmer. He is best known as the creator of LiveJournal and is the author of a variety of free software projects such as memcached, PubSubHubbub, OpenID, and Perkeep. Early life ...

Brian Aker Brian Aker, born August 4, 1972 in Lexington, Kentucky, US, is an open-source hacker who has worked on various Apache modules, the Slash system, and numerous storage engines for the MySQL database. Aker was Director of Architecture at MySQL A ...

and Eric Day rewrote the framework in C.

How Gearman Works

Gearman assigns each involved computer a role as

client Client(s) or The Client may refer to: * Client (business) * Client (computing), hardware or software that accesses a remote service on another computer * Customer or client, a recipient of goods or services in return for monetary or other valuable ...

, job

server Server may refer to: Computing *Server (computing), a computer program or a device that provides functionality for other programs or devices, called clients Role * Waiting staff, those who work at a restaurant or a bar attending customers and su ...

, or worker. A worker machine can be assigned multiple instances of the worker role, which allows more powerful computers to complete more portions of a given task. Tasks originate on a client, are transmitted from the client to the job server, and performed on one or more workers. The completed task's output is then returned, again by way of the job server, to the client where the task originated. Gearman is conceptually related to

MapReduce MapReduce is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster. A MapReduce program is composed of a ''map'' procedure, which performs filtering ...

; Gearman handles MapReduce by allowing worker nodes to map out work to other workers, with the original worker acting as the reducer. Gearman performs coalescence on the work sent by a client. If two or more clients ask for work to be completed on the same body of work, either by seeing that the same blocks are being sent or by using the unique value sent by the client, it will coalesce the work so that only one worker is used. It does this specifically to avoid

thundering herd problem In computer science, the thundering herd problem occurs when a large number of processes or threads waiting for an event are awoken when that event occurs, but only one process is able to handle the event. When the processes wake up, they will each ...

s which are common to cache hit failures. To mitigate the damage that would be done if a job server (or its network connection) were to fail, clients can be configured with more than one assigned job server; if the first assigned job server fails, another can be transparently substituted. Gearman implements a

protocol Protocol may refer to: Sociology and politics * Protocol (politics), a formal agreement between nation states * Protocol (diplomacy), the etiquette of diplomacy and affairs of state * Etiquette, a code of personal behavior Science and technolog ...

that consists of binary packets containing requests and responses; this protocol defines the structure of messages passing between the three parts of a Gearman implementation. By default, the Gearman protocol uses TCP port 4730. It previously operated on port 7003, but this conflicted with the AFS port range and the new port (4730) was assigned by

IANA The Internet Assigned Numbers Authority (IANA) is a standards organization that oversees global IP address allocation, autonomous system number allocation, root zone management in the Domain Name System (DNS), media types, and other Interne ...

. The name "Gearman" was chosen as an

anagram An anagram is a word or phrase formed by rearranging the letters of a different word or phrase, typically using all the original letters exactly once. For example, the word ''anagram'' itself can be rearranged into ''nag a ram'', also the word ...

for "Manager", "since it dispatches jobs to be done, but does not do anything useful itself."

Features

* Job retries * Round robin scheduling * Coalescence * Persistence storage via: ** libmemcached ** libdrizzle **

SQLite SQLite (, ) is a database engine written in the C programming language. It is not a standalone app; rather, it is a library that software developers embed in their apps. As such, it belongs to the family of embedded databases. It is the most ...

MySQL MySQL () is an open-source relational database management system (RDBMS). Its name is a combination of "My", the name of co-founder Michael Widenius's daughter My, and "SQL", the acronym for Structured Query Language. A relational database o ...

Postgres PostgreSQL (, ), also known as Postgres, is a free and open-source relational database management system (RDBMS) emphasizing extensibility and SQL compliance. It was originally named POSTGRES, referring to its origins as a successor to the In ...

** tokyocabinet **

Redis Redis (; Remote Dictionary Server) is an in-memory data structure store, used as a distributed, in-memory key–value database, cache and message broker, with optional durability. Redis supports different kinds of abstract data structures, su ...

(unreleased - currently in development) **

MongoDB MongoDB is a source-available cross-platform document-oriented database program. Classified as a NoSQL database program, MongoDB uses JSON-like documents with optional schemas. MongoDB is developed by MongoDB Inc. and licensed under the Serve ...

(unreleased - currently in development)

Implementations

Gearmand, up to version 1.1.12

Gearmand, from version 1.1.13

java-gearman-service

Gearman::Server

TclGearman

Clients

Currently there are client libraries for C, Perl, Node.js, Python, PHP, Ruby, Java, .NET, JMS, MySQL,

PostgreSQL PostgreSQL (, ), also known as Postgres, is a free and open-source relational database management system (RDBMS) emphasizing extensibility and SQL compliance. It was originally named POSTGRES, referring to its origins as a successor to the In ...

, and

Drizzle Drizzle is a light precipitation consisting of liquid water drops smaller than those of rain – generally smaller than in diameter. Drizzle is normally produced by low stratiform clouds and stratocumulus clouds. Precipitation rates from dri ...

.Gearman page Client & Worker APIs
/ref>

Similar software

{{Disputed section, date=August 2021 *

HAProxy HAProxy is a free and open source software that provides a high availability load balancer and reverse proxy for TCP and HTTP-based applications that spreads requests across multiple servers. It is written in C and has a reputation for being fa ...

Squid True squid are molluscs with an elongated soft body, large eyes, eight arms, and two tentacles in the superorder Decapodiformes, though many other molluscs within the broader Neocoleoidea are also called squid despite not strictly fitting t ...

Varnish Varnish is a clear transparent hard protective coating or film. It is not a stain. It usually has a yellowish shade from the manufacturing process and materials used, but it may also be pigmented as desired, and is sold commercially in various ...

* Træfɪk

External links

* http://danga.com/words/2007_06_usenix/usenix.pdf * http://gearman.org/documentation/ * http://gearman.org/download/ * http://lists.danga.com/pipermail/gearman/2008-April/000076.html * https://web.archive.org/web/20081209012837/http://krow.livejournal.com/628025.html
GearmanBundle for Symfony2 projects

Citations

Application programming interfaces Message-oriented middleware Software using the BSD license