|
Cuneiform (programming Language)
Cuneiform is an open-source workflow language for large-scale scientific data analysis. It is a statically typed functional programming language promoting parallel computing. It features a versatile foreign function interface allowing users to integrate software from many external programming languages. At the organizational level Cuneiform provides facilities like conditional branching and general recursion making it Turing-complete. In this, Cuneiform is the attempt to close the gap between scientific workflow systems like Taverna, KNIME, or Galaxy and large-scale data analysis programming models like MapReduce or Pig Latin while offering the generality of a functional programming language. Cuneiform is implemented in distributed Erlang. If run in distributed mode it drives a POSIX-compliant distributed file system like Gluster or Ceph (or a FUSE integration of some other file system, e.g., HDFS). Alternatively, Cuneiform scripts can be executed on top of HTCondor or Hadoop ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Functional Programming
In computer science, functional programming is a programming paradigm where programs are constructed by Function application, applying and Function composition (computer science), composing Function (computer science), functions. It is a declarative programming paradigm in which function definitions are Tree (data structure), trees of Expression (computer science), expressions that map Value (computer science), values to other values, rather than a sequence of Imperative programming, imperative Statement (computer science), statements which update the State (computer science), running state of the program. In functional programming, functions are treated as first-class citizens, meaning that they can be bound to names (including local Identifier (computer languages), identifiers), passed as Parameter (computer programming), arguments, and Return value, returned from other functions, just as any other data type can. This allows programs to be written in a Declarative programming, ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Apache Taverna
Apache Taverna was an open source software tool for designing and executing workflows, initially created by the myGrid project under the name ''Taverna Workbench'', then a project under the Apache incubator. Taverna allowed users to integrate many different software components, including WSDL SOAP or REST Web services, such as those provided by the National Center for Biotechnology Information, the European Bioinformatics Institute, the DNA Databank of Japan (DDBJ), SoapLab, BioMOBY and EMBOSS. The set of available services was not finite and users could import new service descriptions into the Taverna Workbench. Taverna Workbench provided a desktop authoring environment and enactment engine for scientific workflows. The Taverna workflow engine was also available separately, as a Java API, command line tool or as a server. Taverna was used by users in many domains, such as bioinformatics, cheminformatics, medicine, astronomy, social science, music, and digital preservation. Some ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Python (programming Language)
Python is a high-level, general-purpose programming language. Its design philosophy emphasizes code readability with the use of significant indentation. Python is dynamically-typed and garbage-collected. It supports multiple programming paradigms, including structured (particularly procedural), object-oriented and functional programming. It is often described as a "batteries included" language due to its comprehensive standard library. Guido van Rossum began working on Python in the late 1980s as a successor to the ABC programming language and first released it in 1991 as Python 0.9.0. Python 2.0 was released in 2000 and introduced new features such as list comprehensions, cycle-detecting garbage collection, reference counting, and Unicode support. Python 3.0, released in 2008, was a major revision that is not completely backward-compatible with earlier versions. Python 2 was discontinued with version 2.7.18 in 2020. Python consistently ranks as ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
R (programming Language)
R is a programming language for statistical computing and graphics supported by the R Core Team and the R Foundation for Statistical Computing. Created by statisticians Ross Ihaka and Robert Gentleman, R is used among data miners, bioinformaticians and statisticians for data analysis and developing statistical software. Users have created packages to augment the functions of the R language. According to user surveys and studies of scholarly literature databases, R is one of the most commonly used programming languages used in data mining. R ranks 12th in the TIOBE index, a measure of programming language popularity, in which the language peaked in 8th place in August 2020. The official R software environment is an open-source free software environment within the GNU package, available under the GNU General Public License. It is written primarily in C, Fortran, and R itself (partially self-hosting). Precompiled executables are provided for various operating systems. R ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Dataflow Programming
In computer programming, dataflow programming is a programming paradigm that models a program as a directed graph of the data flowing between operations, thus implementing dataflow principles and architecture. Dataflow programming languages share some features of functional languages, and were generally developed in order to bring some functional concepts to a language more suitable for numeric processing. Some authors use the term ''datastream'' instead of '' dataflow'' to avoid confusion with dataflow computing or dataflow architecture, based on an indeterministic machine paradigm. Dataflow programming was pioneered by Jack Dennis and his graduate students at MIT in the 1960s. Considerations Traditionally, a program is modelled as a series of operations happening in a specific order; this may be referred to as sequential, procedural, control flow (indicating that the program chooses a specific path), or imperative programming. The program focuses on commands, in line with the v ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
HTCondor
HTCondor is an open-source high-throughput computing software framework for coarse-grained distributed parallelization of computationally intensive tasks. It can be used to manage workload on a dedicated cluster of computers, or to farm out work to idle desktop computersso-called cycle scavenging. HTCondor runs on Linux, Unix, Mac OS X, FreeBSD, and Microsoft Windows operating systems. HTCondor can integrate both dedicated resources (rack-mounted clusters) and non-dedicated desktop machines (cycle scavenging) into one computing environment. HTCondor is developed by the HTCondor team at the University of Wisconsin–Madison and is freely available for use. HTCondor follows an open-source philosophy and is licensed under the Apache License 2.0. While HTCondor makes use of unused computing time, leaving computers turned on for use with HTCondor will increase energy consumption and associated costs. Starting from version 7.1.1, HTCondor can hibernate and wake machines based ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Apache Hadoop
Apache Hadoop () is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage and processing of big data using the MapReduce programming model. Hadoop was originally designed for computer clusters built from commodity hardware, which is still the common use. It has since also found use on clusters of higher-end hardware. All the modules in Hadoop are designed with a fundamental assumption that hardware failures are common occurrences and should be automatically handled by the framework. The core of Apache Hadoop consists of a storage part, known as Hadoop Distributed File System (HDFS), and a processing part which is a MapReduce programming model. Hadoop splits files into large blocks and distributes them across nodes in a cluster. It then transfers packaged code into nodes to process the data in parallel. This a ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Filesystem In Userspace
Filesystem in Userspace (FUSE) is a software interface for Unix and Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code. This is achieved by running file system code in user space while the FUSE module provides only a bridge to the actual kernel interfaces. FUSE is available for Linux, FreeBSD, OpenBSD, NetBSD (as puffs), OpenSolaris, Minix 3, macOS, and Windows. FUSE is free software originally released under the terms of the GNU General Public License and the GNU Lesser General Public License. History The FUSE system was originally part of ''AVFS'' (''A Virtual Filesystem''), a filesystem implementation heavily influenced by the translator concept of the GNU Hurd. It superseded Linux Userland Filesystem, and provided a translational interface using in libfuse1. FUSE was originally released under the terms of the GNU General Public License and the GNU Lesser General Public License, later also reimple ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Ceph (software)
Ceph (pronounced ) is an open-source software-defined storage platform that implements object storage on a single distributed computer cluster and provides 3-in-1 interfaces for object-, block- and file-level storage. Ceph aims primarily for completely distributed operation without a single point of failure, scalability to the exabyte level, and to be freely available. Since version 12, Ceph does not rely on other filesystems and can directly manage HDDs and SSDs with its own storage backend BlueStore and can completely self reliantly expose a POSIX filesystem. Ceph replicates data and makes it fault-tolerant, using commodity hardware and Ethernet IP and requiring no specific hardware support. The Ceph’s system offers disaster recovery and data redundancy through techniques such as replication, erasure coding, snapshots and storage cloning. As a result of its design, the system is both self-healing and self-managing, aiming to minimize administration time and other co ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Gluster
Gluster Inc. (formerly known as Z RESEARCH) was a software company that provided an open source platform for scale-out public and private cloud storage. The company was privately funded and headquartered in Sunnyvale, California, with an engineering center in Bangalore, India. Gluster was funded by Nexus Venture Partners and Index Ventures. Gluster was acquired by Red Hat on October 7, 2011. History The name ''Gluster'' comes from the combination of the terms ''GNU'' and ''cluster''. Despite the similarity in names, Gluster is not related to the Lustre file system and does not incorporate any Lustre code. Gluster based its product on ''GlusterFS'', an open-source software-based network-attached filesystem that deploys on commodity hardware. The initial version of GlusterFS was written by Anand Babu Periasamy, Gluster's founder and CTO. In May 2010 Ben Golub became the president and chief executive officer. Red Hat became the primary author and maintainer of the GlusterFS ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
POSIX
The Portable Operating System Interface (POSIX) is a family of standards specified by the IEEE Computer Society for maintaining compatibility between operating systems. POSIX defines both the system- and user-level application programming interfaces (APIs), along with command line shells and utility interfaces, for software compatibility (portability) with variants of Unix and other operating systems. POSIX is also a trademark of the IEEE. POSIX is intended to be used by both application and system developers. Name Originally, the name "POSIX" referred to IEEE Std 1003.1-1988, released in 1988. The family of POSIX standards is formally designated as IEEE 1003 and the ISO/IEC standard number is ISO/IEC 9945. The standards emerged from a project that began in 1984 building on work from related activity in the ''/usr/group'' association. Richard Stallman suggested the name ''POSIX'' (pronounced as ''pahz-icks,'' as in ''positive'', not as ''poh-six'') to the IEEE instead of former ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Pig (programming Tool)
Apache Pig is a high-level platform for creating programs that run on Apache Hadoop. The language for this platform is called Pig Latin. Pig can execute its Hadoop jobs in MapReduce, Apache Tez, or Apache Spark. Pig Latin abstracts the programming from the Java MapReduce idiom into a notation which makes MapReduce programming high level, similar to that of SQL for relational database management systems. Pig Latin can be extended using user-defined functions (UDFs) which the user can write in Java, Python, JavaScript, Ruby or Groovy and then call directly from the language. History Apache Pig was originally developed at Yahoo Research around 2006 for researchers to have an ad hoc way of creating and executing MapReduce jobs on very large data sets. In 2007, it was moved into the Apache Software Foundation. Naming Regarding the naming of the Pig programming language, the name was chosen arbitrarily and stuck because it was memorable, easy to spell, and for novelty. Exampl ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
