![Hilbert InfoGrowth](https://upload.wikimedia.org/wikipedia/commons/7/7c/Hilbert_InfoGrowth.png)
Though used sometimes loosely partly because of a lack of formal definition, the interpretation that seems to best describe Big data is the one associated with large body of information that we could not comprehend when used only in smaller amounts. In it primary definition though, Big data refers to
data set A data set (or dataset) is a collection of data. In the case of tabular data, a data set corresponds to one or more database tables, where every column of a table represents a particular variable, and each row corresponds to a given record of the ...
s that are too large or complex to be dealt with by traditional
data-processing application software
Application may refer to:
Mathematics and computing
* Application software, computer software designed to help the user to perform specific tasks
** Application layer, an abstraction layer that specifies protocols and interface methods used in a c ...
. Data with many fields (rows) offer greater
statistical power
In statistics, the power of a binary hypothesis test is the probability that the test correctly rejects the null hypothesis (H_0) when a specific alternative hypothesis (H_1) is true. It is commonly denoted by 1-\beta, and represents the chances ...
, while data with higher complexity (more attributes or columns) may lead to a higher
false discovery rate
In statistics, the false discovery rate (FDR) is a method of conceptualizing the rate of type I errors in null hypothesis testing when conducting multiple comparisons. FDR-controlling procedures are designed to control the FDR, which is the expec ...
. Big data analysis challenges include
capturing data,
data storage
Data storage is the recording (storing) of information (data) in a storage medium. Handwriting, phonographic recording, magnetic tape, and optical discs are all examples of storage media. Biological molecules such as RNA and DNA are conside ...
,
data analysis
Data analysis is a process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making. Data analysis has multiple facets and approaches, enco ...
, search,
sharing
Sharing is the joint use of a resource or space. It is also the process of dividing and distributing. In its narrow sense, it refers to joint or alternating use of inherently finite goods, such as a common pasture or a shared residence. Still ...
,
transfer
Transfer may refer to:
Arts and media
* ''Transfer'' (2010 film), a German science-fiction movie directed by Damir Lukacevic and starring Zana Marjanović
* ''Transfer'' (1966 film), a short film
* ''Transfer'' (journal), in management studies
...
,
visualization
Visualization or visualisation may refer to:
*Visualization (graphics), the physical or imagining creation of images, diagrams, or animations to communicate a message
* Data visualization, the graphic representation of data
* Information visualiz ...
,
querying, updating,
information privacy
Information privacy is the relationship between the collection and dissemination of data, technology, the public expectation of privacy, contextual information norms, and the legal and political issues surrounding them. It is also known as data pr ...
, and data source. Big data was originally associated with three key concepts: ''volume'', ''variety'', and ''velocity''.
The analysis of big data presents challenges in sampling, and thus previously allowing for only observations and sampling. Thus a fourth concept, ''veracity,'' refers to the quality or insightfulness of the data. Without sufficient investment in expertise for big data veracity, then the volume and variety of data can produce costs and risks that exceed an organization's capacity to create and capture ''value'' from big data''.''
Current usage of the term ''big data'' tends to refer to the use of
predictive analytics
Predictive analytics encompasses a variety of statistical techniques from data mining, predictive modeling, and machine learning that analyze current and historical facts to make predictions about future or otherwise unknown events.
In busine ...
,
user behavior analytics
User behavior analytics (UBA) is a cybersecurity process regarding the detection of insider threats, targeted attacks, and financial fraud that tracks a system's users. UBA looks at patterns of human behavior, and then analyzes observations to det ...
, or certain other advanced data analytics methods that extract
value
Value or values may refer to:
Ethics and social
* Value (ethics) wherein said concept may be construed as treating actions themselves as abstract objects, associating value to them
** Values (Western philosophy) expands the notion of value beyo ...
from big data, and seldom to a particular size of data set. "There is little doubt that the quantities of data now available are indeed large, but that's not the most relevant characteristic of this new data ecosystem."
Analysis of data sets can find new correlations to "spot business trends, prevent diseases, combat crime and so on". Scientists, business executives, medical practitioners, advertising and
governments
A government is the system or group of people governing an organized community, generally a state.
In the case of its broad associative definition, government normally consists of legislature, executive, and judiciary. Government is a ...
alike regularly meet difficulties with large data-sets in areas including
Internet searches,
fintech
Fintech, a portmanteau of "financial technology", refers to firms using new technology to compete with traditional financial methods in the delivery of financial services. Artificial intelligence, blockchain, cloud computing, and big data are r ...
, healthcare analytics, geographic information systems,
urban informatics
Urban informatics refers to the study of people creating, applying and using information and communication technology and data in the context of cities and urban environments. It sits at the conjunction of urban science, geomatics, and informatics ...
, and
business informatics. Scientists encounter limitations in
e-Science
E-Science or eScience is computationally intensive science that is carried out in highly distributed network environments, or science that uses immense data sets that require grid computing; the term sometimes includes technologies that enable dist ...
work, including
meteorology
Meteorology is a branch of the atmospheric sciences (which include atmospheric chemistry and physics) with a major focus on weather forecasting. The study of meteorology dates back millennia, though significant progress in meteorology did not ...
,
genomics
Genomics is an interdisciplinary field of biology focusing on the structure, function, evolution, mapping, and editing of genomes. A genome is an organism's complete set of DNA, including all of its genes as well as its hierarchical, three-dim ...
,
connectomics
Connectomics is the production and study of connectomes: comprehensive maps of connections within an organism's nervous system. More generally, it can be thought of as the study of neuronal wiring diagrams with a focus on how structural connectivi ...
, complex physics simulations, biology, and environmental research.
The size and number of available data sets have grown rapidly as data is collected by devices such as
mobile device
A mobile device (or handheld computer) is a computer small enough to hold and operate in the hand. Mobile devices typically have a flat LCD or OLED screen, a touchscreen interface, and digital or physical buttons. They may also have a physical ...
s, cheap and numerous information-sensing
Internet of things
The Internet of things (IoT) describes physical objects (or groups of such objects) with sensors, processing ability, software and other technologies that connect and exchange data with other devices and systems over the Internet or other comm ...
devices, aerial (
remote sensing
Remote sensing is the acquisition of information about an object or phenomenon without making physical contact with the object, in contrast to in situ or on-site observation. The term is applied especially to acquiring information about Earth ...
), software logs,
cameras
A camera is an optical instrument that can capture an image. Most cameras can capture 2D images, with some more advanced models being able to capture 3D images. At a basic level, most cameras consist of sealed boxes (the camera body), with a ...
, microphones,
radio-frequency identification
Radio-frequency identification (RFID) uses electromagnetic fields to automatically identify and track tags attached to objects. An RFID system consists of a tiny radio transponder, a radio receiver and transmitter. When triggered by an electromag ...
(RFID) readers and
wireless sensor networks
Wireless sensor networks (WSNs) refer to networks of spatially dispersed and dedicated sensors that monitor and record the physical conditions of the environment and forward the collected data to a central location. WSNs can measure environmental c ...
. The world's technological per-capita capacity to store information has roughly doubled every 40 months since the 1980s;
, every day 2.5
exabyte
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable unit ...
s (2.5×2
60 bytes) of data are generated. Based on an
IDC report prediction, the global data volume was predicted to grow exponentially from 4.4
zettabyte
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable uni ...
s to 44 zettabytes between 2013 and 2020. By 2025, IDC predicts there will be 163 zettabytes of data. According to IDC, global spending on big data and business analytics (BDA) solutions is estimated to reach $215.7 billion in 2021. While
Statista
Statista is an online platform specialized in market and consumer data, which offers statistics & reports, market insights, cosumer insights and company insights in German, English, Spanish and French. In addition to publicly available thi ...
report, the global big data market is forecasted to grow to $103 billion by 2027. In 2011
McKinsey & Company reported, if US healthcare were to use big data creatively and effectively to drive efficiency and quality, the sector could create more than $300 billion in value every year.
In the developed economies of Europe, government administrators could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data.
And users of services enabled by personal-location data could capture $600 billion in consumer surplus.
[Big data: The next frontier for innovation, competition, and productivity](_blank)
McKinsey Global Institute May 2011 One question for large enterprises is determining who should own big-data initiatives that affect the entire organization.
Relational database management system
A relational database is a (most commonly digital) database based on the relational model of data, as proposed by E. F. Codd in 1970. A system used to maintain relational databases is a relational database management system (RDBMS). Many relatio ...
s and desktop statistical software packages used to visualize data often have difficulty processing and analyzing big data. The processing and analysis of big data may require "massively parallel software running on tens, hundreds, or even thousands of servers". What qualifies as "big data" varies depending on the capabilities of those analyzing it and their tools. Furthermore, expanding capabilities make big data a moving target. "For some organizations, facing hundreds of
gigabyte
The gigabyte () is a multiple of the unit byte for digital information. The prefix ''giga'' means 109 in the International System of Units (SI). Therefore, one gigabyte is one billion bytes. The unit symbol for the gigabyte is GB.
This defini ...
s of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration."
Definition
The term ''big data'' has been in use since the 1990s, with some giving credit to
John Mashey
John R. Mashey (born 1946) is an American computer scientist, director and entrepreneur.
Career
Mashey holds a Ph.D. in computer science from Pennsylvania State University, where he developed the ASSIST assembler language teaching software. He wo ...
for popularizing the term.
Big data usually includes data sets with sizes beyond the ability of commonly used software tools to
capture,
curate
A curate () is a person who is invested with the ''care'' or ''cure'' (''cura'') ''of souls'' of a parish. In this sense, "curate" means a parish priest; but in English-speaking countries the term ''curate'' is commonly used to describe clergy w ...
, manage, and process data within a tolerable elapsed time.
Big data philosophy encompasses unstructured, semi-structured and structured data; however, the main focus is on unstructured data.
Big data "size" is a constantly moving target; ranging from a few dozen terabytes to many
zettabyte
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable uni ...
s of data.
Big data requires a set of techniques and technologies with new forms of
integration
Integration may refer to:
Biology
*Multisensory integration
*Path integration
* Pre-integration complex, viral genetic material used to insert a viral genome into a host genome
*DNA integration, by means of site-specific recombinase technology, ...
to reveal insights from
data-sets that are diverse, complex, and of a massive scale.
"Variety", "veracity", and various other "Vs" are added by some organizations to describe it, a revision challenged by some industry authorities. The Vs of big data were often referred to as the "three Vs", "four Vs", and "five Vs". They represented the qualities of big data in volume, variety, velocity, veracity, and value.
Variability is often included as an additional quality of big data.
A 2018 definition states "Big data is where parallel computing tools are needed to handle data", and notes, "This represents a distinct and clearly defined change in the computer science used, via parallel programming theories, and losses of some of the guarantees and capabilities made by
Codd's relational model."
In a comparative study of big datasets,
Kitchin and McArdle found that none of the commonly considered characteristics of big data appear consistently across all of the analyzed cases. For this reason, other studies identified the redefinition of power dynamics in knowledge discovery as the defining trait. Instead of focusing on intrinsic characteristics of big data, this alternative perspective pushes forward a relational understanding of the object claiming that what matters is the way in which data is collected, stored, made available and analyzed.
Big data vs. business intelligence
The growing maturity of the concept more starkly delineates the difference between "big data" and "
business intelligence
Business intelligence (BI) comprises the strategies and technologies used by enterprises for the data analysis and management of business information. Common functions of business intelligence technologies include reporting, online analytical pr ...
":
* Business intelligence uses applied mathematics tools and
descriptive statistics
A descriptive statistic (in the count noun sense) is a summary statistic that quantitatively describes or summarizes features from a collection of information, while descriptive statistics (in the mass noun sense) is the process of using and an ...
with data with high information density to measure things, detect trends, etc.
* Big data uses mathematical analysis, optimization,
inductive statistics, and concepts from
nonlinear system identification System identification is a method of identifying or measuring the mathematical model of a system from measurements of the system inputs and outputs. The applications of system identification include any system where the inputs and outputs can be me ...
[Billings S.A. "Nonlinear System Identification: NARMAX Methods in the Time, Frequency, and Spatio-Temporal Domains". Wiley, 2013] to infer laws (regressions, nonlinear relationships, and causal effects) from large sets of data with low information density to reveal relationships and dependencies, or to perform predictions of outcomes and behaviors.
Characteristics
![Big Data](https://upload.wikimedia.org/wikipedia/commons/e/ee/Big_Data.png)
Big data can be described by the following characteristics:
; Volume: The quantity of generated and stored data. The size of the data determines the value and potential insight, and whether it can be considered big data or not. The size of big data is usually larger than terabytes and petabytes.
; Variety: The type and nature of the data. The earlier technologies like RDBMSs were capable to handle structured data efficiently and effectively. However, the change in type and nature from structured to semi-structured or unstructured challenged the existing tools and technologies. The big data technologies evolved with the prime intention to capture, store, and process the semi-structured and unstructured (variety) data generated with high speed (velocity), and huge in size (volume). Later, these tools and technologies were explored and used for handling structured data also but preferable for storage. Eventually, the processing of structured data was still kept as optional, either using big data or traditional RDBMSs. This helps in analyzing data towards effective usage of the hidden insights exposed from the data collected via social media, log files, sensors, etc. Big data draws from text, images, audio, video; plus it completes missing pieces through
data fusion
Data fusion is the process of integrating multiple data sources to produce more consistent, accurate, and useful information than that provided by any individual data source.
Data fusion processes are often categorized as low, intermediate, or hig ...
.
; Velocity: The speed at which the data is generated and processed to meet the demands and challenges that lie in the path of growth and development. Big data is often available in real-time. Compared to
small data
''Small Data: the Tiny Clues that Uncover Huge Trends'' is Martin Lindstrom's seventh book. It chronicles his work as a branding expert, working with consumers across the world to better understand their behavior. The theory behind the book is tha ...
, big data is produced more continually. Two kinds of velocity related to big data are the frequency of generation and the frequency of handling, recording, and publishing.
;Veracity: The truthfulness or reliability of the data, which refers to the data quality and the data value. Big data must not only be large in size, but also must be reliable in order to achieve value in the analysis of it. The
data quality
Data quality refers to the state of qualitative or quantitative pieces of information. There are many definitions of data quality, but data is generally considered high quality if it is "fit for tsintended uses in operations, decision making and ...
of captured data can vary greatly, affecting an accurate analysis.
; Value: The worth in information that can be achieved by the processing and analysis of large datasets. Value also can be measured by an assessment of the other qualities of big data. Value may also represent the profitability of information that is retrieved from the analysis of big data.
; Variability: The characteristic of the changing formats, structure, or sources of big data. Big data can include structured, unstructured, or combinations of structured and unstructured data. Big data analysis may integrate raw data from multiple sources. The processing of raw data may also involve transformations of unstructured data to structured data.
Other possible characteristics of big data are:
;Exhaustive: Whether the entire system (i.e.,
=all) is captured or recorded or not. Big data may or may not include all the available data from sources.
; Fine-grained and uniquely lexical: Respectively, the proportion of specific data of each element per element collected and if the element and its characteristics are properly indexed or identified.
; Relational: If the data collected contains common fields that would enable a conjoining, or meta-analysis, of different data sets.
; Extensional: If new fields in each element of the data collected can be added or changed easily.
; Scalability: If the size of the big data storage system can expand rapidly.
Architecture
Big data repositories have existed in many forms, often built by corporations with a special need. Commercial vendors historically offered parallel database management systems for big data beginning in the 1990s. For many years, WinterCorp published the largest database report.
Teradata
Teradata Corporation is an American software company that provides cloud database and analytics-related software, products, and services. The company was formed in 1979 in Brentwood, California, as a collaboration between researchers at Caltech ...
Corporation in 1984 marketed the parallel processing
DBC 1012
The DBC/1012 Data Base Computer was a database machine introduced by Teradata Corporation in 1984, as a back-end data base management system for mainframe computers.
The DBC/1012 harnessed multiple Intel microprocessors, each with its own dedicat ...
system. Teradata systems were the first to store and analyze 1 terabyte of data in 1992. Hard disk drives were 2.5 GB in 1991 so the definition of big data continuously evolves. Teradata installed the first petabyte class RDBMS based system in 2007. , there are a few dozen petabyte class Teradata relational databases installed, the largest of which exceeds 50 PB. Systems up until 2008 were 100% structured relational data. Since then, Teradata has added unstructured data types including
XML
Extensible Markup Language (XML) is a markup language and file format for storing, transmitting, and reconstructing arbitrary data. It defines a set of rules for encoding documents in a format that is both human-readable and machine-readable ...
,
JSON
JSON (JavaScript Object Notation, pronounced ; also ) is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays (or other ser ...
, and Avro.
In 2000, Seisint Inc. (now
LexisNexis Risk Solutions
LexisNexis Risk Solutions is a global data and analytics company that provides data and technology services, analytics, predictive insights and fraud prevention for a wide range of industries. It is headquartered in Alpharetta, Georgia (part of ...
) developed a
C++
C++ (pronounced "C plus plus") is a high-level general-purpose programming language created by Danish computer scientist Bjarne Stroustrup as an extension of the C programming language, or "C with Classes". The language has expanded significan ...
-based distributed platform for data processing and querying known as the
HPCC Systems
HPCC (High-Performance Computing Cluster), also known as DAS (Data Analytics Supercomputer), is an open source, data-intensive computing system platform developed by LexisNexis Risk Solutions. The HPCC platform incorporates a software architec ...
platform. This system automatically partitions, distributes, stores and delivers structured, semi-structured, and unstructured data across multiple commodity servers. Users can write data processing pipelines and queries in a declarative dataflow programming language called ECL. Data analysts working in ECL are not required to define data schemas upfront and can rather focus on the particular problem at hand, reshaping data in the best possible manner as they develop the solution. In 2004, LexisNexis acquired Seisint Inc. and their high-speed parallel processing platform and successfully used this platform to integrate the data systems of Choicepoint Inc. when they acquired that company in 2008. In 2011, the HPCC systems platform was open-sourced under the Apache v2.0 License.
CERN
The European Organization for Nuclear Research, known as CERN (; ; ), is an intergovernmental organization that operates the largest particle physics laboratory in the world. Established in 1954, it is based in a northwestern suburb of Gene ...
and other physics experiments have collected big data sets for many decades, usually analyzed via
high-throughput computing
In computer science, high-throughput computing (HTC) is the use of many computing resources over long periods of time to accomplish a computational task.
Challenges
The HTC community is also concerned with robustness and reliability of jobs over ...
rather than the map-reduce architectures usually meant by the current "big data" movement.
In 2004,
Google
Google LLC () is an American multinational technology company focusing on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronics. ...
published a paper on a process called
MapReduce
MapReduce is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster.
A MapReduce program is composed of a ''map'' procedure, which performs filtering ...
that uses a similar architecture. The MapReduce concept provides a parallel processing model, and an associated implementation was released to process huge amounts of data. With MapReduce, queries are split and distributed across parallel nodes and processed in parallel (the "map" step). The results are then gathered and delivered (the "reduce" step). The framework was very successful, so others wanted to replicate the algorithm. Therefore, an
implementation
Implementation is the realization of an application, or execution of a plan, idea, model, design, specification, standard, algorithm, or policy.
Industry-specific definitions
Computer science
In computer science, an implementation is a real ...
of the MapReduce framework was adopted by an Apache open-source project named "
Hadoop
Apache Hadoop () is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage an ...
".
Apache Spark
Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance. Originally developed at the University of Californi ...
was developed in 2012 in response to limitations in the MapReduce paradigm, as it adds
in-memory processing
In computer science, in-memory processing is an emerging technology for processing of data stored in an in-memory database. In-memory processing is one method of addressing the performance and power bottlenecks caused by the movement of data be ...
and the ability to set up many operations (not just map followed by reducing).
MIKE2.0 is an open approach to information management that acknowledges the need for revisions due to big data implications identified in an article titled "Big Data Solution Offering". The methodology addresses handling big data in terms of useful
permutation
In mathematics, a permutation of a set is, loosely speaking, an arrangement of its members into a sequence or linear order, or if the set is already ordered, a rearrangement of its elements. The word "permutation" also refers to the act or proc ...
s of data sources,
complexity
Complexity characterises the behaviour of a system or model whose components interaction, interact in multiple ways and follow local rules, leading to nonlinearity, randomness, collective dynamics, hierarchy, and emergence.
The term is generall ...
in interrelationships, and difficulty in deleting (or modifying) individual records.
Studies in 2012 showed that a multiple-layer architecture was one option to address the issues that big data presents. A
distributed parallel architecture distributes data across multiple servers; these parallel execution environments can dramatically improve data processing speeds. This type of architecture inserts data into a parallel DBMS, which implements the use of MapReduce and Hadoop frameworks. This type of framework looks to make the processing power transparent to the end-user by using a front-end application server.
The
data lake
A data lake is a system or repository of data stored in its natural/raw format, usually object blobs or files. A data lake is usually a single store of data including raw copies of source system data, sensor data, social data etc., and transform ...
allows an organization to shift its focus from centralized control to a shared model to respond to the changing dynamics of information management. This enables quick segregation of data into the data lake, thereby reducing the overhead time.
Technologies
A 2011
McKinsey Global Institute
McKinsey & Company is a global management consulting firm founded in 1926 by University of Chicago professor James O. McKinsey, that offers professional services to corporations, governments, and other organizations. McKinsey is the oldest and ...
report characterizes the main components and ecosystem of big data as follows:
* Techniques for analyzing data, such as
A/B testing
A/B testing (also known as bucket testing, split-run testing, or split testing) is a user experience research methodology. A/B tests consist of a randomized experiment that usually involves two variants (A and B), although the concept can be al ...
,
machine learning
Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence.
Machine ...
, and
natural language processing
Natural language processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to pro ...
* Big data technologies, like
business intelligence
Business intelligence (BI) comprises the strategies and technologies used by enterprises for the data analysis and management of business information. Common functions of business intelligence technologies include reporting, online analytical pr ...
,
cloud computing
Cloud computing is the on-demand availability of computer system resources, especially data storage ( cloud storage) and computing power, without direct active management by the user. Large clouds often have functions distributed over mul ...
, and
database
In computing, a database is an organized collection of data stored and accessed electronically. Small databases can be stored on a file system, while large databases are hosted on computer clusters or cloud storage. The design of databases sp ...
s
* Visualization, such as charts, graphs, and other displays of the data
Multidimensional big data can also be represented as
OLAP
Online analytical processing, or OLAP (), is an approach to answer multi-dimensional analytical (MDA) queries swiftly in computing. OLAP is part of the broader category of business intelligence, which also encompasses relational databases, repor ...
data cubes or, mathematically,
tensor
In mathematics, a tensor is an algebraic object that describes a multilinear relationship between sets of algebraic objects related to a vector space. Tensors may map between different objects such as vectors, scalars, and even other tenso ...
s.
Array database systems have set out to provide storage and high-level query support on this data type.
Additional technologies being applied to big data include efficient tensor-based computation, such as
multilinear subspace learning
Multilinear subspace learning is an approach to dimensionality reduction.M. A. O. Vasilescu, D. Terzopoulos (2003"Multilinear Subspace Analysis of Image Ensembles" "Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVP ...
,
massively parallel-processing (
MPP
MPP or M.P.P. may refer to:
* Marginal physical product
* Master of Public Policy, an academic degree
* Member of Provincial Parliament (Ontario), Canada
* Member of Provincial Parliament (Western Cape), South Africa
* ''Merriweather Post Pavilio ...
) databases,
search-based application
Search-based applications are software applications in which a search engine platform is used as the core infrastructure for information access and reporting. Search-based applications use semantic technologies to aggregate, normalize and classi ...
s,
data mining,
distributed file system
A clustered file system is a file system which is shared by being simultaneously mounted on multiple servers. There are several approaches to clustering, most of which do not employ a clustered file system (only direct attached storage for ...
s, distributed cache (e.g.,
burst buffer
In the high-performance computing environment, burst buffer is a fast intermediate storage layer positioned between the front-end computing processes and the back-end storage systems. It bridges the performance gap between the processing speed of ...
and
Memcached
Memcached (pronounced variously ''mem-cash-dee'' or ''mem-cashed'') is a general-purpose distributed memory-caching system. It is often used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of t ...
),
distributed database
A distributed database is a database in which data is stored across different physical locations. It may be stored in multiple computers located in the same physical location (e.g. a data centre); or maybe dispersed over a network of interconnect ...
s,
cloud
In meteorology, a cloud is an aerosol consisting of a visible mass of miniature liquid droplets, frozen crystals, or other particles suspended in the atmosphere of a planetary body or similar space. Water or various other chemicals may co ...
and
HPC-based infrastructure (applications, storage and computing resources), and the Internet. Although, many approaches and technologies have been developed, it still remains difficult to carry out machine learning with big data.
Some
MPP
MPP or M.P.P. may refer to:
* Marginal physical product
* Master of Public Policy, an academic degree
* Member of Provincial Parliament (Ontario), Canada
* Member of Provincial Parliament (Western Cape), South Africa
* ''Merriweather Post Pavilio ...
relational databases have the ability to store and manage petabytes of data. Implicit is the ability to load, monitor, back up, and optimize the use of the large data tables in the
RDBMS
A relational database is a (most commonly digital) database based on the relational model of data, as proposed by E. F. Codd in 1970. A system used to maintain relational databases is a relational database management system (RDBMS). Many relation ...
.
DARPA
The Defense Advanced Research Projects Agency (DARPA) is a research and development agency of the United States Department of Defense responsible for the development of emerging technologies for use by the military.
Originally known as the Adv ...
's
Topological Data Analysis
In applied mathematics, topological based data analysis (TDA) is an approach to the analysis of datasets using techniques from topology. Extraction of information from datasets that are high-dimensional, incomplete and noisy is generally challengin ...
program seeks the fundamental structure of massive data sets and in 2008 the technology went public with the launch of a company called "Ayasdi".
The practitioners of big data analytics processes are generally hostile to slower shared storage, preferring direct-attached storage (
DAS) in its various forms from solid state drive (
SSD) to high capacity
SATA
SATA (Serial AT Attachment) is a computer bus interface that connects host bus adapters to mass storage devices such as hard disk drives, optical drives, and solid-state drives. Serial ATA succeeded the earlier Parallel ATA (PATA) standard to ...
disk buried inside parallel processing nodes. The perception of shared storage architectures—
storage area network
A storage area network (SAN) or storage network is a computer network which provides access to consolidated, block-level data storage. SANs are primarily used to access data storage devices, such as disk arrays and tape libraries from serve ...
(SAN) and
network-attached storage
Network-attached storage (NAS) is a file-level (as opposed to block-level storage) computer data storage server connected to a computer network providing data access to a heterogeneous group of clients. The term "NAS" can refer to both the tech ...
(NAS)— is that they are relatively slow, complex, and expensive. These qualities are not consistent with big data analytics systems that thrive on system performance, commodity infrastructure, and low cost.
Real or near-real-time information delivery is one of the defining characteristics of big data analytics. Latency is therefore avoided whenever and wherever possible. Data in direct-attached memory or disk is good—data on memory or disk at the other end of an
FC SAN connection is not. The cost of an
SAN at the scale needed for analytics applications is much higher than other storage techniques.
Applications
Big data has increased the demand of information management specialists so much so that
Software AG
Founded in 1969, Software AG is an enterprise software company with over 10,000 enterprise customers in over 70 countries. The company is the second largest software vendor in Germany, and the seventh largest in Europe. Software AG is traded on t ...
,
Oracle Corporation
Oracle Corporation is an American multinational computer technology corporation headquartered in Austin, Texas. In 2020, Oracle was the third-largest software company in the world by revenue and market capitalization. The company sells da ...
,
IBM,
Microsoft
Microsoft Corporation is an American multinational technology corporation producing computer software, consumer electronics, personal computers, and related services headquartered at the Microsoft Redmond campus located in Redmond, Washing ...
,
SAP
Sap is a fluid transported in xylem cells (vessel elements or tracheids) or phloem sieve tube elements of a plant. These cells transport water and nutrients throughout the plant.
Sap is distinct from latex, resin, or cell sap; it is a separ ...
,
EMC,
HP, and
Dell
Dell is an American based technology company. It develops, sells, repairs, and supports computers and related products and services. Dell is owned by its parent company, Dell Technologies.
Dell sells personal computers (PCs), servers, data ...
have spent more than $15 billion on software firms specializing in data management and analytics. In 2010, this industry was worth more than $100 billion and was growing at almost 10 percent a year, about twice as fast as the software business as a whole.
Developed economies increasingly use data-intensive technologies. There are 4.6 billion mobile-phone subscriptions worldwide, and between 1 billion and 2 billion people accessing the internet. Between 1990 and 2005, more than 1 billion people worldwide entered the middle class, which means more people became more literate, which in turn led to information growth. The world's effective capacity to exchange information through telecommunication networks was 281
petabytes
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable uni ...
in 1986, 471
petabytes
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable uni ...
in 1993, 2.2 exabytes in 2000, 65
exabytes
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable unit ...
in 2007
and predictions put the amount of internet traffic at 667 exabytes annually by 2014. According to one estimate, one-third of the globally stored information is in the form of alphanumeric text and still image data,
which is the format most useful for most big data applications. This also shows the potential of yet unused data (i.e. in the form of video and audio content).
While many vendors offer off-the-shelf products for big data, experts promote the development of in-house custom-tailored systems if the company has sufficient technical capabilities.
Government
![AlgorithmicGovernance](https://upload.wikimedia.org/wikipedia/commons/5/5a/AlgorithmicGovernance.svg)
The use and adoption of big data within governmental processes allows efficiencies in terms of cost, productivity, and innovation, but does not come without its flaws. Data analysis often requires multiple parts of government (central and local) to work in collaboration and create new and innovative processes to deliver the desired outcome. A common government organization that makes use of big data is the National Security Administration (
NSA
The National Security Agency (NSA) is a national-level intelligence agency of the United States Department of Defense, under the authority of the Director of National Intelligence (DNI). The NSA is responsible for global monitoring, collecti ...
), which monitors the activities of the Internet constantly in search for potential patterns of suspicious or illegal activities their system may pick up.
Civil registration and vital statistics
CRVS Systems stands for Civil Registration and Vital Statistics Systems and represents the interoperability of three separate systems: Civil Registration, Health Information, and Vital Statistics.
The United Nations (UN) defines Civil Registratio ...
(CRVS) collects all certificates status from birth to death. CRVS is a source of big data for governments.
International development
Research on the effective usage of information and communication technologies for development (also known as "ICT4D") suggests that big data technology can make important contributions but also present unique challenges to
international development
International development or global development is a broad concept denoting the idea that societies and countries have differing levels of economic or human development on an international scale. It is the basis for international classifications ...
. Advancements in big data analysis offer cost-effective opportunities to improve decision-making in critical development areas such as health care, employment,
economic productivity
Productivity is the efficiency of production of goods or services expressed by some measure. Measurements of productivity are often expressed as a ratio of an aggregate output to a single input or an aggregate input used in a production proce ...
, crime, security, and
natural disaster
A natural disaster is "the negative impact following an actual occurrence of natural hazard in the event that it significantly harms a community". A natural disaster can cause loss of life or damage property, and typically leaves some econ ...
and resource management.
Additionally, user-generated data offers new opportunities to give the unheard a voice. However, longstanding challenges for developing regions such as inadequate technological infrastructure and economic and human resource scarcity exacerbate existing concerns with big data such as privacy, imperfect methodology, and interoperability issues.
The challenge of "big data for development"
is currently evolving toward the application of this data through machine learning, known as "artificial intelligence for development (AI4D).
Benefits
A major practical application of big data for development has been "fighting poverty with data". In 2015, Blumenstock and colleagues estimated predicted poverty and wealth from mobile phone metadata and in 2016 Jean and colleagues combined satellite imagery and machine learning to predict poverty. Using digital trace data to study the labor market and the digital economy in Latin America, Hilbert and colleagues
[Hilbert, M., & Lu, K. (2020). The online job market trace in Latin America and the Caribbean (UN ECLAC LC/TS.2020/83; p. 79). United Nations Economic Commission for Latin America and the Caribbean. https://www.cepal.org/en/publications/45892-online-job-market-trace-latin-america-and-caribbean ] argue that digital trace data has several benefits such as:
* Thematic coverage: including areas that were previously difficult or impossible to measure
* Geographical coverage: our international sources provided sizable and comparable data for almost all countries, including many small countries that usually are not included in international inventories
* Level of detail: providing fine-grained data with many interrelated variables, and new aspects, like network connections
* Timeliness and timeseries: graphs can be produced within days of being collected
Challenges
At the same time, working with digital trace data instead of traditional survey data does not eliminate the traditional challenges involved when working in the field of international quantitative analysis. Priorities change, but the basic discussions remain the same. Among the main challenges are:
* Representativeness. While traditional development statistics is mainly concerned with the representativeness of random survey samples, digital trace data is never a random sample.
* Generalizability. While observational data always represents this source very well, it only represents what it represents, and nothing more. While it is tempting to generalize from specific observations of one platform to broader settings, this is often very deceptive.
* Harmonization. Digital trace data still requires international harmonization of indicators. It adds the challenge of so-called "data-fusion", the harmonization of different sources.
* Data overload. Analysts and institutions are not used to effectively deal with a large number of variables, which is efficiently done with interactive dashboards. Practitioners still lack a standard workflow that would allow researchers, users and policymakers to efficiently and effectively.
Finance
Big Data is being rapidly adopted in Finance to 1) speed up processing and 2) deliver better, more informed inferences, both internally and to the clients of the financial institutions.. The financial applications of Big Data range from investing decisions and trading (processing volumes of available price data, limit order books, economic data and more, all at the same time), portfolio management (optimizing over an increasingly large array of financial instruments, potentially selected from different asset classes), risk management (credit rating based on extended information), and any other aspect where the data inputs are large.
Healthcare
Big data analytics was used in healthcare by providing personalized medicine and prescriptive analytics, clinical risk intervention and predictive analytics, waste and care variability reduction, automated external and internal reporting of patient data, standardized medical terms and patient registries.
Some areas of improvement are more aspirational than actually implemented. The level of data generated within
healthcare systems
Health care or healthcare is the improvement of health via the prevention, diagnosis, treatment, amelioration or cure of disease, illness, injury, and other physical and mental impairments in people. Health care is delivered by health profess ...
is not trivial. With the added adoption of mHealth, eHealth and wearable technologies the volume of data will continue to increase. This includes
electronic health record
An electronic health record (EHR) is the systematized collection of patient and population electronically stored health information in a digital format. These records can be shared across different health care settings. Records are shared throu ...
data, imaging data, patient generated data, sensor data, and other forms of difficult to process data. There is now an even greater need for such environments to pay greater attention to data and information quality. "Big data very often means '
dirty data' and the fraction of data inaccuracies increases with data volume growth." Human inspection at the big data scale is impossible and there is a desperate need in health service for intelligent tools for accuracy and believability control and handling of information missed.
While extensive information in healthcare is now electronic, it fits under the big data umbrella as most is unstructured and difficult to use. The use of big data in healthcare has raised significant ethical challenges ranging from risks for individual rights, privacy and
autonomy
In developmental psychology and moral, political, and bioethical philosophy, autonomy, from , ''autonomos'', from αὐτο- ''auto-'' "self" and νόμος ''nomos'', "law", hence when combined understood to mean "one who gives oneself one's ...
, to transparency and trust.
Big data in health research is particularly promising in terms of exploratory biomedical research, as data-driven analysis can move forward more quickly than hypothesis-driven research. Then, trends seen in data analysis can be tested in traditional, hypothesis-driven follow up biological research and eventually clinical research.
A related application sub-area, that heavily relies on big data, within the healthcare field is that of
computer-aided diagnosis
Computer-aided detection (CADe), also called computer-aided diagnosis (CADx), are systems that assist doctors in the interpretation of medical imaging, medical images. Imaging techniques in X-ray, MRI, Endoscopy, and ultrasound diagnostics yield ...
in medicine.
For instance, for
epilepsy
Epilepsy is a group of non-communicable neurological disorders characterized by recurrent epileptic seizures. Epileptic seizures can vary from brief and nearly undetectable periods to long periods of vigorous shaking due to abnormal electrical ...
monitoring it is customary to create 5 to 10 GB of data daily.
Similarly, a single uncompressed image of breast
tomosynthesis
Tomosynthesis, also digital tomosynthesis (DTS), is a method for performing high-resolution limited-angle tomography at radiation dose levels comparable with projectional radiography. It has been studied for a variety of clinical applications, incl ...
averages 450 MB of data.
These are just a few of the many examples where
computer-aided diagnosis
Computer-aided detection (CADe), also called computer-aided diagnosis (CADx), are systems that assist doctors in the interpretation of medical imaging, medical images. Imaging techniques in X-ray, MRI, Endoscopy, and ultrasound diagnostics yield ...
uses big data. For this reason, big data has been recognized as one of the seven key challenges that computer-aided diagnosis systems need to overcome in order to reach the next level of performance.
Education
A
McKinsey Global Institute
McKinsey & Company is a global management consulting firm founded in 1926 by University of Chicago professor James O. McKinsey, that offers professional services to corporations, governments, and other organizations. McKinsey is the oldest and ...
study found a shortage of 1.5 million highly trained data professionals and managers
and a number of universities including
University of Tennessee
The University of Tennessee (officially The University of Tennessee, Knoxville; or UT Knoxville; UTK; or UT) is a public land-grant research university in Knoxville, Tennessee. Founded in 1794, two years before Tennessee became the 16th state, ...
and
UC Berkeley
The University of California, Berkeley (UC Berkeley, Berkeley, Cal, or California) is a public university, public land-grant university, land-grant research university in Berkeley, California. Established in 1868 as the University of Californi ...
, have created masters programs to meet this demand. Private boot camps have also developed programs to meet that demand, including free programs like
The Data Incubator
The Data Incubator is a data science education company. It offers corporate data science training and placement services. It is best known for an 8-week educational fellowship preparing students with Master's degrees and PhDs for careers in ...
or paid programs like
General Assembly
A general assembly or general meeting is a meeting of all the members of an organization or shareholders of a company.
Specific examples of general assembly include:
Churches
* General Assembly (presbyterian church), the highest court of presby ...
. In the specific field of marketing, one of the problems stressed by Wedel and Kannan is that marketing has several sub domains (e.g., advertising, promotions,
product development, branding) that all use different types of data.
Media
To understand how the media uses big data, it is first necessary to provide some context into the mechanism used for media process. It has been suggested by Nick Couldry and Joseph Turow that practitioners in media and advertising approach big data as many actionable points of information about millions of individuals. The industry appears to be moving away from the traditional approach of using specific media environments such as newspapers, magazines, or television shows and instead taps into consumers with technologies that reach targeted people at optimal times in optimal locations. The ultimate aim is to serve or convey, a message or content that is (statistically speaking) in line with the consumer's mindset. For example, publishing environments are increasingly tailoring messages (advertisements) and content (articles) to appeal to consumers that have been exclusively gleaned through various
data-mining activities.
* Targeting of consumers (for advertising by marketers)
* Data capture
*
Data journalism
Data journalism or data-driven journalism (DDJ) is a journalistic process based on analyzing and filtering large data sets for the purpose of creating or elevating a news story.
Data journalism is a type of journalism reflecting the increased ...
: publishers and journalists use big data tools to provide unique and innovative insights and
infographic
Infographics (a clipped compound of "information" and "graphics") are graphic visual representations of information, data, or knowledge intended to present information quickly and clearly.Doug Newsom and Jim Haynes (2004). ''Public Relations Wr ...
s.
Channel 4
Channel 4 is a British free-to-air public broadcast television network operated by the state-owned enterprise, state-owned Channel Four Television Corporation. It began its transmission on 2 November 1982 and was established to provide a four ...
, the British
public-service television broadcaster, is a leader in the field of big data and
data analysis
Data analysis is a process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making. Data analysis has multiple facets and approaches, enco ...
.
Insurance
Health insurance providers are collecting data on social "determinants of health" such as food and
TV consumption, marital status, clothing size, and purchasing habits, from which they make predictions on health costs, in order to spot health issues in their clients. It is controversial whether these predictions are currently being used for pricing.
Internet of things (IoT)
Big data and the IoT work in conjunction. Data extracted from IoT devices provides a mapping of device inter-connectivity. Such mappings have been used by the media industry, companies, and governments to more accurately target their audience and increase media efficiency. The IoT is also increasingly adopted as a means of gathering sensory data, and this sensory data has been used in medical, manufacturing and transportation
contexts.
Kevin Ashton
Kevin () is the anglicized form of the Irish masculine given name (; mga, Caoimhghín ; sga, Cóemgein ; Latinized as ). It is composed of "dear; noble"; Old Irish and ("birth"; Old Irish ).
The variant ''Kevan'' is anglicized from , an ...
, the digital innovation expert who is credited with coining the term, defines the Internet of things in this quote: "If we had computers that knew everything there was to know about things—using data they gathered without any help from us—we would be able to track and count everything, and greatly reduce waste, loss, and cost. We would know when things needed replacing, repairing, or recalling, and whether they were fresh or past their best."
Information technology
Especially since 2015, big data has come to prominence within
business operations
Business operations is the ''harvesting'' of value from assets owned by a business. Assets can be either ''physical'' or '' intangible''. An example of value derived from a physical asset, like a building, is rent. An example of value derived fro ...
as a tool to help employees work more efficiently and streamline the collection and distribution of
information technology
Information technology (IT) is the use of computers to create, process, store, retrieve, and exchange all kinds of data . and information. IT forms part of information and communications technology (ICT). An information technology system (I ...
(IT). The use of big data to resolve IT and data collection issues within an enterprise is called
IT operations analytics In the fields of information technology (IT) and systems management, IT operations analytics (ITOA) is an approach or method to retrieve, analyze, and report data for IT operations. ITOA may apply big data analytics to large datasets to produce bus ...
(ITOA).
By applying big data principles into the concepts of
machine intelligence
Artificial intelligence (AI) is intelligence—perceiving, synthesizing, and inferring information—demonstrated by machines, as opposed to intelligence displayed by animals and humans. Example tasks in which this is done include speech rec ...
and deep computing, IT departments can predict potential issues and prevent them.
ITOA businesses offer platforms for
systems management
Systems management refers to enterprise-wide administration of distributed systems including (and commonly in practice) computer systems. Systems management is strongly influenced by network management initiatives in telecommunications. The ap ...
that bring
data silos
An information silo, or a group of such silos, is an insular management system in which one information system or subsystem is incapable of reciprocal operation with others that are, or should be, related. Thus information is not adequately shared ...
together and generate insights from the whole of the system rather than from isolated pockets of data.
Case studies
Government
China
*By 2020, China plans to give all its citizens a personal "social credit" score based on how they behave. The
Social Credit System
The Social Credit System () is a national credit rating and blacklist being developed by the government of the People's Republic of China. The social credit initiative calls for the establishment of a record system so that businesses, indivi ...
, now being piloted in a number of Chinese cities, is considered a form of
mass surveillance
Mass surveillance is the intricate surveillance of an entire or a substantial fraction of a population in order to monitor that group of citizens. The surveillance is often carried out by local and federal governments or governmental organizati ...
which uses big data analysis technology.
India
* Big data analysis was tried out for the
BJP
The Bharatiya Janata Party (BJP; ; ) is a political party in India, and one of the two major Indian political parties alongside the Indian National Congress. Since 2014, it has been the ruling political party in India under Narendra Modi ...
to win the 2014 Indian General Election.
* The
Indian government
The Government of India (ISO 15919, ISO: ; often abbreviated as GoI), known as the Union Government or Central Government but often simply as the Centre, is the Government, national government of the Republic of India, a federal democracy lo ...
uses numerous techniques to ascertain how the Indian electorate is responding to government action, as well as ideas for policy augmentation.
Israel
* Personalized diabetic treatments can be created through GlucoMe's big data solution.
United Kingdom
Examples of uses of big data in public services:
* Data on prescription drugs: by connecting origin, location and the time of each prescription, a research unit was able to exemplify and examine the considerable delay between the release of any given drug, and a UK-wide adaptation of the
National Institute for Health and Care Excellence
The National Institute for Health and Care Excellence (NICE) is an executive non-departmental public body of the Department of Health and Social Care in England that publishes guidelines in four areas:
* the use of health technologies withi ...
guidelines. This suggests that new or most up-to-date drugs take some time to filter through to the general patient.
* Joining up data: a local authority
blended data about services, such as road gritting rotas, with services for people at risk, such as
Meals on Wheels
Meals on Wheels is a programme that delivers meals to individuals at home who are unable to purchase or prepare their own meals. The name is often used generically to refer to home-delivered meals programmes, not all of which are actually named ...
. The connection of data allowed the local authority to avoid any weather-related delay.
United States
* In 2012, the
Obama administration
Barack Obama's tenure as the 44th president of the United States began with his first inauguration on January 20, 2009, and ended on January 20, 2017. A Democrat from Illinois, Obama took office following a decisive victory over Republican ...
announced the Big Data Research and Development Initiative, to explore how big data could be used to address important problems faced by the government.
The initiative is composed of 84 different big data programs spread across six departments.
* Big data analysis played a large role in
Barack Obama
Barack Hussein Obama II ( ; born August 4, 1961) is an American politician who served as the 44th president of the United States from 2009 to 2017. A member of the Democratic Party, Obama was the first African-American president of the U ...
's successful
2012 re-election campaign.
* The
United States Federal Government
The federal government of the United States (U.S. federal government or U.S. government) is the Federation#Federal governments, national government of the United States, a federal republic located primarily in North America, composed of 50 ...
owns five of the ten most powerful
supercomputer
A supercomputer is a computer with a high level of performance as compared to a general-purpose computer. The performance of a supercomputer is commonly measured in floating-point operations per second ( FLOPS) instead of million instructions ...
s in the world.
* The
Utah Data Center
The Utah Data Center (UDC), also known as the Intelligence Community Comprehensive National Cybersecurity Initiative Data Center, is a data storage facility for the United States Intelligence Community that is designed to store data estimated to b ...
has been constructed by the United States
National Security Agency
The National Security Agency (NSA) is a national-level intelligence agency of the United States Department of Defense, under the authority of the Director of National Intelligence (DNI). The NSA is responsible for global monitoring, collecti ...
. When finished, the facility will be able to handle a large amount of information collected by the NSA over the Internet. The exact amount of storage space is unknown, but more recent sources claim it will be on the order of a few
exabyte
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable unit ...
s. This has posed security concerns regarding the anonymity of the data collected.
Retail
*
Walmart
Walmart Inc. (; formerly Wal-Mart Stores, Inc.) is an American multinational retail corporation that operates a chain of hypermarkets (also called supercenters), discount department stores, and grocery stores from the United States, headquarter ...
handles more than 1 million customer transactions every hour, which are imported into databases estimated to contain more than 2.5 petabytes (2560 terabytes) of data—the equivalent of 167 times the information contained in all the books in the US
Library of Congress
The Library of Congress (LOC) is the research library that officially serves the United States Congress and is the ''de facto'' national library of the United States. It is the oldest federal cultural institution in the country. The library is ...
.
*
Windermere Real Estate uses location information from nearly 100 million drivers to help new home buyers determine their typical drive times to and from work throughout various times of the day.
* FICO Card Detection System protects accounts worldwide.
Science
* The
Large Hadron Collider
The Large Hadron Collider (LHC) is the world's largest and highest-energy particle collider. It was built by the European Organization for Nuclear Research (CERN) between 1998 and 2008 in collaboration with over 10,000 scientists and hundred ...
experiments represent about 150 million sensors delivering data 40 million times per second. There are nearly 600 million collisions per second. After filtering and refraining from recording more than 99.99995% of these streams, there are 1,000 collisions of interest per second.
** As a result, only working with less than 0.001% of the sensor stream data, the data flow from all four LHC experiments represents 25 petabytes annual rate before replication (). This becomes nearly 200 petabytes after replication.
** If all sensor data were recorded in LHC, the data flow would be extremely hard to work with. The data flow would exceed 150 million petabytes annual rate, or nearly 500
exabyte
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable unit ...
s per day, before replication. To put the number in perspective, this is equivalent to 500
quintillion
Two naming scales for large numbers have been used in English and other European languages since the early modern era: the long and short scales. Most English variants use the short scale today, but the long scale remains dominant in many non-E ...
(5×10
20) bytes per day, almost 200 times more than all the other sources combined in the world.
* The
Square Kilometre Array
The Square Kilometre Array (SKA) is an intergovernmental international radio telescope project being built in Australia (low-frequency) and South Africa (mid-frequency). The combining infrastructure, the Square Kilometre Array Observatory (SKA ...
is a radio telescope built of thousands of antennas. It is expected to be operational by 2024. Collectively, these antennas are expected to gather 14 exabytes and store one petabyte per day. It is considered one of the most ambitious scientific projects ever undertaken.
* When the
Sloan Digital Sky Survey
The Sloan Digital Sky Survey or SDSS is a major multi-spectral imaging and spectroscopic redshift survey using a dedicated 2.5-m wide-angle optical telescope at Apache Point Observatory in New Mexico, United States. The project began in 2000 a ...
(SDSS) began to collect astronomical data in 2000, it amassed more in its first few weeks than all data collected in the history of astronomy previously. Continuing at a rate of about 200 GB per night, SDSS has amassed more than 140 terabytes of information.
When the
Large Synoptic Survey Telescope
The Vera C. Rubin Observatory, previously referred to as the Large Synoptic Survey Telescope (LSST), is an astronomical observatory currently under construction in Chile. Its main task will be carrying out a synoptic astronomical survey, the Le ...
, successor to SDSS, comes online in 2020, its designers expect it to acquire that amount of data every five days.
*
Decoding the human genome originally took 10 years to process; now it can be achieved in less than a day. The DNA sequencers have divided the sequencing cost by 10,000 in the last ten years, which is 100 times cheaper than the reduction in cost predicted by
Moore's law
Moore's law is the observation that the number of transistors in a dense integrated circuit (IC) doubles about every two years. Moore's law is an observation and projection of a historical trend. Rather than a law of physics, it is an empir ...
.
* The
NASA
The National Aeronautics and Space Administration (NASA ) is an independent agency of the US federal government responsible for the civil space program, aeronautics research, and space research.
NASA was established in 1958, succeeding t ...
Center for Climate Simulation (NCCS) stores 32 petabytes of climate observations and simulations on the Discover supercomputing cluster.
* Google's DNAStack compiles and organizes DNA samples of genetic data from around the world to identify diseases and other medical defects. These fast and exact calculations eliminate any "friction points", or human errors that could be made by one of the numerous science and biology experts working with the DNA. DNAStack, a part of Google Genomics, allows scientists to use the vast sample of resources from Google's search server to scale social experiments that would usually take years, instantly.
*
23andme
23andMe Holding Co. is a publicly held personal genomics and biotechnology company based in South San Francisco, California. It is best known for providing a direct-to-consumer genetic testing service in which customers provide a saliva sample t ...
's
DNA database
A DNA database or DNA databank is a database of DNA profiles which can be used in the analysis of genetic diseases, genetic fingerprinting for criminology, or genetic genealogy. DNA databases may be public or private, the largest ones being nat ...
contains the genetic information of over 1,000,000 people worldwide. The company explores selling the "anonymous aggregated genetic data" to other researchers and pharmaceutical companies for research purposes if patients give their consent.
Ahmad Hariri, professor of psychology and neuroscience at
Duke University
Duke University is a private research university in Durham, North Carolina. Founded by Methodists and Quakers in the present-day city of Trinity in 1838, the school moved to Durham in 1892. In 1924, tobacco and electric power industrialist James ...
who has been using 23andMe in his research since 2009 states that the most important aspect of the company's new service is that it makes genetic research accessible and relatively cheap for scientists.
[ A study that identified 15 genome sites linked to depression in 23andMe's database lead to a surge in demands to access the repository with 23andMe fielding nearly 20 requests to access the depression data in the two weeks after publication of the paper.
*Computational fluid dynamics ( CFD) and hydrodynamic ]turbulence
In fluid dynamics, turbulence or turbulent flow is fluid motion characterized by chaotic changes in pressure and flow velocity. It is in contrast to a laminar flow, which occurs when a fluid flows in parallel layers, with no disruption between ...
research generate massive data sets. The Johns Hopkins Turbulence Databases
JHTDB
contains over 350 terabytes of spatiotemporal fields from Direct Numerical simulations of various turbulent flows. Such data have been difficult to share using traditional methods such as downloading flat simulation output files. The data within JHTDB can be accessed using "virtual sensors" with various access modes ranging from direct web-browser queries, access through Matlab, Python, Fortran and C programs executing on clients' platforms, to cut out services to download raw data. The data have been used in over 150 scientific publications.
Sports
Big data can be used to improve training and understanding competitors, using sport sensors. It is also possible to predict winners in a match using big data analytics.
Future performance of players could be predicted as well. Thus, players' value and salary is determined by data collected throughout the season.
In Formula One races, race cars with hundreds of sensors generate terabytes of data. These sensors collect data points from tire pressure to fuel burn efficiency.
Based on the data, engineers and data analysts decide whether adjustments should be made in order to win a race. Besides, using big data, race teams try to predict the time they will finish the race beforehand, based on simulations using data collected over the season.
Technology
* eBay.com
eBay Inc. ( ) is an American multinational corporation, multinational e-commerce company based in San Jose, California, that facilitates consumer to consumer, consumer-to-consumer and business to consumer, business-to-consumer sales through ...
uses two data warehouse
In computing, a data warehouse (DW or DWH), also known as an enterprise data warehouse (EDW), is a system used for Business reporting, reporting and data analysis and is considered a core component of business intelligence. DWs are central Repos ...
s at 7.5 petabytes
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable uni ...
and 40PB as well as a 40PB Hadoop
Apache Hadoop () is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage an ...
cluster for search, consumer recommendations, and merchandising.
* Amazon.com
Amazon.com, Inc. ( ) is an American multinational technology company focusing on e-commerce, cloud computing, online advertising, digital streaming, and artificial intelligence. It has been referred to as "one of the most influential economi ...
handles millions of back-end operations every day, as well as queries from more than half a million third-party sellers. The core technology that keeps Amazon running is Linux-based and they had the world's three largest Linux databases, with capacities of 7.8 TB, 18.5 TB, and 24.7 TB.
* Facebook
Facebook is an online social media and social networking service owned by American company Meta Platforms. Founded in 2004 by Mark Zuckerberg with fellow Harvard College students and roommates Eduardo Saverin, Andrew McCollum, Dustin M ...
handles 50 billion photos from its user base. , Facebook reached 2 billion monthly active users
Active users is a measurement metric that is commonly used to measure the level of engagement for a particular product or object, by quantifying the number of active interactions from visitors within a relevant range of time (daily, weekly and m ...
.
* Google
Google LLC () is an American multinational technology company focusing on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronics. ...
was handling roughly 100 billion searches per month .
COVID-19
During the COVID-19 pandemic
The COVID-19 pandemic, also known as the coronavirus pandemic, is an ongoing global pandemic of coronavirus disease 2019 (COVID-19) caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The novel virus was first identif ...
, big data was raised as a way to minimise the impact of the disease. Significant applications of big data included minimising the spread of the virus, case identification and development of medical treatment.
Governments used big data to track infected people to minimise spread. Early adopters included China, Taiwan, South Korea, and Israel.
Research activities
Encrypted search and cluster formation in big data were demonstrated in March 2014 at the American Society of Engineering Education. Gautam Siwach engaged at ''Tackling the challenges of Big Data'' by MIT Computer Science and Artificial Intelligence Laboratory
Computer Science and Artificial Intelligence Laboratory (CSAIL) is a research institute at the Massachusetts Institute of Technology (MIT) formed by the 2003 merger of the Laboratory for Computer Science (LCS) and the Artificial Intelligence Lab ...
and Amir Esmailpour at the UNH Research Group investigated the key features of big data as the formation of clusters and their interconnections. They focused on the security of big data and the orientation of the term towards the presence of different types of data in an encrypted form at cloud interface by providing the raw definitions and real-time examples within the technology. Moreover, they proposed an approach for identifying the encoding technique to advance towards an expedited search over encrypted text leading to the security enhancements in big data.
In March 2012, The White House announced a national "Big Data Initiative" that consisted of six federal departments and agencies committing more than $200 million to big data research projects.
The initiative included a National Science Foundation "Expeditions in Computing" grant of $10 million over five years to the AMPLab at the University of California, Berkeley. The AMPLab also received funds from DARPA
The Defense Advanced Research Projects Agency (DARPA) is a research and development agency of the United States Department of Defense responsible for the development of emerging technologies for use by the military.
Originally known as the Adv ...
, and over a dozen industrial sponsors and uses big data to attack a wide range of problems from predicting traffic congestion to fighting cancer.
The White House Big Data Initiative also included a commitment by the Department of Energy to provide $25 million in funding over five years to establish the Scalable Data Management, Analysis and Visualization (SDAV) Institute, led by the Energy Department's Lawrence Berkeley National Laboratory
Lawrence Berkeley National Laboratory (LBNL), commonly referred to as the Berkeley Lab, is a United States Department of Energy National Labs, United States national laboratory that is owned by, and conducts scientific research on behalf of, t ...
. The SDAV Institute aims to bring together the expertise of six national laboratories and seven universities to develop new tools to help scientists manage and visualize data on the department's supercomputers.
The U.S. state of announced the Massachusetts Big Data Initiative in May 2012, which provides funding from the state government and private companies to a variety of research institutions. The Massachusetts Institute of Technology
The Massachusetts Institute of Technology (MIT) is a private land-grant research university in Cambridge, Massachusetts. Established in 1861, MIT has played a key role in the development of modern technology and science, and is one of the ...
hosts the Intel Science and Technology Center for Big Data in the MIT Computer Science and Artificial Intelligence Laboratory
Computer Science and Artificial Intelligence Laboratory (CSAIL) is a research institute at the Massachusetts Institute of Technology (MIT) formed by the 2003 merger of the Laboratory for Computer Science (LCS) and the Artificial Intelligence Lab ...
, combining government, corporate, and institutional funding and research efforts.
The European Commission is funding the two-year-long Big Data Public Private Forum through their Seventh Framework Program to engage companies, academics and other stakeholders in discussing big data issues. The project aims to define a strategy in terms of research and innovation to guide supporting actions from the European Commission in the successful implementation of the big data economy. Outcomes of this project will be used as input for Horizon 2020
The Framework Programmes for Research and Technological Development, also called Framework Programmes or abbreviated FP1 to FP9, are funding programmes created by the European Union/European Commission to support and foster research in the Europea ...
, their next framework program.
The British government announced in March 2014 the founding of the Alan Turing Institute
The Alan Turing Institute is the United Kingdom's national institute for data science and artificial intelligence, founded in 2015 and largely funded by the UK government. It is named after Alan Turing, the British mathematician and computing ...
, named after the computer pioneer and code-breaker, which will focus on new ways to collect and analyze large data sets.
At the University of Waterloo Stratford Campus
The Stratford School of Interaction Design and Business, also known as the University of Waterloo Stratford School and formerly the University of Waterloo Stratford Campus, is a satellite campus of the University of Waterloo located in Stratfo ...
Canadian Open Data Experience (CODE) Inspiration Day, participants demonstrated how using data visualization can increase the understanding and appeal of big data sets and communicate their story to the world.
Computational social science
Computational social science is the academic sub-discipline concerned with computational approaches to the social sciences. This means that computers are used to model, simulate, and analyze social phenomena. Fields include computational economics ...
s – Anyone can use application programming interfaces (APIs) provided by big data holders, such as Google and Twitter, to do research in the social and behavioral sciences. Often these APIs are provided for free. Tobias Preis et al. used Google Trends
Google Trends is a website by Google that analyzes the popularity of top web search query, search queries in Google Search across various regions and languages. The website uses graphs to compare the search volume of different queries over time. ...
data to demonstrate that Internet users from countries with a higher per capita gross domestic products (GDPs) are more likely to search for information about the future than information about the past. The findings suggest there may be a link between online behaviors and real-world economic indicators. The authors of the study examined Google queries logs made by ratio of the volume of searches for the coming year (2011) to the volume of searches for the previous year (2009), which they call the " future orientation index". They compared the future orientation index to the per capita GDP of each country, and found a strong tendency for countries where Google users inquire more about the future to have a higher GDP.
Tobias Preis and his colleagues Helen Susannah Moat and H. Eugene Stanley
Harry Eugene Stanley (born March 28, 1941) is an American physicist and University Professor at Boston University. He has made seminal contributions to statistical physics and is one of the pioneers of interdisciplinary science. His current r ...
introduced a method to identify online precursors for stock market moves, using trading strategies based on search volume data provided by Google Trends. Their analysis of Google
Google LLC () is an American multinational technology company focusing on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronics. ...
search volume for 98 terms of varying financial relevance, published in '' Scientific Reports'', suggests that increases in search volume for financially relevant search terms tend to precede large losses in financial markets.
Big data sets come with algorithmic challenges that previously did not exist. Hence, there is seen by some to be a need to fundamentally change the processing ways.
The Workshops on Algorithms for Modern Massive Data Sets (MMDS) bring together computer scientists, statisticians, mathematicians, and data analysis practitioners to discuss algorithmic challenges of big data. Regarding big data, such concepts of magnitude are relative. As it is stated "If the past is of any guidance, then today's big data most likely will not be considered as such in the near future."[
]
Sampling big data
A research question that is asked about big data sets is whether it is necessary to look at the full data to draw certain conclusions about the properties of the data or if is a sample is good enough. The name big data itself contains a term related to size and this is an important characteristic of big data. But sampling enables the selection of right data points from within the larger data set to estimate the characteristics of the whole population. In manufacturing different types of sensory data such as acoustics, vibration, pressure, current, voltage, and controller data are available at short time intervals. To predict downtime it may not be necessary to look at all the data but a sample may be sufficient. Big data can be broken down by various data point categories such as demographic, psychographic, behavioral, and transactional data. With large sets of data points, marketers are able to create and use more customized segments of consumers for more strategic targeting.
There has been some work done in sampling algorithms for big data. A theoretical formulation for sampling Twitter data has been developed.
Critique
Critiques of the big data paradigm come in two flavors: those that question the implications of the approach itself, and those that question the way it is currently done. One approach to this criticism is the field of critical data studies.
Critiques of the big data paradigm
"A crucial problem is that we do not know much about the underlying empirical micro-processes that lead to the emergence of the etypical network characteristics of Big Data." In their critique, Snijders, Matzat, and Reips point out that often very strong assumptions are made about mathematical properties that may not at all reflect what is really going on at the level of micro-processes. Mark Graham has leveled broad critiques at Chris Anderson Chris Anderson may refer to:
Sports
* Chris Anderson (baseball) (born 1992), American baseball player
* Chris Anderson (cheese roller), 22-time winner of annual cheese rolling
* Chris Anderson (footballer, born 1925) (1925–1986), Scottish footb ...
's assertion that big data will spell the end of theory: focusing in particular on the notion that big data must always be contextualized in their social, economic, and political contexts. Even as companies invest eight- and nine-figure sums to derive insight from information streaming in from suppliers and customers, less than 40% of employees have sufficiently mature processes and skills to do so. To overcome this insight deficit, big data, no matter how comprehensive or well analyzed, must be complemented by "big judgment", according to an article in the ''Harvard Business Review
''Harvard Business Review'' (''HBR'') is a general management magazine published by Harvard Business Publishing, a wholly owned subsidiary of Harvard University. ''HBR'' is published six times a year and is headquartered in Brighton, Massach ...
''.
Much in the same line, it has been pointed out that the decisions based on the analysis of big data are inevitably "informed by the world as it was in the past, or, at best, as it currently is".[Hilbert, M. (2016). Big Data for Development: A Review of Promises and Challenges. Development Policy Review, 34(1), 135–174. https://doi.org/10.1111/dpr.12142 free access: https://www.martinhilbert.net/big-data-for-development/ ] Fed by a large number of data on past experiences, algorithms can predict future development if the future is similar to the past.[Big Data requires Big Visions for Big Change.](_blank)
, Hilbert, M. (2014). London: TEDx UCL, x=independently organized TED talks If the system's dynamics of the future change (if it is not a stationary process
In mathematics and statistics, a stationary process (or a strict/strictly stationary process or strong/strongly stationary process) is a stochastic process whose unconditional joint probability distribution does not change when shifted in time. Con ...
), the past can say little about the future. In order to make predictions in changing environments, it would be necessary to have a thorough understanding of the systems dynamic, which requires theory. As a response to this critique Alemany Oliver and Vayre suggest to use "abductive reasoning as a first step in the research process in order to bring context to consumers' digital traces and make new theories emerge".
Additionally, it has been suggested to combine big data approaches with computer simulations, such as agent-based models and complex systems
A complex system is a system composed of many components which may interact with each other. Examples of complex systems are Earth's global climate, organisms, the human brain, infrastructure such as power grid, transportation or communication s ...
. Agent-based models are increasingly getting better in predicting the outcome of social complexities of even unknown future scenarios through computer simulations that are based on a collection of mutually interdependent algorithms. Finally, the use of multivariate methods that probe for the latent structure of the data, such as factor analysis
Factor analysis is a statistical method used to describe variability among observed, correlated variables in terms of a potentially lower number of unobserved variables called factors. For example, it is possible that variations in six observed ...
and cluster analysis
Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense) to each other than to those in other groups (clusters). It is a main task of ...
, have proven useful as analytic approaches that go well beyond the bi-variate approaches (e.g. contingency tables) typically employed with smaller data sets.
In health and biology, conventional scientific approaches are based on experimentation. For these approaches, the limiting factor is the relevant data that can confirm or refute the initial hypothesis.
A new postulate is accepted now in biosciences: the information provided by the data in huge volumes (omics
The branches of science known informally as omics are various disciplines in biology whose names end in the suffix '' -omics'', such as genomics, proteomics, metabolomics, metagenomics, phenomics and transcriptomics. Omics aims at the collect ...
) without prior hypothesis is complementary and sometimes necessary to conventional approaches based on experimentation. In the massive approaches it is the formulation of a relevant hypothesis to explain the data that is the limiting factor. The search logic is reversed and the limits of induction ("Glory of Science and Philosophy scandal", C. D. Broad
Charlie Dunbar Broad (30 December 1887 – 11 March 1971), usually cited as C. D. Broad, was an English people, English epistemology, epistemologist, history of philosophy, historian of philosophy, philosophy of science, philosopher of sc ...
, 1926) are to be considered.
Privacy
Privacy (, ) is the ability of an individual or group to seclude themselves or information about themselves, and thereby express themselves selectively.
The domain of privacy partially overlaps with security, which can include the concepts of a ...
advocates are concerned about the threat to privacy represented by increasing storage and integration of personally identifiable information; expert panels have released various policy recommendations to conform practice to expectations of privacy. The misuse of big data in several cases by media, companies, and even the government has allowed for abolition of trust in almost every fundamental institution holding up society.
Nayef Al-Rodhan argues that a new kind of social contract will be needed to protect individual liberties in the context of big data and giant corporations that own vast amounts of information, and that the use of big data should be monitored and better regulated at the national and international levels. Barocas and Nissenbaum argue that one way of protecting individual users is by being informed about the types of information being collected, with whom it is shared, under what constraints and for what purposes.
Critiques of the "V" model
The "V" model of big data is concerning as it centers around computational scalability and lacks in a loss around the perceptibility and understandability of information. This led to the framework of cognitive big data
Cognition refers to "the mental action or process of acquiring knowledge and understanding through thought, experience, and the senses". It encompasses all aspects of intellectual functions and processes such as: perception, attention, thought, ...
, which characterizes big data applications according to:
* Data completeness: understanding of the non-obvious from data
* Data correlation, causation, and predictability: causality as not essential requirement to achieve predictability
* Explainability and interpretability: humans desire to understand and accept what they understand, where algorithms do not cope with this
* Level of automated decision-making
Automated decision-making (ADM) involves the use of data, machines and algorithms to make decisions in a range of contexts, including public administration, business, health, education, law, employment, transport, media and entertainment, with var ...
: algorithms that support automated decision making and algorithmic self-learning
Critiques of novelty
Large data sets have been analyzed by computing machines for well over a century, including the US census analytics performed by IBM's punch-card machines which computed statistics including means and variances of populations across the whole continent. In more recent decades, science experiments such as CERN
The European Organization for Nuclear Research, known as CERN (; ; ), is an intergovernmental organization that operates the largest particle physics laboratory in the world. Established in 1954, it is based in a northwestern suburb of Gene ...
have produced data on similar scales to current commercial "big data". However, science experiments have tended to analyze their data using specialized custom-built high-performance computing
High-performance computing (HPC) uses supercomputers and computer clusters to solve advanced computation problems.
Overview
HPC integrates systems administration (including network and security knowledge) and parallel programming into a mult ...
(super-computing) clusters and grids, rather than clouds of cheap commodity computers as in the current commercial wave, implying a difference in both culture and technology stack.
Critiques of big data execution
Ulf-Dietrich Reips Prof. Dr. Ulf-Dietrich Reips is a full professor in the Faculty of Sciences at the University of Konstanz, where he holds the Chair for Psychological Methods, Assessment, and iScience. Between 2009 and 2013 he was a full-time tenured IKERBASQUE res ...
and Uwe Matzat wrote in 2014 that big data had become a "fad" in scientific research. Researcher danah boyd
danah boyd (stylized in lowercase, born November 24, 1977 as Danah Michele Mattas) She noted her mother added lowercase 'h' in birth name "danah" for typographical balance, reflecting the lowercase first letter 'd' and later changed her last na ...
has raised concerns about the use of big data in science neglecting principles such as choosing a representative sample
In statistics, quality assurance, and survey methodology, sampling is the selection of a subset (a statistical sample) of individuals from within a statistical population to estimate characteristics of the whole population. Statisticians attempt ...
by being too concerned about handling the huge amounts of data. This approach may lead to results that have a bias
Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individual, a group, ...
in one way or another. Integration across heterogeneous data resources—some that might be considered big data and others not—presents formidable logistical as well as analytical challenges, but many researchers argue that such integrations are likely to represent the most promising new frontiers in science.
In the provocative article "Critical Questions for Big Data", the authors title big data a part of mythology
Myth is a folklore genre consisting of narratives that play a fundamental role in a society, such as foundational tales or origin myths. Since "myth" is widely used to imply that a story is not objectively true, the identification of a narrat ...
: "large data sets offer a higher form of intelligence and knowledge .. with the aura of truth, objectivity, and accuracy". Users of big data are often "lost in the sheer volume of numbers", and "working with Big Data is still subjective, and what it quantifies does not necessarily have a closer claim on objective truth". Recent developments in BI domain, such as pro-active reporting especially target improvements in the usability of big data, through automated filtering
Filter, filtering or filters may refer to:
Science and technology
Computing
* Filter (higher-order function), in functional programming
* Filter (software), a computer program to process a data stream
* Filter (video), a software component tha ...
of non-useful data and correlations.[Failure to Launch: From Big Data to Big Decisions](_blank)
, Forte Wares. Big structures are full of spurious correlations either because of non-causal coincidences (law of truly large numbers
The law of truly large numbers (a statistical adage), attributed to Persi Diaconis and Frederick Mosteller, states that with a large enough number of independent samples, any highly implausible (i.e. unlikely in any single sample, but with const ...
), solely nature of big randomness (Ramsey theory
Ramsey theory, named after the British mathematician and philosopher Frank P. Ramsey, is a branch of mathematics that focuses on the appearance of order in a substructure given a structure of a known size. Problems in Ramsey theory typically ask a ...
), or existence of non-included factors so the hope, of early experimenters to make large databases of numbers "speak for themselves" and revolutionize scientific method, is questioned. Catherine Tucker
Catherine Tucker (born May 16, 1977) is the Sloan Distinguished Professor of Management at MIT Sloan, where she is also chair of the PhD program. She is known for her research into the consequences of digital data for electronic privacy, algorit ...
has pointed to "hype" around big data, writing "By itself, big data is unlikely to be valuable." The article explains: "The many contexts where data is cheap relative to the cost of retaining talent to process it, suggests that processing skills are more important than data itself in creating value for a firm."
Big data analysis is often shallow compared to analysis of smaller data sets. In many big data projects, there is no large data analysis happening, but the challenge is the extract, transform, load
In computing, extract, transform, load (ETL) is a three-phase process where data is extracted, transformed (cleaned, sanitized, scrubbed) and loaded into an output data container. The data can be collated from one or more sources and it can also ...
part of data pre-processing.
Big data is a buzzword
A buzzword is a word or phrase, new or already existing, that becomes popular for a period of time. Buzzwords often derive from technical terms yet often have much of the original technical meaning removed through fashionable use, being simply used ...
and a "vague term", but at the same time an "obsession" with entrepreneurs, consultants, scientists, and the media. Big data showcases such as Google Flu Trends
Google Flu Trends (GFT) was a web service operated by Google. It provided estimates of influenza activity for more than 25 countries. By aggregating Google Search queries, it attempted to make accurate predictions about flu activity. This project ...
failed to deliver good predictions in recent years, overstating the flu outbreaks by a factor of two. Similarly, Academy awards
The Academy Awards, better known as the Oscars, are awards for artistic and technical merit for the American and international film industry. The awards are regarded by many as the most prestigious, significant awards in the entertainment ind ...
and election predictions solely based on Twitter were more often off than on target.
Big data often poses the same challenges as small data; adding more data does not solve problems of bias, but may emphasize other problems. In particular data sources such as Twitter are not representative of the overall population, and results drawn from such sources may then lead to wrong conclusions. Google Translate
Google Translate is a multilingual neural machine translation service developed by Google to translate text, documents and websites from one language into another. It offers a website interface, a mobile app for Android and iOS, and an API ...
—which is based on big data statistical analysis of text—does a good job at translating web pages. However, results from specialized domains may be dramatically skewed.
On the other hand, big data may also introduce new problems, such as the multiple comparisons problem
In statistics, the multiple comparisons, multiplicity or multiple testing problem occurs when one considers a set of statistical inferences simultaneously or infers a subset of parameters selected based on the observed values.
The more inferences ...
: simultaneously testing a large set of hypotheses is likely to produce many false results that mistakenly appear significant.
Ioannidis argued that "most published research findings are false" due to essentially the same effect: when many scientific teams and researchers each perform many experiments (i.e. process a big amount of scientific data; although not with big data technology), the likelihood of a "significant" result being false grows fast – even more so, when only positive results are published.
Furthermore, big data analytics results are only as good as the model on which they are predicated. In an example, big data took part in attempting to predict the results of the 2016 U.S. Presidential Election with varying degrees of success.
Critiques of big data policing and surveillance
Big data has been used in policing and surveillance by institutions like law enforcement
Law enforcement is the activity of some members of government who act in an organized manner to enforce the law by discovering, deterring, rehabilitating, or punishing people who violate the rules and norms governing that society. The term en ...
and corporations
A corporation is an organization—usually a group of people or a company—authorized by the state to act as a single entity (a legal entity recognized by private and public law "born out of statute"; a legal person in legal context) and r ...
. Due to the less visible nature of data-based surveillance as compared to traditional methods of policing, objections to big data policing are less likely to arise. According to Sarah Brayne's ''Big Data Surveillance: The Case of Policing'', big data policing can reproduce existing societal inequalities in three ways:
* Placing people under increased surveillance by using the justification of a mathematical and therefore unbiased algorithm
* Increasing the scope and number of people that are subject to law enforcement tracking and exacerbating existing racial overrepresentation in the criminal justice system
* Encouraging members of society to abandon interactions with institutions that would create a digital trace, thus creating obstacles to social inclusion
If these potential problems are not corrected or regulated, the effects of big data policing may continue to shape societal hierarchies. Conscientious usage of big data policing could prevent individual level biases from becoming institutional biases, Brayne also notes.
In popular culture
Books
*'' Moneyball'' is a non-fiction book that explores how the Oakland Athletics used statistical analysis to outperform teams with larger budgets. In 2011 a film adaptation
A film adaptation is the transfer of a work or story, in whole or in part, to a feature film. Although often considered a type of derivative work, film adaptation has been conceptualized recently by academic scholars such as Robert Stam as a dial ...
starring Brad Pitt
William Bradley Pitt (born December 18, 1963) is an American actor and film producer. He is the recipient of various accolades, including two Academy Awards, a British Academy Film Award, two Golden Globe Awards, and a Primetime Emmy Award. ...
was released.
Film
*In '' Captain America: The Winter Soldier'', H.Y.D.R.A (disguised as S.H.I.E.L.D
S.H.I.E.L.D. is a fictional espionage, special law enforcement, and counter-terrorism agency appearing in American comic books published by Marvel Comics. Created by Stan Lee and Jack Kirby in ''Strange Tales'' #135 (August 1965), it often dea ...
) develops helicarriers that use data to determine and eliminate threats over the globe.
*In ''The Dark Knight
''The Dark Knight'' is a 2008 superhero film directed by Christopher Nolan from a screenplay he co-wrote with his brother Jonathan. Based on the DC Comics superhero, Batman, it is the sequel to ''Batman Begins'' (2005) and the second insta ...
'', Batman
Batman is a superhero appearing in American comic books published by DC Comics. The character was created by artist Bob Kane and writer Bill Finger, and debuted in Detective Comics 27, the 27th issue of the comic book ''Detective Comics'' on ...
uses a sonar device that can spy on all of Gotham City
Gotham City ( ), or simply Gotham, is a fictional city appearing in American comic books published by DC Comics, best known as the home of the superhero Batman and his allies and foes. Created by writer Bill Finger and artist Bob Kane, the cit ...
. The data is gathered from the mobile phones of people within the city.
See also
References
Further reading
*
*
*
*
*
*
*
External links
*
*
{{Authority control
Data management
Distributed computing problems
Transaction processing
Technology forecasting
Data analysis
Databases