data processing
Data processing is the collection and manipulation of digital data to produce meaningful information. Data processing is a form of ''information processing'', which is the modification (processing) of information in any manner detectable by an o ...
where
data
Data ( , ) are a collection of discrete or continuous values that convey information, describing the quantity, quality, fact, statistics, other basic units of meaning, or simply sequences of symbols that may be further interpreted for ...
is moved between two systems so that it ends up in a
staging area
A staging area (otherwise staging base, staging facility, staging ground, staging point, or staging post) is a location in which organisms, people, vehicles, equipment, or material are assembled before use. It may refer to:
* In aviation, a desi ...
on the target system.
With the traditional extract, transform and load (ETL) method, the load job is the last step, and the data that is loaded has already been transformed. With the alternative method extract, load and transform (ELT), the loading job is the middle step, and the transformed data is loaded in its original format for data transformation in the target system.
Traditionally, loading jobs on large systems have taken a long time, and have typically been run at night outside a company's opening hours.
Purpose
Two main goals of data loading are to obtain fresher data in the systems after loading, and that the loading is fast so that the data can be updated frequently. For full data refresh, faster loading can be achieved by turning off referential integrity, secondary indexes and
logging
Logging is the process of cutting, processing, and moving trees to a location for transport. It may include skidder, skidding, on-site processing, and loading of trees or trunk (botany), logs onto logging truck, trucksmeasures are calculated for the updated table.
Full refresh is easy to implement, but involves moving of much data which can take a long time, and can make it challenging to keep historical data.
Incremental update
Incremental update or incremental refresh means that only new or updated data is retrieved from the source system. The updated data is then added to the existing data in the target system, and the existing data in the target system is updated. The indices and statistics are updated accordingly. Incremental update can make loading faster and make it easier to keep track of history, but can be demanding to set up and maintain.
Tricle feed
Tricle feed or trickle loading means that when the source system is updated, the changes in the target system will occur almost immediately.
Loading to systems that are in use
When loading data into a system that is currently in use by users or other systems, one must decide when the system should be updated and what will happen to tables that are in use at the same time as the system is to be updated. One possible solution is to make use of shadow tables.
See also
*
Database
In computing, a database is an organized collection of data or a type of data store based on the use of a database management system (DBMS), the software that interacts with end users, applications, and the database itself to capture and a ...
*
Data warehouse
In computing, a data warehouse (DW or DWH), also known as an enterprise data warehouse (EDW), is a system used for Business intelligence, reporting and data analysis and is a core component of business intelligence. Data warehouses are central Re ...