Investigative Data Warehouse (IDW) is a searchable

database In computing, a database is an organized collection of data stored and accessed electronically. Small databases can be stored on a file system, while large databases are hosted on computer clusters or cloud storage. The design of databases sp ...

operated by the

FBI The Federal Bureau of Investigation (FBI) is the domestic Intelligence agency, intelligence and Security agency, security service of the United States and its principal Federal law enforcement in the United States, federal law enforcement age ...

. It was created in 2004. Much of the nature and scope of the database is classified. The database is a centralization of multiple federal and state databases, including

criminal record A criminal record, police record, or colloquially RAP sheet (Record of Arrests and Prosecutions) is a record of a person's criminal history. The information included in a criminal record and the existence of a criminal record varies between coun ...

s from various

law enforcement agencies A law enforcement agency (LEA) is any government agency responsible for the enforcement of the laws. Jurisdiction LEAs which have their ability to apply their powers restricted in some way are said to operate within a jurisdiction. LEAs ...

, the

U.S. Department of the Treasury The Department of the Treasury (USDT) is the national treasury and finance department of the federal government of the United States, where it serves as an executive department. The department oversees the Bureau of Engraving and Printing and t ...

Financial Crimes Enforcement Network The Financial Crimes Enforcement Network (FinCEN) is a bureau of the United States Department of the Treasury that collects and analyzes information about financial transactions in order to combat domestic and international money laundering, terr ...

(FinCEN), and

public records Public records are documents or pieces of information that are not considered confidential and generally pertain to the conduct of government. For example, in California, when a couple fills out a marriage license application, they have the optio ...

databases. According to Michael Morehart's testimony before the

House Committee on Financial Services The United States House Committee on Financial Services, also referred to as the House Banking Committee and previously known as the Committee on Banking and Currency, is the committee of the United States House of Representatives that oversees t ...

in 2006, the "IDW is a centralized, web-enabled, closed system repository for intelligence and investigative data. This system, maintained by the FBI, allows appropriately trained and authorized personnel throughout the country to query for information of relevance to investigative and intelligence matters."Morehart 2005, op. cit.

Overview

The size of the database appears to be growing rapidly. In 2004, according to a government solicitation for bids to manage the project, it was approximately 10 TB in size. In 2005, according to one FBI official, the IDW contained approximately 100 million documents. In 2006 it contained more than 560 million documents and was accessible by more than 12,000 individuals. According to the FBI's website, as of August 22, 2007, the database contained 700 million records from 53 databases and was accessible by 13,000 individuals around the world. , the FBI is the subject of a lawsuit brought by the EFF (

Electronic Frontier Foundation The Electronic Frontier Foundation (EFF) is an international non-profit digital rights group based in San Francisco, California. The foundation was formed on 10 July 1990 by John Gilmore, John Perry Barlow and Mitch Kapor to promote Internet ci ...

) because of a lack of public notice describing the database and the criteria for including personal information, as required by the

Privacy Act of 1974 The Privacy Act of 1974 (, ), a United States federal law, establishes a Code of Fair Information Practice that governs the collection, maintenance, use, and dissemination of personally identifiable information about individuals that is maintaine ...

. The lawsuits are a result of two

Freedom of Information Act Freedom of Information Act may refer to the following legislations in different jurisdictions which mandate the national government to disclose certain data to the general public upon request: * Freedom of Information Act 1982, the Australian act * ...

requests filed by the EFF in 2006. It was built in part by Chiliad corporation, the FBI Office of the Chief Technology Officer, and others. Companies listed on the FOIA files include Northrop Grumman and others.

Purpose

Investigative Data Warehouse–Secret (IDW-S) "provides data and data processing/analysis services to FBI agents and analysts as they perform counter-terrorism, counter-intelligence, and law enforcement missions". The core subsystem supports the Counter-Terrorism Division (CTD), the Special Event Unit, and via DOCLAB-S, the Joint Intelligence Committee Investigation (JICI) and IntelPlus. According to a 2005 email, "IDW will also be used for criminal and other authorized non-CT investigations as it evolves." (CT being counter terrorism)

Subsystems

Within the system, there were subsystems named IDW-S Core, SPT, and DOCLAB-S The special projects team (SPT):

allows for the rapid import of new specialized data sources. These data sources are not made available to the general IDW users but instead are provided to a small group of users who have a demonstrated "need-to-know". The SPT System is similar in function to the IDW-S system, with the main difference is a different set of data sources. The SPT System allows its users to access not only the standard IDW Data Store but the specialized SPT Data Store.

Privacy

According to internal emails, the FBI performed several

Privacy Impact Assessment A Privacy Impact Assessment (PIA) is a process which assists organizations in identifying and managing the privacy risks arising from new projects, initiatives, systems, processes, strategies, policies, business relationships etc. It benefits variou ...

s (PIAs) of the IDW system. They worked with lawyers from their National Security Law Branch (NSLB) to attempt to make sure their system was complying with various laws regarding sharing of information and secrecy (for example, rule 6e of the Federal Rules of Criminal Procedure, regarding the secrecy of Grand Jury material ). The Information Sharing Policy Group (ISPG) formed a Discretionary Access Control Team (DACT), to work on "approval of data sets" and "access control requirements" for IDW and DataMart, and responding to other Intelligence Community agencies requesting access. The EFF FOIA IDW website states "Despite the vast amount of personal information contained in the IDW, the FBI has never published a Privacy Act notice describing the system or explaining the ways in which the records might be used." There was also a 2005 email from someone on the Office of General Council (OGC) about "preliminary staff musings that maybe we should limit FBI PIA requirements to non-NS systems" (NS being National Security). There was also an email from 2006 saying that 'national security systems are exempt from E-Gov', apparently referring to the

E-Government Act of 2002 The E-Government Act of 2002 (, , , H.R. 2458/S. 803), is a United States statute enacted on 17 December 2002, with an effective date for most provisions of 17 April 2003. Its stated purpose is to improve the management and promotion of electronic ...

, which has a section that deals with privacy.

Data sources

The IDW used many data sources. The FOIA documents from EFF are heavily redacted, but some of the sources are as follows: *FBI Automated Case Support system (ACS), subset of the Electronic Case File (ECF) systemFBI, IDW Privileged Users Guide, 2004 Dec 1 *Joint Intelligence Committee Investigation documents (JICI),FBI, IDW-S System Security Plan, 2003 Dec 3 with OCR text FBI IDW Status Update, 2005 Sep 21 *"Open Source News" (public websites, such as the Washington Post and others) *Secure Automated Messaging Network (SAMNet) * Violent Gang and Terrorist Organizing File (VGTOF) *

DARPA TIDES program Translingual Information Detection, Extraction and Summarization (TIDES) is a technology development program funded by the U.S. Defense Advanced Research Projects Agency (DARPA), focused on the automated processing and understanding of language dat ...

('open source news' that has been organized and collected) *IntelPlus Filerooms, with OCR text *FBI National Crime Information Center (NCIC) *FBI Records Management Division (RMD), Document Laboratory (DocLab), FBIHQ *

MiTAP MiTAP, or Mitre Text and Audio Processing, is a computer system that tries to automatically gather, translate, organize, and present information "for monitoring infectious disease outbreaks and other global events."Damianos et al, AI Magazine, Vol 2 ...

(collects data from public sources, websites, etc.) *SPT-Specific data sources (partial list, FOIA files have large parts redacted): **Unified Name Index (UNI) extracts **Financial Center (FinCen), including

Bank Secrecy Act The Bank Secrecy Act of 1970 (BSA), also known as the Currency and Foreign Transactions Reporting Act, is a U.S. law requiring financial institutions in the United States to assist U.S. government agencies in detecting and preventing money laund ...

data **"Various Sources", including the

Transportation Security Administration The Transportation Security Administration (TSA) is an agency of the United States Department of Homeland Security (DHS) that has authority over the security of transportation systems within, and connecting to the United States. It was created ...

**FBI Counterterrorism Division (CTD) **Telephone numbers / addresses from ACS **Case data from ACS **Terrorist Watch List (TWL) **"Other NJTTF data" **DoS ... Lost/Stolen Passport data **

No Fly List The No Fly List maintained by the United States federal government's Terrorist Screening Center (TSC) is one of several lists included in algorithmic rulesets used by government agencies and airlines to decide who to allow to board airline flight ...

, from TSA ** Selectee list, from TSA **ACS/ECF with some case types excluded **

CIA The Central Intelligence Agency (CIA ), known informally as the Agency and historically as the Company, is a civilian intelligence agency, foreign intelligence service of the federal government of the United States, officially tasked with gat ...

non-TS/non-SCI Technical Discussions (TDs) and Intelligence Information Reports (IIRs) from 1978 to the May 2004 There was also talk of linking the FTTTF "Data Mart" with IDW. The data in IDW is classified at the 'Secret' level or lower. Higher classifications are not allowed, and can be removedEFF FOIA files, 2008 Apr 2, idw01
, page 43

References

; Sources consulted * * * (Contains various emails from inside the FBI regarding the IDW) * (Contains various emails from inside the FBI regarding the IDW) * (Contains various emails from inside the FBI regarding the IDW) * * * * * * * * ; Endnotes {{Reflist, 35em Federal Bureau of Investigation Information science Government databases in the United States