The Homeland Security Department is ditching a controversial data-mining program that was capable of analyzing one billion pieces per hour of “structured” information, such as databases, and one million pieces per hour of “unstructured” information, such as intelligence reports, e-mail, news articles and blogs. Oh, those pesky privacy laws.

The six-year anniversary of the September 11, 2001, terrorist attacks seems as apt a time as any to focus a bit on the agency that tragic day spawned and one of its recently suspended, little-known but much-criticized security measures.

News recently surfaced that the United States Department of Homeland Security scrapped an ambitious anti-terrorism data-mining tool after investigators found that information about real people was used during testing without required privacy safeguards.

The Christian Science Monitor reports:

From late 2004 until mid-2006, a little-known data-mining computer system developed by the US Department of Homeland Security to hunt terrorists, weapons of mass destruction, and biological weapons sifted through Americans’ personal data with little regard for federal privacy laws.

“Data mining to help fight the war on terror has become an accepted, even mandated, method to provide timely security information,” writes CS Monitor‘s Mark Clayton. But this one was “special.”

The department has spent $42 million since 2003 developing the software tool known as Analysis, Dissemination, Visualization, Insight and Semantic Enhancement (ADVISE), at the Lawrence Livermore and Pacific Northwest national laboratories. It was intended for wide use by DHS components — including immigration, customs, border protection, biological defense and its intelligence office — and was one of the broadest of 12 data-mining projects in the agency.

According to a report on the program completed in June by the DHS Office of Inspector General (OIG):

ADVISE is a collection of software, hardware and operational standards that can be adapted and tailored to meet the specific needs of the user organization. ADVISE provides the ability to search, integrate, and gain rapid insights from large quantities of information across disparate databases, a process that would otherwise be overwhelming to intelligence analysts.

In 2004, a DHS research official said the program would be able to ingest 1 billion pieces per hour of structured information, such as databases of cargo shippers, and 1 million pieces per hour from “unstructured text,” such as government intelligence reports. “Unstructured” information also includes e-mail, news articles and blogs.

“Sifting that enormous mass at lightning speed, ADVISE was to display data patterns visually as semantic graphs” — a sort of illuminated information constellation, as CS Monitor describes it — “in which an analyst’s eye could spot links between people, places, events, travel, calls and organizations worldwide.”

Yet ADVISE, whose existence and scope IMT first addressed in an “Of Interest” item in February 2006, seems to have run afoul of its own ambitious scope. It failed to incorporate federal privacy laws into its system design.

Pilot tests of the ADVISE program were quietly suspended in the spring due to privacy concerns, after Homeland Security’s inspector general and privacy office reports surfaced that live data on real personal, rather than dummy information, had been used in the testing process. At the time, the Government Accountability Office (GAO) warned that “the ADVISE tool could misidentify or erroneously associate an individual with undesirable activity such as fraud, crime or terrorism.”

The Associated Press reports that the pilots used live data, including personally identifiable information, from multiple sources in attempts to identify potential terrorist activity. Among the data the privacy office found had been plugged into ADVISE pilot projects:

• The no-fly list of people barred from domestic air travel and the list of people who require special inspections before flying;

• More than 3.6 million shipping records from a commercial data provider with names of cargo shippers and consignees;

• Terrorist Screening Center lists of people who tried to cross the U.S.-Canadian border at a port-of-entry;

• Classified intelligence reports about illicit traffic in weapons of mass effect; and

• Lists of foreign exchange students, immigrants under investigation and people from special interest countries.

Although DHS spokesman Russ Knocke said ADVISE “was never used in an operational environment” and DHS had assured Congress in 2006 it was not operational, the inspector general found that “on at least one occasion, the data was used to produce classified intelligence information,” according to AP.

The privacy office concluded that although required privacy analyses were ignored, the Privacy Act was not technically violated because the live data were covered by privacy notices issued earlier for other programs that originally gathered the information. Others argue those were too vague to alert citizens how ADVISE would use their data.

The OIG report on the program in June seems to have been the nail in the coffin for this particular domestic intelligence operation. The report criticized the operation not only for failing to provide required privacy protections, but also for failing to take into account the needs of the DHS “components” that ultimately would be expected to pay for the program. The DHS’s own Office of Intelligence and Analysis (OIA) rejected the program in its entirety.

The inspector general also said ADVISE was poorly planned, time-consuming for analysts to use and lacked adequate justifications. That is to say, the inspector general report indicates serious program management failures. ZDNet goes so far as to call the scrapped data-mining program an “inexcusable IT waste.”

This is hardly good news for the DHS, which, according to a 328-page report by Congressional auditors this week (32-page statement HERE), has failed to meet even half its performance expectations in the four years it has been in existence.

As such, “ADVISE is not expected to be restarted,” according to Knocke.

Like UK’s The Register, we advise skepticism.

Earlier: Database of Our Lives: Gov’t System Sweeps for Security

Resources

US Suspends Vast ADVISE Data-Sifting System
by Mark Clayton
The Christian Science Monitor, Aug. 28, 2007

ADVISE Could Support Intelligence Analysis More Effectively
DHS Office of Inspector General, June 2007

ADVISE Report
DHS Privacy Office, July 11, 2007

DHS Ends Criticized Data-Mining Program
by Michael J. Sniffen
The Associated Press, Sept. 6, 2007

DHS: Progress Report on Implementation of Mission and Management Functions
Report and Statement of U.S. Comptroller General
Government Accountability Office, August 2007

ADVISE Data-Mining Program Cut by Homeland Security
by Burke Hansen
The Register, Sept. 6, 2007