This publication—and publications to follow—draws on analysis that involved merging five lists, de-duplicating victim records, and estimating the number of unobserved homicides. The five lists were made available to the UN by the National Police, the Attorney General’s Office, the National Institute of Forensic Medicine, the Vice-President’s Human Rights Observatory (all four of them governmental sources), and the Colombian Commission of Jurists (a non-governmental source). We will publish the estimates file in a few weeks, and a longer scientific article later this year. The recorded and estimated homicides include all homicides in Colombia, including homicides related and unrelated to the various illegal armed groups and security forces.

The estimates were generated with our R package, which we’re calling dga for “decomposable graphs approach,” and which performs Bayesian model averaging for MSE package (see this blogpost by Kristian Lum). For more insight into the “dark figure” of crime in general, and homicides in particular, check out this article by Megan Price and Patrick Ball, “The Limits of Observation for Understanding Mass Violence,” published this year in the Canadian Journal of Law and Society.

Some of the findings regarding the Colombia data change the interpretation of past patterns of violence. We look forward to analysis reinterpreting the impact of the many policies aimed at reducing violence with this new information about total recorded and unrecorded homicides.

HRDAG plans to continue its work with Michael Reed Hurtado, the UN, and Colombian analysts to arrive at best uses for the estimates of unobserved homicides.

CC BY-NC-SA, excluding images.

Organized police documents at the AHPN.

We have accomplished so much in the last 10 years at the Historical Archive of the National Police. And yet, despite the efforts, dedication, and commitment of each person who since 2006 has worked in the AHPN, we still can not say “mission accomplished.”

In 10 years the environment at the Archive has changed so much and become so full of life. Where the building once sheltered unknown stories, over time some of those stories have been revealed. But Guatemala has a long way to go in letting the world get to know more deeply about the secrets within the documents stored there.

Guatemalans and the rest of the world have a very important tool for learning about our stories from one of the protagonists: the former National Police. Thanks to the commitment and professionalism demonstrated during these 10 years, documents have been organized and made available to the world, and they are available to anyone in the world with the click of a button.

This Archive holds not only the secrets of history still to be learned about the recent history of Guatemala, but also the institutional history of the National Police itself. The files within the Archive are administrative files, records generated and safeguarded from 1892 to 1996.

I end this reflection with an invitation for anyone interested in history or human rights to learn more about this invaluable source of information, since it is the largest documentary deposit ever made available in Latin America: 8 kilometers of paper are waiting to be known. More and more students, professionals, families of victims and the general public are represented by those who come to the Archives, but we need to continue spreading the word. There is so much to be learned, seemingly limitless, ranging from quantitative to qualitative research, in this repository.

Mural near the AHPN, Guatemala City, 2013

Ten years ago, in July 2005, human rights officers stumbled upon a nondescript warehouse in a commercial zone of Guatemala City and changed history. They had discovered an archive–its existence kept secret–belonging to the Guatemalan National Police, whose officers committed human rights atrocities on behalf of the government during the civil war.

Inside the building was the bureaucratic detritus typical of a large government agency: 80 million pages detailing shifts worked, tasks assigned, assignments fulfilled, workers’ whereabouts, and who was supervising whom. The documents, which were found stacked on dirty floors, shoved into bags, and tied up in mildewing bundles, provided insight into a security force so incorrigible that it was disbanded by the peace accord that ended the civil war in 1996. With these documents, the Archive supplied clues to the who-what-when-and-where of the wrongful imprisonments, abductions, disappearances, torture, and killings committed by the National Police for 36 years. They would help establish which commanding officers were responsible for which human rights crimes, and, ultimately, result in convictions.

The team working at the Historic Archive of the National Police (AHPN) realized how valuable its contents were, but they worried that the documents would be destroyed or become unavailable. They had an urgent question, and they asked Patrick Ball and HRDAG to help them answer it. The question was this: how can we learn the most about what the Archive has to tell us in less time than it would take to read and record every file?

Patrick, Ann Harrison, and Tamy Guberek visited the Archive in early 2006 and imagined a multi-stage random sample of documents. Soon afterward, Tamy and Daniel Guzmán met with Gary Schapiro and Paul Zador at Westat to formalize the sampling design. In May 2006, Tamy, Daniel, and Romesh Silva went to Guatemala City to conduct the sample, and to implement the data model and coding frame for the study.

People were hired to clean, organize, and classify the documents. Carolina López, who joined the AHPN team in early 2006, was one of them. Over several months, HRDAG selected and trained 20 coders, including Carolina, to select the sample documents and look for specific information relevant to the quantitative research questions, transferring key content from documents into a database. With the HRDAG team, the coders learned about inter-rater reliability, the archival process, sampling, and securing data through Benetech’s Martus program. Carolina has written 7 blogposts about aspects of her work at the AHPN: the first post is here.

Over many years, Patrick Ball and Daniel Guzmán worked on a critical analysis of AHPN documents to establish command responsibility for certain crimes. In September 2013, this analysis helped to convict Colonel Héctor Rafael Bol de la Cruz, a war criminal responsible for the 1984 kidnapping and disappearance of Fernando Garcia, a student union leader. By assessing the flow of thousands of internal police records sent between the director’s office and his subordinate commands, Patrick and Daniel were able to establish the Colonel’s command responsibility and prove that the chain of command was functioning at the time of García’s capture. Colonel Bol de la Cruz was convicted and sentenced to 40 years in prison.

Meanwhile, in 2009 Megan Price joined the HRDAG team and began working closely with Carolina on the sampling and coding. Their goal? To provide a how-to guide, or road map, for any human rights team lucky enough to find themselves with the problem of making sense of millions of pages of documents that could provide clues to war crimes. This year, after years of painstaking work, that project will come to fruition. The result is a comprehensive report describing the collaborative quantitative project–sampling, coding, data analysis, and results–that involves 11 samples and approximately 18,000 documents. It will be published in English and Spanish later this year.

On this tenth anniversary of the discovery of the Archive, we at HRDAG support its continued work, and the work of all human rights teams dedicated to making sure that everybody counts. (The digital archive of the AHPN is maintained online by the University of Texas, Austin.)

HRDAG believes that the concerns highlighted by these examples are important for a wide variety of audiences, including both the foreign policy readers reached by The SAIS Review and the international statisticians reached by IOAS’s Journal.

This article integrates the correction and update of the analysis of event size bias in the Iraq Body Count database that was discussed in detail in a blogpost in November 2014. The updated Syria example also includes additional background information and a deeper contextual consideration of that ongoing conflict.

15CGU Commencement-The 2015 Commencement exercises for Claremont Graduate University. May 16, 2015.

We’re happy to announce that our executive director, Patrick Ball, has been presented an honorary degree from Claremont Graduate University in Claremont, California. University President Deborah Freund presented the degree to Patrick at the university’s 88th annual commencement ceremony on Saturday, May 16, 2015. The degree conferred was Doctor of Science honoris causa.

“We at CGU are thrilled that Patrick Ball accepted our Honorary Degree invitation and joined us for commencement,” said Thomas Horan, CGU Professor and Director, Center for Information Systems and Technology. “Patrick’s work stands as a model for conducting first-rate statistic analyses in a manner that can have meaningful impacts around the world, such as he has done with his pursuit of global human rights.”

Patrick was one of four data scientists to be honored with an honorary degree by Claremont Graduate University this year. The other recipients are Jeff Jonas, IBM Fellow and Chief Scientist of Context Computing; Jean-Baptiste Michel, founder of Quantified Labs; and Fernanda Viégas, co-leader of Google’s “Big Picture” data visualization group in Cambridge, Massachusetts.

CGU Big Data-Big Data Commencement panel discussion and dinner at Claremont Graduate University, May 15, 2015.

The degree statement mentions Patrick’s work in Yugoslavia, Guatemala, and Sierra Leone, and states: “Unconstrained by traditional academic boundaries, you bring together mathematical statistics, computer science, demography, and social science to advance your work. You harness the emerging power of big data analysis, but with careful attention to how and when it can be used most effectively, accurately and—perhaps most importantly—responsibly.”

“It was a tremendous honor to be given this degree by Claremont Graduate University,” Patrick said. “I am proud to accept it in the name of the victims of mass violence who have never been documented by name. If we cannot name them, at least they should be counted.”

On the day before commencement, May 15, Patrick participated in a “Big Data” plenary panel, where he addressed a crowd of more than 100 people at the university during a forum on the potential and limitations of big data. The event was moderated by Thomas Horan, and the panel was composed of Patrick, Jeff Jonas, Jean-Baptiste Michel, Fernanda Viégas, and Mukesh Aghi, President of the United States-India Business Council.

About Claremont Graduate University

15CGU Commencement-The 2015 Commencement exercises for Claremont Graduate University. May 16, 2015.

Founded in 1925, Claremont Graduate University is the graduate university of the Claremont Colleges. Its five academic schools conduct leading-edge research and award masters and doctoral degrees in 24 disciplines. Because the world’s problems are not simple nor easily defined, diverse faculty and students research and study across the traditional discipline boundaries to create new and practical solutions for the major problems affecting the world. A Southern California-based graduate school devoted entirely to graduate research and study, CGU boasts a low student-to-faculty ratio.

Images: Courtesy of Claremont Graduate University.

Graves in Timor-Leste.

This blog is a part of International Justice Monitor’s technology for truth series, which focuses on the use of technology for evidence and features views from key proponents in the field.

As highlighted by other posts in this series, emerging technology is increasing the amount and type of information available, in some contexts, to criminal and other investigations. Much of what is produced by these emerging technologies (Facebook posts, tweets, YouTube videos, text messages) falls in the category we refer to as “found” data. By “found” data we mean data not generated for a specific investigation, but instead, that is generated for some unrelated purpose.

Such data are not new or specific to emerging technology: examples include bureaucratic and administrative records, such as the Historic Archive of the National  Police in Guatemala, documents  generated  by the Documentation and Security Directorate (DDS) in Chad, border crossing records kept by the Albanian border guards during the 1999 conflict between Yugoslavia and NATO, andcemetery registries in Colombia.

Data that has been initially collected for different purposes, or perhaps generated with no specific purpose in mind, has long provided starting points for investigations by truth commissions, legal teams, inquiries by the United Nations, journalists, and other organizations.  Emerging technology is expanding the list of possible sources of this valuable data even more. However, many of the challenges and uses of such data remain the same, whether the data are physical (e.g., archived paper documents) or digital (e.g., YouTube videos).

One of the primary challenges to “found” data is simply preserving it – photocopying border crossing records in Albania, taking digital photographs of decaying documents in Chad, or archiving YouTube videos and Facebook posts.

Once preserved, criminal investigators will need processes to both sift through a large amount of data to identify information most pertinent to case investigations and to verify the authenticity of information. Several organizations are tackling this problem as described in a report by the Human Rights Center at University of California, Berkeley.

In addition to potentially unique challenges of preserving and meeting the demands of the chain of custody, data generated by emerging technology also pose many of the same challenges as traditional data sources regarding questions of how the data were collected and what information the data include and exclude.

The experience of the Human Rights Data Analysis Group (HRDAG) over the past two decades in more than 30 countries has taught us that most data sources are incomplete, regardless of how systematically, carefully, or technologically they may have been collected. This is not to question the validity of any particular set of data, but rather a recognition that raw data is merely a starting point. Data must be contextualized with qualitative information and analyzed using appropriate statistical techniques that take into account the way in which the data were collected and what may be missing from the data.

For example, regardless of the data source, researchers should always ask themselves, “Whose stories are not captured by this source?  How are those stories, victims, and witnesses likely to differ from the ones documented by this source?”

Field experts can address these questions qualitatively.  Recent research in Syria has indicated that ongoing documentation efforts are less able to reach certain geographic areas, sometimes due to lack of infrastructure (no electricity or cell phone coverage) and other times due to which group has control of a region and the affect this has on the relationship between community members and data collectors.

Statistical analyses, under certain conditions, can directly estimate the amount of information missing from raw data. Our own work at HRDAG focuses primarily on a class of methods called multiple systems estimation, which uses the overlaps between several incomplete lists of human rights violations to determine the total number of violations, but there are a variety of analytical techniques designed to account and adjust for missing data.

This poses its own challenges, as not only do such analyses require experts trained in appropriate methodology, but if such analyses are to be presented in court they also require the experts to be able to adequately explain potentially complex methods to a not-necessarily-technical audience of lawyers and judges.  These challenges must be met, as quantitative analyses that fail to account for missing data tell only part of the story, at best, and at worst may draw precisely the wrong conclusions.

We welcome Dr Ahmed’s summary of various points of scientific debate about mortality due to violence, specifically in Iraq and Colombia. We think these are very important questions for the analysis of data about violent conflict, and indeed, about data analysis more generally. We appreciate his exploration of the technical nuances of this difficult field.

Unfortunately, parts of Dr Ahmed’s article focus on sources of funding that IBC and Professor Spagat have received, and on speculation about how such funding might affect their substantive conclusions. We find such criticism to distract from the important points of scientific debate.

Policy arguments that draw on scientific findings will inevitably include scientists who have personal opinions and political associations. Those personal opinions often influence which areas we choose to research. The challenge for all scientists is to do rigorous, reviewable, transparent work that supports or rebuts existing theories — or advances a new theory — about how the world works. Such work should be done without the scientists working in fear of personal attacks.

Good scientific work does not depend on who did the science. Every scientific finding must be assessed by the quality and appropriateness of the data and methods, and their relevance to the theory being tested. Science depends on anonymous peer review precisely so that the reviewers are not biased by who the authors might be. The personalities and biographies of the scientists involved must be irrelevant to the quality and veracity of the scientific result. Similarly, science is tested by replication: given a finding, another scientist should be able to use the same data and method to reach the same result. Anonymous review and replication are fundamental to science precisely in order to distinguish scientific knowledge from the scientists who do the work.

In particular, we think the article’s criticism of the United States Institute of Peace (USIP) is excessive and inaccurate. Our colleague Professor Amelia Hoover Green, the lead author of our white paper criticizing the Dirty War Index (DWI) measure cited extensively in Dr Ahmed’s article, has been supported by USIP grant funding. Our sister project, Martus, has received funding from USIP. Many other academics whose work is deeply critical of U.S. policy, Colombian policy, and the academic proponents thereof have received USIP support. (For full disclosure, HRDAG’s current and past funders are listed here.)

HRDAG has many times disagreed with Professor Spagat and IBC, and at other times, we have agreed.  HRDAG and Professor Spagat have had productive debates, and we expect to have more with him in the future, because that is how scientific research works and advances.  We appreciate that Professor Spagat and IBC have been forthcoming with information and data when HRDAG has requested it.  This is noteworthy because Professor Spagat and IBC are fully aware that we are likely to disagree with them.

Lastly, HRDAG would like to correct two of Dr. Ahmed’s points. He repeatedly asserts that IBC might be missing many large events (e.g., massacres). HRDAG’s analysis, and a 2008 RAND study cited by Dr Ahmed, leads HRDAG to conclude that IBC is likely missing many smaller events with fewer victims, while missing relatively few large events. Further, Dr Ahmed refers to HRDAG as based in Los Angeles. Although our fiscal sponsor, Community Partners, is based in Los Angeles, we are happy to report that we are based in San Francisco.

The main function in this package implements a model created by David Madigan and Jeremy York. This model was designed to overcome two of the principal challenges we often face in multiple systems estimation: (1) adequately incorporating statistical uncertainty about the structure of dependence between the lists into the estimates; and, (2) obtaining estimates in the presence of data sparsity that causes maximum likelihood estimation to be highly unstable. In our research, we find these benefits to result from Bayesian approaches to MSE generally. This Bayesian model offers a principled way to account for list dependence uncertainty, allow estimation with sparse data, and, where appropriate, to incorporate expert input about the likely number of undocumented deaths.

The dga package includes some additional functions to process, visualize, and run diagnostics on data. It also includes functions to visualize the results. (The package does not perform record linkage.) Currently, dga can produce estimates for three-, four-, or five-list systems. In the future, we plan to expand this to up to seven lists. A limitation of the method is that it becomes computationally intractable to enumerate all of the possible list dependence structures for more than about seven lists. Fortunately, large numbers of lists are rarely encountered in human rights applications. In later versions of the package, we hope to incorporate a stochastic search feature to allow scaling to higher numbers of lists.

Let us know what you think about our package!