The Power of Public Information

On December 6, 1941, the day before the Japanese attack on Pearl Harbor, the Foreign Broadcast Information Service (FBIS), the media monitoring and analysis branch of the Office of Strategic Services (OSS), released a report that read: “Japanese radio intensifies still further its defiant, hostile tone; in contrast to its behavior during earlier periods of Pacific tension, Radio Tokyo makes no peace appeals. Comment on the United States is bitter and increased.”

Radio Tokyo’s increased belligerence on the one hand, and the absence of its regular overtures of peace on the other, constituted a significant shift in the behavior of this state-controlled media outlet and marked a shift in the attitude of the Government of Japan as a whole. Whether or not this report (and others like it) could have changed the course of the events that were about to transpire is a matter of conjecture. However, it echoes a saying by William Donovan (first director of the OSS) that still rings true today: “Even a regimented press will again and again betray their nation’s interests to a painstaking observer.”

Map of geo-tagged Instagram images.

When fighting broke out in eastern Ukraine in 2014, the Russian Government initially denied the presence of its military units or advisors inside Ukrainian territory, claiming that fighting in the region was being conducted by independent separatist militias opposed to the central government in Kiev. However, several social media posts by Russian soldiers told a different story. In one case, a soldier posted that his unit had lost 40 men, and his geo-tagged selfies revealed his location as inside rebel-controlled areas in eastern Ukraine. Photographs posted by other soldiers showed Russian armored columns moving into the region. Thus, the online public information released by a handful of soldiers via their personal social media accounts had inadvertently disclosed state secrets.

“Even a regimented press will again and again betray their nation’s interests to a painstaking observer.”

These examples raise some key questions: How does one find such information? And, once found, how can it be interpreted in an accurate and timely manner? The answer to the first question lies in the constant discovery and assessment of new sources, the regular reevaluation of existing sources, and the deliberate and methodical monitoring, collection, and validation of the information they provide.

The second question may be more difficult to address. Accuracy requires an adequate mechanism for the storage, aggregation, and retrieval of relevant data on demand, as well as the ability to piece together disparate bits of information to form a single, comprehensive representation of reality. Timeliness requires that all of this happen rapidly, while the window of opportunity for action and decision-making remains open, and before the usefulness of the intelligence is overtaken by new developments on the ground.

So in order for FBIS to produce its report, it first needed to identify Radio Tokyo as a valid source, assess its value as a mouthpiece of the state and characterize it as such, monitor its broadcasts regularly and collect the relevant information it provides, track its rhetoric and tone over time to establish a baseline for comparison, and finally compare new information against the baseline to detect noteworthy changes. It also had to accomplish all of this before the outbreak of hostilities in order to give the United States the upper hand.

In the case of the Ukraine conflict, investigators and journalists tracking the relevant social media sources had to act fast to collect and document everything, as once the Russian Government realized that information was being leaked by front line troops, military commanders issued orders banning all soldiers from posting to social media while on active duty, and all existing posts were quickly deleted.

OSINT work relies on a combination of observation skills, expertise, methods, and tools.

The two questions above form the basis of most intelligence work,

regardless of whether the information comes from spies and informants, satellite images, electronically intercepted communications, or a host of other sources. When the information comes from publicly-available sources, such as Radio Tokyo’s broadcasts or Russian soldiers’ social media accounts, the intelligence extracted from that information is called Open Source Intelligence (OSINT).

Disciplines of Intelligence gathering

Similar to other intelligence disciplines, OSINT work relies on a combination of observation skills, expertise, methods, and tools.

Observation is the ability to look at a piece of information and pick out minute details and nuances which may reveal other important information. An image posted online by a person of interest may not seem very revealing at first glance, but upon closer inspection things such as the angle and length of shadows cast by people or objects, weather conditions, reflections of objects on surfaces, and many other background details may yield useful clues.

Expertise is based on a deep underlying knowledge of the subject matter at hand gained from experience. But another form of expertise comes from understanding which investigative methods and tools to use to acquire the knowledge. Regardless of the approach, the goal of expertise is to view any individual piece of information not in isolation, but as part of a larger whole; to understand it in context.

Methods are the processes for developing new leads and generating useful intelligence. For example, when attempting to uncover new information based on existing leads, answering a series of preliminary questions can help us reveal what information is already known and what we can ascertain from it, as well as what information is still missing and how we can go about acquiring it. This method, known as “gap analysis,” is a simple yet highly effective technique.

The goal of expertise is to view any individual piece of information not in isolation, but as part of a larger whole.

And finally, tools are the mechanisms that enable us to conduct our research, monitoring, collection, and analysis. OSINT tools come in many shapes and sizes and can be found everywhere. Some general-purpose search tools the likes of Google, Bing, and Yahoo have become so ingrained in our daily activities that we use them without a second thought. Ironically, the vast majority of users do not even come close to exploiting these tools to their full potential.

Other tools are used only occasionally to find specific types of information, such as looking up an applicant’s resume on LinkedIn, searching for a person’s network of friends and family on Facebook, digging up information on an IP address or domain name of interest, searching for the current and past locations of a specific aircraft or ship using dedicated tracking sites, exploring a particular geographic location using maps, satellite and street images, and Google Earth, and endless other examples.

Still other tools are designed to expediate or automate tasks such as monitoring specific web pages or issues, for example using alerts and RSS readers, collecting and storing online data for later retrieval and analysis, using custom web scrapers and public APIs, and aggregating and analyzing our data, using tools such as Microsoft Excel and Python.

OSINT is not a new discipline, nor is it necessarily a complicated endeavor in every instance; in fact, anyone who uses the Internet to find information can be said to be engaging in OSINT on a basic level. Yet while the average researcher may be content to simply skim the surface of the vast ocean of public information that’s out there, the true experts are those with the desire and the means to explore and fathom its depths, and it is they who are likely to find its many sunken treasures.