The Case for Self-Reliance in OSINT

For Open Source Intelligence (OSINT) practitioners worldwide, June 7th 2019 is, to quote FDR, "a date which will live in infamy." That day, Facebook made a huge, unannounced change to its "graph search" feature, decimating investigators' ability to query the platform using custom search strings and URLs and effectively rendering scores of purpose-built websites and tools, which were until that point a mainstay of most OSINT tool kits, dead in the water.

This was not the first time that Facebook (and other platforms) had made such sudden changes that severely hindered search capabilities and blocked access to information, nor will it be the last.

The lesson to be learned from this is threefold:

First, know your sources and their users. If a primary source is rendered unusable or inaccessible for whatever reason, you need to be able to quickly shift gears and focus on other sources to keep the information flowing. This requires understanding the media landscape and evaluating your sources by asking questions such as:

  • What is the Internet penetration rate in my area of interest?

  • Who are the Internet and social media users and what are their demographics?

  • Is there any form of censorship imposed on the Internet, the media, social media and content sharing platforms?

  • How do people in this area, community or group usually consume and share information? And do they tend to be open or reserved in their opinions?

  • What are the best, most-reliable sources to follow on a particular issue?

  • Which other sources out there carry similar or complementary information? For example, if you can no longer access the Facebook page of a certain group, then does that same group post its content on Twitter, Instagram or other lesser known sources?

The questions dealing with statistical data can be answered fairly quickly by looking at sources such as SimilarWeb, InternetWorldStats, etc., while others dealing with source and user behaviors and norms will likely require more personal "time on target". Yet it is the last two questions that will require the most work, as they involve building a knowledge base of sources and ranking those sources based on their quality and access to relevant information, truthfulness and objectivity, influence and impact, etc. Such source lists can be compiled as standalone files or as bookmarks in your browser of choice.

Second, expand your tool kit. Using a third-party tool for a particular task may be quite convenient (and sometimes unavoidable), yet an over-reliance on any single tool means if that tool ever goes offline, you will be left scrambling for what to do next. It therefore makes sense to maintain a constantly-updated list of alternative tools so you can quickly adapt your approach and continue your work with a minimal loss of time and data. Once again, browser bookmarks on your own device are a good method of maintaining tools lists, as are online bookmarking sites such as directories.

Obviously, if changes happen on the source or platform level (as in our Facebook example), then this will in all likelihood affect all tools designed for that source, which brings us to our next point.

Third, understand the underlying code. Acquiring even a little additional knowledge about the inner workings of the Internet and websites in general can go a long way toward significantly advancing your OSINT skills and allowing you dig deeper and find hidden information that you might have otherwise missed. For example, knowing how to make sense of a website's source code (HTML, XML, JSON, etc.) and how to use your browser's built-in "developer" features, available in most modern browsers, may enable you to download content and locate behind-the-scenes folders, files, and data elements, which are not available to regular users operating on the surface layer alone. Luckily, there are plentiful online resources like w3schools to help get you started.

Furthermore, learning to code in a user-friendly programming language such as Python can boost your abilities further still, enabling you to automate, accelerate, and scale your investigative work in areas such as source monitoring and alerts, data collection and cleanup, and data mining and analysis, as well as allowing you to tinker with and modify existing Python-based OSINT tools.

In the field of online investigations, ignorance is not bliss, and what you don't know can (and probably will) hurt you. The good news is that the steps described above are quite straightforward to implement, and each one will bring you closer to a more independent mode of operation.