The importance of OSINT in cyber security
Introduction to OSINT
There is a growing discipline which can be used to take advantage of the depth of public information now sitting online just beyond our keyboards. The same discipline can also highlight vulnerabilities to an organisation or individuals who may be sharing too much personal data with the world. This discipline is called Open Source Intelligence or OSINT (pronounced “oh-sint”) and plays an increasingly important role, both in the success of modern cyber-attacks, and also with respect to defensive-in-depth security assessment for organisations. In this blog, I will cover the importance of OSINT in cyber security, its history, what it is used for and where it sits in relation to penetration testing.
OSINT is not tied to a specific field and has an almost unlimited number of use cases. The heart of OSINT aligns with its traditional roots in law enforcement and federal security services (at local, national, and international levels). However, in recent years, with the natural cross-over with cybersecurity domains, such as penetration testing, the term OSINT has also become synonymous with certain types of security assessment activities. This is because a small amount of extremely potent information (obtained from open sources), has the ability to pose a threat or be used to facilitate certain types of online attacks against people and organisations with potentially devastating consequences.
What is OSINT?
Open Source Intelligence (OSINT), relates to the gathering, analysis and use of information retrieved from publicly available sources. These public sources predominantly relate today to data which can be retrieved via the Internet, from online resources such as search engines, social media platforms, online mapping tools, news articles, public registries, and websites belonging to individuals or specific organisations.
The emerging importance of OSINT
Oscar Wilde once said:
“Private information is practically the source of every large modern fortune.”
If we consider the time at which Oscar Wilde made this statement (over 100 years ago), his words certainly still ring very true today. However, an opposing statement could now be equally true,
“Public information is often the source of many large modern misfortunes.”
This statement, I think, is a reflection made increasingly clearer by the constant flowing digital waters running off the cyber landscape we rarely disconnect from. For example, information leaks in the public domain often correlate to misfortune and they do appear to occur in the mainstream media with constant regularity, including the data breaches of sizeable companies, or the personal exposures of private communications relating to either high-profile individuals or A (through to D) list celebrities.
The current climate of cancel culture is undoubtedly intertwined with online platforms, devices, and user accounts aligned to social media usage. More often than not the root cause of the misfortune coincides with a point in time where information has been exposed publicly or too much information has been shared digitally with an unintended or untrustworthy audience. Furthermore, often the blame can be related to poor digital hygiene or a lack of education can be part of the reason people overshare.
Software design can also be part of the problem. Putting usability ahead of security can lead to many people leaking data unintentionally, as opt-out settings are overlooked in favour of exposing as much of our user-created content as possible to the widest audience (by default!). The notion of a “private” space in the cloud should still, in my opinion, be taken with a pinch of information security salt.
With the history of global adoption of large digital information sharing platforms such as Facebook, Twitter, Wikipedia, Instagram, LinkedIn, and Tiktok (and thousands of other companies offering up their data sets, either for paid for or free tier services) now give the discipline of OSINT more power. Unfortunately, this power can be used to exploit an abundance of data points out there relating to individuals and organisations.
The OSINT origin story
You’d be mistaken for thinking that OSINT is simply an overzealous usage of Google. In reality, the discipline is vast and has a rich history dating back prior to the worldwide adoption of the Internet.
The heart of OSINT lies in investigative professions such as law enforcement and journalism. To be proficient in OSINT today usually means applying OSINT to a specific lane of investigation and so the tools, techniques and knowledge required equates to familiarity with more than just how to drive a few search engines.
Certain data types are useful depending on what the “mission” of the OSINT investigation is, and the tools, techniques and resources vary significantly depending on what you are looking to discover. This includes personal data, organisational data, image data, audio data, video data, geographical data or financial data.
OSINT specialists are referred to as “OSINT investigators” within the field and finding ones with deep and wide skill sets, are rare. However, the resources to support the learning and proficiency of OSINT have matured over recent years. As the discipline has become more prevalent in the digital realm so too have the responses to the tricks and techniques OSINT investigators have used.
Many social media sites have literally changed the way they do things, to stop data points from being mined or the dots connected by those looking to leverage online data to support their investigative OSINT practices. Remember OSINT can be used for good, but it can be used by the bad guys too – so often, OSINT techniques can become outlawed very quickly if they are abused, typically with paywalls and rate limiting being the response.
OSINT relies on many free services and tools which can often be available one day but may not be useable or fit for purpose the next. The OSINT investigator must be able to skin a cat multiple ways, but luckily the Internet is a place where avenues for exploration are forever increasing. This means that the challenge of the OSINT investigator is, much like that of a pen tester, they just need to keep up-to-date with what the latest and greatest tricks are, in parallel with having a robust investigative methodology to succeed.
What is OSINT used for today?
To further frame the discussion, it may help to list some of the missions an OSINT investigator is looking to complete. The following examples are some common use cases for OSINT:
- Finding intelligence to support the last seen, probable or current locations of an individual (missing person, wanted criminal).
- Finding intelligence to gather information relating to criminal investigations (evidence preservation, cybercrime).
- Finding intelligence to conduct a threat assessment (relating to individuals (e.g. stalking), events, diplomatic/VIP protection)
- Investigative journalists may often apply OSINT techniques to support their work.
- Due diligence processes (background checking employees, vendors, contractors, 3rd party supply chains)
- Finding intelligence to determine potential vulnerabilities against an organisation (e.g. gathering intelligence on employees such as personal information, email, passwords in breach databases, exposed services, files, contact information, company structure, office locations etc.).
- Conducting OSINT investigations to support other security assessments including Red Teaming, or Phishing. (e.g. mapping exposed online targets, and uncovering data about technologies used by the organisation, finding domain names registered by the target organisation(s) or IP address ranges owned etc.)
OSINT – a hacker’s best friend
As previously mentioned, just like OSINT can be used for good, it can, unfortunately, be used in nefarious ways. Testing the security of web applications is a core feature of the penetration testing projects security testers like Evalian, complete during the average work week. I have been testing the security of online resources for over a decade and the more I test the more I note various tricks, tools and techniques for improving the success rate in finding vulnerabilities. In recent years, I have started to see increased success in exploiting web applications, not necessarily by knowing more about the technology I am interacting with (although that is a very important part), but by finding out more details in relation to the people, and organisations linked to the target assets I am assessing for vulnerabilities.
A small piece of pertinent information can make or break the success of an exploit chain, by turning a hunch into a working proof of concept attack. Information, whether it’s private, or public has greater value than often meets the eye these days – especially if you are a hacker.
Recently I was able to prove an access control flaw existed in a web application’s API (I also take a deep dive into how API testing is conducted in my latest blog on why API security is critical for your organisation) by being able to determine the client relationships of one of our customers, simply due to a post made by their CEO on LinkedIn. Using information mentioned by the CEO in the post I was able to perform targeted enumeration against an API request parameter and found I was able to read data related to different client tenants of the web application, by using this gathered intelligence. Prior to this, I had tried thousands of generic values with no success. Only from applying the discipline of OSINT did my proof of concept pay off.
Commonly used OSINT tools and resources
Often a good, well-rounded OSINT investigator will also adopt non-OSINT tools as well as the plethora of free online resources. Premium services that do all the work for you are not really OSINT. Much like in pen testing where comprehensive vulnerability scanners can do some heavy lifting, the same applies in the OSINT world to ensure a depth of coverage and due diligence. For example, in law enforcement, officers can run number plate matches, and look up governmental databases such as criminal records and other information to help provide context and intelligence supporting the mission – but these are closed and private systems.
Take online breach data services as an example; they provide well-maintained datasets which contain compilations of all well-publicised data breaches. Within seconds an entire domain belonging to an organisation can be scrutinised for matching credentials, with a connection to the searched-for domain, across hundreds of breach datasets (most likely amounting to terabytes of data). Individual OSINT investigators maintaining their own breach datasets, often need to work many hours to have anything anywhere close to effective or as comprehensive as these types of premium comparative services.
Ultimately OSINT is a research-based activity, which supports an investigation against a mission brief. So typically for myself, as a pen tester, my mission brief is often finding vulnerabilities in target X, where X could be a web application, an API or a range of IP addresses owned by a client.
I can use OSINT tools and resources to support what I do, to give me the same opportunities that an external threat actor would have in being successful in that mission. Ignoring the wealth of data out there to help you simulate an adversary is simply not realistic, so I encourage my team to always have their OSINT hats on when conducting their pen tests because you just don’t know what could provide an advantage.
There is no substitute, in my humble opinion, for human lead pen-testing, and throwing intelligence in the mix is useful, but this often requires OSINT knowledge and experience to connect the dots efficiently.
I could list one hundred tools and techniques related to OSINT investigation, and the best place to get an idea of the scope of the tooling and resources which can be adopted into an OSINT investigation are extensive. The “OSINT Framework” is typically the resource we point to for those unaccustomed to the practice. A quick Google does not constitute a comprehensive OSINT investigation.
This site is driven by a simple interactive graph interface. As an example, the following screenshot shows the investigative branches one can follow relating just to IP address (a data point familiar to us pen testers):
People and organisations nowadays are all intimately linked with digital systems, and much of this interconnectedness can be determined through pen testing techniques, combined with OSINT tools and resources when following a robust investigative methodology. As you can see, the expansion of just one thread of investigation can provide many options. An OSINT investigation, however, doesn’t need to be an exhaustive check of every tool and every avenue of research, it has to keep coming back to the central point of using OSINT, which is the mission. Simply dumping out large volumes of data is not what OSINT is about.
OSINT: A deep dive
The depth of an OSINT assessment can go as far as monitoring live audio feeds from streaming radio services. I have seen tools which you can put in search terms and have them bring up hits in broadcasts within Emergency Services, Aviation, Marine, Weather and even Amateur radio services. Certain types of radio communications can also be mapped to locations (e.g. flight trackers, or sites like http://rx.linkfanel.net/ will show you where wideband radio receivers are geolocated). This sort of real-time detail usually relates more to law enforcement-type missions, but it shows there is almost no corner of the Internet that OSINT can’t use to its advantage.
The darker side of OSINT
Other more obscure OSINT territory includes investigations on the Darknet, but we strongly advise that no one casually venture there unless you really know what you are doing. There are sites on the Internet which provide a somewhat safe distance perspective and allow data on the Dark web to be scrutinised through a detached search engine service, such as AHMIA (https://ahmia.fi/).
Venturing uninitiated into the Darknet is like going to the worst parts of the most dangerous places in the real world by yourself, at night, without your phone and with no information about where you are. The uninitiated open themselves to threats just by going there. The Darknet, however, does have a place in OSINT investigations but is geared toward law enforcement investigations as opposed to considering general cyber security best security practices.
Individuals casually using the Darknet, can actually cause problems for themselves and the mission they may be investigating unless the OSINT investigator is well-versed in Operational Security (a subject which could fill an entire article on its own).
In the world of law enforcement, it is typical that all OSINT investigations are conducted on anonymous connections to the Internet. This is done by using burner email accounts for all online services, using pay-as-you-go mobile data services and/or VPNs, and having AI-generated profile pictures on social media platforms to bypass bot checks (again to create more burner accounts). They also use fresh virtual machine snapshots for every investigation and apply all privacy filters on web browsing clients, all to prevent any exposure of themselves during the investigative process.
OSINT and Penetrating Testing
OSINT, in penetration testing-related investigations, is often used to map data points to confirm certain pieces of information, which are typically useful to attackers (e.g. IP addresses, hostnames, email addresses, usernames, passwords, password hash values, URL paths etc.). The example scenario detailed below demonstrates a mixture of various OSINT tools and techniques which can be combined to gain access to a system.
Note: Typically, these adjunct investigations sit outside the scope of traditional pen testing methodologies.
For example, you may start with a single work email address of an individual (format: firstname.lastname@example.com). In OSINT territory the uniqueness of their name may get you a long way on its own.
- You Google their name and find a video of them on YouTube discussing a work-related topic.
- You now have potential visual confirmation of their likeness, you check their LinkedIn profile, and confirm the likeness via their profile picture (and can equate the two data points as relating to the same individual).
- You then perform more searches to extract PDFs from Google featuring this individual’s unique name, and you see they are a treasurer for the local boy scout group and find a mobile number, and personal email address inside a PDF of meeting minutes.
- You then put their personal email address into a breach data search engine and find a cryptographic hash of their password from a recent data breach.
- You then run a comprehensive cracking tool including rules which focus on variants of their son’s name which you determined from their Facebook profile (which you matched against their own profile picture from the likeness in the first YouTube video and LinkedIn profile photo). The password crack successfully determines the password.
- You perform a comprehensive review of the organisation that the target individual works for and enumerate a dozen cloud services using scripts to query DNS services and open services.
- You find three public web-facing web applications in this listing of services previously enumerated, one of them is in the scope of the penetration test you are performing.
- You enter the password into the application’s login page with an incorrect password and get confirmation that the original provided work email address exists as a registered user, but the password is wrong.
- You then seek permission from the client to attempt the login to the application with the user’s credentials found from the breach data, and you now have access to organisational data, courtesy of OSINT and because someone re-used an old password on a work-related online service.
This above investigative thread using OSINT highlights the importance of multi-factor authentication, and educating staff on best security practices when it comes to choosing suitable passwords for work vs. personal online accounts. Starting from just one data point you can often find several that can lead to a successful exploitation and all made possible by making proven connections between them.
Such investigative flows are not random luck, they emerge from an understanding of how to pivot during an investigation on the discovery of data points. Both OSINT and pen testing are methodology-based disciplines, and each requires a need to traverse pathways of investigation given a starting point or target.
Download your free Guide to Penetration Testing
How to Succeed at OSINT
Like anything, practice makes perfect, but it is often good to learn from the best. One of the most respected figures in the field of teaching modern-day OSINT techniques is Mike Bazzell. Mike Bazzell wrote the book(s) on OSINT and has been publishing authoritative and comprehensive books on OSINT and privacy for the past decade. He currently is the figurehead for what I would say is the top benchmark for OSINT online training with over 80-100 hours of video instruction (that’s like watching the whole of Game of Thrones!). His website also maintains a blog.
The training course offers a separate accreditation called OSIP (Open Source Intelligence Professional Certification), a professionally recognised certificate in the discipline of OSINT.
Having personally undertaken training on this platform and attended courses run by protégé’s of Michael’s I can say completing such courses and certifications is no mean feat and requires a lot of time, effort and dedication.
However, dipping into such training is also beneficial for people looking for high-quality guided learning in the subject to round out their current skills, whether they work in law enforcement, cyber security, or investigative journalism, there is a vast wealth of information and practical use cases detailed in the contents of this course. The examination is not for the faint-hearted and is geared towards someone probably looking to pursue OSINT on a daily basis as part of their profession.
Conclusion
To succeed at OSINT really depends on a number of factors, including training and the domain of investigation which you are applying OSINT to. Additional skills are often needed to complement the OSINT mission. As an example, you can be good at OSINT but not a very good pen tester, or not a particularly good detective, so OSINT isn’t going to necessarily improve your success rate unless you have mastered the fundamentals in your own discipline.
The great thing about OSINT is that it can be done by anyone, you don’t need a lot of special or expensive equipment. It is worth, however, bearing in mind that it doesn’t mean that everyone can do it equally, safely, or comprehensively, but with the right OSINT skills and tools, security teams can prevail against threat actors, and use it as a strategic part of protecting their valuable data.
How can Evalian help support your security teams?
Our skilled penetration testers and cyber security experts can work in partnership with your organisation, to help you build a robust security posture. With up-to-date knowledge and certifications such as CREST, TigerScheme, CISA and OSCP to name a few, our specialists will roll up their sleeves and get stuck in as an extension of your team.
Contact us today to speak about your requirements, or to get a fast quote.
