Manual penetration testing services versus automated tools

In this blog, we discuss the benefits of manual penetration testing versus automated tools. In a manual penetration test, a suitably qualified and experienced penetration tester will emulate a real-world attack on a system, application or network using a mixture of manual and automated techniques. The tester will seek to exploit combinations of vulnerabilities and circumvent security features to determine the organisation’s security posture.  

By contrast, vulnerability scanners are automated tools that identify potential vulnerabilities, such as outdated software versions, missed patches and misconfigurations. A vulnerability scanner is a useful and relatively quick way to detect an organisation’s exposure to surface vulnerabilities – these being vulnerabilities that exist in isolation, independent of other weaknesses.  

Vulnerabilities, though, rarely – if ever – exist in isolation. A real-world attacker would look to leverage a combination of vulnerabilities in an attack, as penetration testers do when they test an organisation’s infrastructure.   

Several penetration testing companies today offer automated security analysis and automated penetration testing tools. Automated tests are increasingly powerful and impressive, but it is in the interest of organisations to understand the limitations of these tools and where their appropriate usage lies. Put simply, an automated analysis or test cannot offer the same depth of assurance as a manual penetration test.  

This is not to say that automated analyses do not have their place in vulnerability management. When used correctly, they can support the process of discovering and remediating vulnerabilities, enable continual assessment and remediation, and ensure a strong security posture. Penetration testers themselves use automated tools as part of the testing process.  

Manual penetration testing service

Penetration testing features a mixture of manual and automated techniques used to simulate an attack on an organisation’s IT infrastructure. While penetration tests mostly look to exploit known vulnerabilities, the tester should use their expertise to identify weaknesses and exploit them. For more information, read our guide to penetration testing.  

The penetration testing process is considered an ‘active’ analysis, whereby the penetration tester proactively identifies and exploits vulnerabilities – often in combination – just as a real-world threat actor would. Vulnerability scanners are used as part of the penetration testing process to discover weaknesses for manual exploitation during the discovery phase. The scanner is used to locate potential vulnerabilities, while the attack phase of the penetration test exploits the vulnerability and confirms its presence.  

Benefits of manual pen testing

• Assurance: Penetration tests are much more in-depth than automated scans. By simulating a real-world attack, they help the organisation to understand and improve its ability to detect and respond to attacks. 
• Reduces the likelihood of a data breach: The remedial advice offered in a penetration testing report helps organisations to improve their cyber security posture, thus reducing the possibility of a successful data breach.   
• Important for compliance: Penetration testing is mandated under Payment Card Industry Data Security Standard (“PCI DSS“) and various regulations mandate security assurance activities.  
• Fulfils supply chain obligations:  As supply chains become more complex, buying organisations are starting to require evidence of annual penetration tests from suppliers as part of contractual obligations. Copies of penetration testing reports are also increasingly requested during supplier due diligence exercises. Read more on this in our supply chain security guidance.

Shortcomings

• Finding trusted provider: A penetration test simulates an attack on IT systems. Organisations need to ensure that they trust suppliers to penetrate their systems ethically. To do this, they should look for companies that are accredited by CREST. Public sector organisations will likely need CHECK accreditation.  
• Time-consuming: The manual, in-depth nature of pen-testing is what makes it so comprehensive and valuable, but this also means that tests are lengthy and take much longer than a vulnerability scan. 

Automated vulnerability scanning

As a standalone, vulnerability scanners should be used to assess and review known vulnerabilities systematically. Organisations typically carry out these scans monthly to evaluate the health of their infrastructure. These assessments are usually performed with a commercial scanning tool like Nessus or Qualys or an open-source equivalent such as OpenVAS. Because the search is done without human intervention, these scans can be quick and vast.  

Once complete, the tool creates a list of security flaws for the IT team to remediate. These are typically prioritised based on the CVSS 3.1 vulnerability scoring methodology, which helps to prioritise the remediation work required. The list of vulnerabilities generally is very long, as it includes every possible or potential issue identified without human verification.  

Vulnerability scanning can be a labour-intensive activity for the employee to interpret the results because of the sheer volume of issues identified. Scanners are known for producing many false positives, which the assessor will have to sift through to find valid weaknesses. Likewise, the scanner may consider something to be a high risk when the context (known to a human) is that it is much lower risk. 

Ultimately, vulnerability scans verify that your organisation has reached a minimum level of security. However, they do not confirm whether the vulnerability can be exploited and do not consider the risk of exploitation in the context of the target organisation’s security posture. As such, scans have their place in security management, but they do not deliver the depth of assurance an expert penetration test gives.  

In an ideal world, we’d recommend regular scanning as part of your vulnerability management and patching programme, coupled with at least annual penetration testing by a qualified and experienced penetration testing services provider. This ensures you apply good security hygiene with regular, independent testing to assure nothing has been missed. 

Benefits to automated scanning

• Important for compliance: Vulnerability management is foundational security control. A personal data breach caused by the exploitation of a vulnerability that could have been picked up and prevented through scanning is, for example, likely to be a breach of Article 32 of the General Data Protection Regulation (“GDPR“). Likewise, PCI DSS compliance requires quarterly scanning of the Cardholder Data Environment.
• Makes pen-tests more valuable:  Regularly checking the health of networks and services can help mitigate the risks of ‘low hanging fruit’, which offer threat actors an easy foothold into enterprise infrastructure. Regular scans also make penetration tests more valuable by alleviating these threats, as the testers can focus more on discovering complex vulnerabilities.   
• Quick and repeatable: Automation requires little human intervention and works at high speed. It’s therefore easy to incorporate these tools into the business and run regular scans. 

Shortcomings

• False positives: Automated tools are renowned for producing false positives. They tend to produce lengthy reports, which assessors must sift through to find valid vulnerabilities. This can be slow and require specialist security knowledge. 
• Unfound vulnerabilities: It’s unlikely a scan will find all potential weaknesses. Vulnerabilities are being discovered by the day, and some may be too complex for the scanner to detect. Moreover, scanners don’t leverage heuristic thinking, meaning they only assess vulnerabilities in isolation of each other, as opposed to analysing the potential of a combination of exposures to be leveraged in an attack.   
• Generic remediation guidance: Once the list of vulnerabilities is received, it will typically come with generic remediation information that the organisation will need to validate and then work out how to apply in the context of their systems. 

Why do organisations need both automated tools and manual penetration testing ?

Penetration testing and vulnerability scanning are two essential components of an effective vulnerability management process. However, one cannot replace the other. Organisations should use automated scanning as a cost-effective means of discovering and mitigating common vulnerabilities in between more in-depth penetration tests.  

Moreover, regular vulnerability scanning will improve the efficiency of penetration testing engagements, enabling the testers to focus on more complex vulnerabilities rather than ‘low hanging fruit’ that could be remediated through a vulnerability scan.  

Because penetration testing is more costly and intense than vulnerability scanning, we recommend testing IT environments annually. Organisations should also conduct a penetration test after any significant changes to their IT infrastructure. The results of the test should be digested and acted upon with urgency. Companies should aim to remediate the urgent vulnerabilities discovered in the test as soon as possible after the debrief with the testing team.  

Ultimately, by combining penetration tests with regular vulnerability scans, organisations can create a well-designed vulnerability management programme that consists of regular scanning combined with periodic penetration testing. This can help mitigate many types of attacks and improve the organisation’s security posture.   

Need advice on penetration testing?

If your organisation needs help running a penetration test on an application or infrastructure, we’re here to help. As a trusted provider of penetration testing services, we can assess your environment and run a full penetration test. We can also advise you on any follow-up actions or remediations from our findings. Contact us for a friendly chat.