The Paradigms of Security Testing

Security testing is an indispensable part of any comprehensive security program. However, depending on the particular security needs of the organization in question, different security testing approaches might be more valuable than others. Gaining a deeper understanding of the various paradigms of security testing and how each contributes to bettering an organization's overall security posture will help companies choose which type best suits their present security needs. This understanding will also help security testers advise their clients and provide the most value in response to their client's unique situations.

Various testing types may be warranted depending on the nature of the organization’s network and application infrastructure, security program size and maturity, compliance requirements, and business models. We will define and briefly describe some of the most common categories and subcategories of testing, then discuss what unique benefits each provides to an organization’s security and when each is best applied to provide maximum value. It is important to note that this does not represent an exhaustive list of all possible security testing methodologies or areas, as these are apt to be slightly different from company to company and are also subject to change with technological advances.

Penetration Testing

The first set of testing paradigms falls under the category of penetration testing. Penetration testing involves testing a scoped environment or application for vulnerabilities that could lead to application or system compromise, network access, privilege escalation, lateral movement, or sensitive data access. The coverage of vulnerabilities is generally as broad as possible within the scoped timeframe. The primary purpose of penetration testing is to discover and exploit vulnerabilities, prioritizing severe vulnerabilities based on the likelihood of exploitation and impact. Testers may sometimes attempt to bypass endpoint detection and response (EDR) or other defensive security tools, but stealth is not generally prioritized within penetration testing. The security operations teams of target organizations are typically informed of the test in advance and often updated as testing proceeds. The nature of the target scope defines the subcategories of penetration testing. 

Network Penetration Testing

Network penetration testing involves the testing of networked infrastructure and systems. External network penetration testing is performed from the Internet on the target organization's external network perimeter to find paths to access exposed services or applications and, ultimately, to the target's internal network. Techniques often include open-source intelligence (OSINT) gathering, DNS reconnaissance, host discovery and port scanning, and automated vulnerability scanning as part of the initial reconnaissance and enumeration phase, followed by manual analysis, fuzzing, and exploitation. A pervasive exploit path in external penetration tests is credential attacks such as password spraying using common password variants and usernames gathered from open-sourced intelligence (OSINT).  If testers can exploit any such vulnerabilities to gain initial access, they may attempt privilege escalation or lateral movement. 


Though a robust external network security posture can significantly reduce the probability of an attacker gaining access to internal resources, in the perpetual arms race between attackers and defenders, it is not ultimately a question of if the perimeter will be breached but when. Only one weak point in an application, service, or employee security can allow unauthorized entry. The question becomes whether and to what degree an attacker with internal network access could move laterally to other systems, escalate privileges within the network, and access sensitive data therein. Using this assumed breach methodology, these are the questions an internal penetration test can help answer. Techniques used in internal tests include those used in external tests and others specific to internal environments, such as local network protocol spoofing attacks and attacks against Microsoft Active Directory (AD) infrastructure, file shares, databases, and other internal network applications. Typically, a more extensive variety of services are exposed internally, so opportunities for exploitation of software vulnerabilities and security misconfigurations are often far more numerous. Internal penetration testing can be done on-site or by tunneling through devices provided to clients. 

Wireless network penetration testing comprises attempts to access targeted 802.11 wireless network infrastructure. Various attacks against WPA2-PSK and older protocols are typically used, including de-authentication attacks with handshake capture and cracking or PMKID capture and cracking and dictionary attacks against pre-shared keys. Rogue access point honeypot attacks may also be used to capture user credentials. Wireless tests are often carried out simultaneously with internal network testing. A critical security flaw that usually allows malicious actors to gain broader internal network access is improper network segmentation between wireless networks intended for guests, where security is often lax, and internal corporate networks. 

Upon completion of any network penetration testing, a client can expect a detailed report of any discovered exploitable vulnerabilities that are present, if applicable, along with a risk rating for each vulnerability calculated based on exploitability and impact and tactical remediation advice. Clients can also expect to gain additional general insights into their overall attack surface and threat models and broader strategic advice to improve overall security posture. 

Application Penetration Testing

In contrast to network testing, application penetration testing is carried out by targeting individual applications and integrated components. These applications may be mobile, desktop, or, most commonly, web applications. Several models of web application penetration testing exist. Black-box testing involves testing the application with no initial access besides basic network access to the endpoints. Authenticated access must be gained by penetration testers without assistance from the client organization, either through self-sign-up workflows or through vulnerabilities in authentication or authorization. Grey-box testing additionally involves testing from an authenticated perspective using provided credentials, often a low-privileged account or multiple accounts with various privileges for testing cross-account authorization vulnerabilities and privilege escalation. White-box, or source code-assisted testing, is by far the most comprehensive, encompassing the other forms of testing and source code review to find vulnerabilities in functions that would unlikely be discovered through manual fuzzing externally. 

White box testing is undoubtedly the best value method for application penetration tests, and it is what we recommend to clients. In cases where no source code access is available or access is impermissible for legal or compliance reasons, grey-box testing provides the second-best alternative. Black-box application testing is not typically recommended, as it often involves incomplete coverage of application functionality and provides the most negligible value in vulnerability discovery for time and money spent. 

Application testing encompasses web, mobile, desktop, and embedded/IoT applications. The most commonly performed application tests are web application penetration tests, followed by mobile. A typical application penetration test will involve testing for vulnerabilities in an application’s authorization and authentication, injection vulnerabilities such as SQL injection, OS command, XML injection, cross-site scripting (XSS), file path traversals, and many other client-side and server-side vulnerabilities. Access to source code can assist both in finding more such vulnerabilities and in identifying the root causes to provide more targeted and specific remediation advice. From a white box application pentest, clients can expect to gain deeper insights into the vulnerabilities present in their application and the insecure coding practices, misconfigurations, and policy and control oversights that underlie them.

Physical Penetration Testing

Physical penetration testing targets an organization’s physical infrastructure, including the facility perimeter, buildings, and related physical access control systems such as doors, locks, biometric access systems, and other physical security infrastructure for vulnerabilities in an attempt to gain unauthorized access to physical locations of the target organization such as server rooms, vaults, executive offices, or other sensitive areas. Physical penetration tests also might involve some degree of social engineering to exploit personnel to access restricted locations within a building or grounds. Consultants who perform these tests request a letter from the organization’s representatives who authorized the pentest, containing written authorization and contact information for security personnel who are aware of the test, which testers can carry on their person. This letter is often referred to as a “get out of jail free” card for reasons that should be obvious. Physical penetration testers who have had the misfortune of getting caught without such a letter have been arrested for breaking and entering, an eventuality that might make for an awkward follow-up meeting with the client, to say nothing of unforeseen legal bills. Physical penetration tests provide clients a means to address blind spots or gaps in their physical access control systems and intrusion detection capabilities, reduce their physical attack surface, and identify connections between physical security and other aspects of their overall security posture. 

Cloud and Organizational Penetration Testing

Several other methodologies don’t fit neatly into the previous categories but still fit within the broader category of penetration testing. Many organizations have moved away from traditional infrastructure models, on-premises data centers, or colocation in favor of cloud-based solutions. Though this infrastructure may not be on-premises, organizations are still responsible for the security of their cloud assets, providing additional attack surfaces by which attackers may access company resources. Cloud-specific testing models are often warranted for these environments. These tests target common configuration issues that can lead to authorization and privilege escalation vulnerabilities within cloud-based environments such as AWS or Azure.
In some cases, cloud penetration testing can be rolled into the scope of general network penetration tests. Various hybrid deployments such as Azure Active Directory (Entra ID) connected to internal AD infrastructure have become more common. Some organizations have moved entirely from on-premises AD to cloud-based Entra ID.  These cloud-based deployments can improve an organization’s security posture in certain ways but also present new security challenges and potential new avenues for exploitation. Cloud penetration tests may be warranted for these customers. These tests involve strategies similar to external network penetration tests for initial access but require different privilege escalation and lateral movement methodologies. 

Lastly, many organizations may not have extensive network infrastructure or proprietary applications at all. Still, they may use various organizational applications such as messaging apps, service ticketing systems such as Jira, knowledge base, or wiki-style applications like Confluence, etc., in addition to other cloud-based platforms or third-party identity providers (IdPs). These applications may contain sensitive, personally identifiable information (PII) or proprietary information and implement role-based access controls. Scorpion Labs offers a custom penetration test type called an organizational penetration test for such cases, which involves testing these typical organizational applications for authorization, privilege escalation, and sensitive information disclosure, as well as penetration testing of any cloud platforms from the perspective of a typical organization employee. 

Depending on the particular nature of the client environment, custom testing models can be devised to provide the greatest value for the customer’s security posture. These models allow clients with unique needs to address gaps in their security posture that would not be discovered or adequately understood using standard testing methodologies.

Red Teaming

In contrast to penetration testing, which focuses on discovering vulnerabilities within network environments or applications, red teaming is a form of testing that seeks to test an organization's overall security readiness and response against simulated threat actors. Factors tested encompass threat detection and response software tools, incident response procedures, human factors vulnerable to social engineering, exploitable technical vulnerabilities, and general readiness. Whereas a penetration test aims for broad vulnerability coverage, a red team's focus is more narrow. The goal of the red team is to emulate a real-world threat actor who wishes to gain entry to an organization's critical infrastructure through any means necessary without detection. Typically, only senior security staff are aware of the engagement and progress of testing, while defensive security teams—the "blue team" are unaware of testing activity. The purpose of this is, of course, to test the effectiveness and promptness of the blue team response as part of the comprehensive security posture of the organization.

In contrast to penetration tests, which may last several days to weeks, depending on scope, a red team engagement usually requires multiple weeks to, occasionally, months to be most effective in emulating real-world threat scenarios. This extended timeline is because planning, setup, and stealth are far more critical in this case, each requiring time and thought. Overall, red teaming is the most realistic threat emulation scenario of all the methods encompassed in that it most closely mimics the stealth and surprise advantages enjoyed by malicious adversaries. 

Social Engineering

In addition to general red teaming, there are more targeted approaches. A red team may be engaged to attempt access to a particular database or to gain access to a specific key employee account, for example. A social engineering engagement is a prevalent test that falls under this approach. This form of testing involves targeting some subset of employees, who, for obvious reasons, are not informed ahead of time, with various social engineering attacks. They may utilize email, voice, SMS, or in-person attempts to induce personnel to perform actions that may grant the tester unauthorized access to the organization’s networks, applications, data, physical locations, or actions that would violate authorization in other ways. This approach aims to gauge employee readiness to deal with common real-world social engineering attacks and test the effectiveness of technical controls and security policies in preventing a social engineer from gaining access to accounts, systems, or data. 

Phishing is a common technique employed in social engineering engagements and by malicious actors, and it has vastly increased in sophistication over time. Phishing involves sending emails to targeted users that contain a link to a page designed to mimic a login page and capture credentials or, in some cases, serve malware to users. Modern phishing techniques can utilize reverse-proxy servers to seamlessly capture credentials and session information, bypassing multifactor authentication (MFA), and often involve sophisticated infrastructure and techniques to bypass email filtering and other network and host security protections. SMS phishing, or smishing, is very similar to email phishing in setup and infrastructure, with the difference being that the vector by which malicious links are sent to targets is via their mobile device. This approach can have the advantage of bypassing a company’s security controls in many cases. Voice phishing, or vishing, involves talking to target users over the phone to gain sensitive information or access. Each method is powerful but can also be combined to gain further access to a company’s infrastructure or resources.

Social engineering testing provides clients insights into the most persistent difficulty associated with information security: the human element. Regular testing can provide trend data to determine the effectiveness of training, policy, and controls on the behavior of employees in response to threat scenarios. Additional insights beyond the human element can also be gained by such testing. Though the human element is challenging to secure persistently, testing can suggest strategies and technical controls that move specific sensitive tasks away from unpredictable human hands and into automated security processes that can be more tightly controlled. 

Purple Teaming

Purple teaming takes its name from combining the red and blue teams, and that is precisely what it does. Like red teaming, purple team engagements aim to emulate real-world threat actors. Unlike red teaming, however, blue teams are made aware of the engagement and actively participate alongside red teamers to test the effectiveness of security detection and response tooling and configurations. Both teams will collaborate in real-time to perform a variety of attack emulations, and blue teamers will check for detection and mitigation of attacks and, in some cases, tweak toolsets in response to failed detection attempts, retrying until detection or mitigation is achieved. 

Like a red team, a purple team will likely involve a longer timeline and the infrastructure needed for real-time collaboration and documentation that provide vital value. Platforms such as Vectr.io offer a centralized toolset for collaboration, a standard testing methodology, and a reporting system. Ultimately, a purple team provides clients with a deeper understanding of the combined effectiveness of technical controls, detection and response tooling, and configurations, as well as the ability for defensive teams to use them effectively—in short, the technical defensive capability of an organization to deal with standard attacks. Providing real-time feedback to defensive teams can allow tweaks to technical controls to be rapidly made and tested, improving overall defensive capability in a short time. In this way, it is superior to red teaming. However, it does not test the real-world response of the human elements within the defensive security posture, as the awareness of the attacks gives the blue team an advantage they would not have when facing a real threat actor. 

What Tests Are Best for Your Organization?

The answer to this question depends on several factors. Some of the most important are the nature of the environment, overall security program maturity, and compliance or other business requirements. In most cases, the most significant value will be achieved by some form of broad-scoped penetration testing. In organizations with a substantial external network footprint, external network penetration testing is essential to a comprehensive security program. White-box application penetration tests are likely to provide the most value in organizations that do not have a large external network footprint and expose only web applications. Internal network penetration testing will benefit organizations with significant internal network infrastructure, especially those employing Active Directory or on-premise network services. For any primarily cloud-based organization, an organizational or cloud penetration test will provide broad coverage of their exposed attack surface and net the most significant value.

Application penetration testing is warranted for organizations with proprietary applications, whether web-based, mobile, or even native applications that implement custom protocols, as well as companies that provide software as a service (SaaS). The particular focus of any application testing will depend upon the nature of the application. The technology stacks it implements, the application's use cases and business logic, and the source code employed in the application's logic. Whenever possible, we recommend source code-assisted testing to provide the most significant coverage of application logic within the shortest timeframe. 

Testing types such as red and purple teaming should generally be reserved for organizations with greater maturity in their security programs, whose security teams are large and well-funded enough to fully use the opportunities these tests provide to enhance security response. Once an organization has gone through several rounds of penetration testing and vulnerability remediation and has developed a comprehensive, well-functioning security program comprising such features as asset inventory, routine patch management, employee security education, credential management policies and procedures, and detection and incident response teams with effective tooling and procedures, a red or purple team test can provide great value in evaluating and tweaking this program for maximum effectiveness. 

On the other hand, targeted social engineering engagements can benefit any company with a significant number of employees, primarily if they’ve previously implemented employee security training, automated phishing tests, technical controls, and policies designed to prevent social engineering attacks from succeeding. This assessment can show an organization how well employees are prepared for social engineering attacks and how well particular training approaches and policies can mitigate them.

By implementing the proper combination of security testing approaches in the appropriate order, tailored to an organization's specific needs, the most significant value can be added to its security program and bottom line through savings. To best achieve this, knowledge of the particular environments to be tested, their attack surface, the threat models specific to the organization and its business model, and the present state of the organization's security posture is paramount. Armed with this knowledge, testers can develop a plan for testing that will be most appropriate for the organization, which will allow them to maximally improve security in the most salient areas without wasting any time or money spent on tests that are not appropriate for the environments or present security posture of the organization. By working closely with clients ahead of planned engagements, we can picture our clients' particular security needs, develop a testing program that will work best for them, and net the greatest return on investment of time and resources for all involved.