A practical guide for organisations deciding when penetration testing is appropriate, what it should assess and how to distinguish it from vulnerability assessment and red teaming.
By Navid Sobbi, Founder and CEO, NSI Global
PUBLISHED: 30 April 2018 SUBSTANTIVELY UPDATED: 1 October 2026
A penetration test should answer a specific security question: can an authorised tester use real attack techniques to exploit weaknesses in a defined environment and demonstrate meaningful business impact? It is not simply a vulnerability scan with a more aggressive name, and it is not automatically a red-team exercise.
For Australian organisations, the need for disciplined testing is becoming more important as internet-facing systems, cloud services, identity platforms and remote access infrastructure expand the attack surface. The Australian Signals Directorate (ASD) also warns that artificial intelligence can help malicious actors identify and exploit vulnerabilities more quickly, particularly in internet-facing services. That makes assurance activity valuable only when it is properly scoped, technically competent and linked to remediation.
The goal is not to prove that a tester can “hack” something. The goal is to produce defensible evidence about which weaknesses can be exploited, what an attacker could achieve, which controls worked or failed and what should be fixed first.
What Is Penetration Testing?
ASD defines penetration testing as the assessment of organisational vulnerabilities through the design and execution of tests that demonstrate how an adversary could subvert security goals or achieve specific objectives. In practical terms, a penetration test combines technical analysis with controlled exploitation inside an agreed scope.
That distinction matters. A scanner can report that a vulnerable service appears to exist. A penetration tester may go further by determining whether the weakness is exploitable in the actual environment, whether it can be chained with other weaknesses, whether privileges can be increased and whether access to sensitive systems or information becomes possible.
Testing should always be authorised and governed by rules of engagement. The target systems, objectives, testing window, safety constraints, communications process and stop conditions should be agreed before active testing begins.
Penetration Testing vs Vulnerability Scanning
Vulnerability scanning, vulnerability assessment and penetration testing are related assurance activities, but they provide different types of evidence.
| Activity | Primary Purpose | What It Tells You |
| Vulnerability scanning | Broadly identifies known vulnerabilities, missing patches and configuration weaknesses, usually with significant automation. | Coverage and discovery. |
| Vulnerability assessment | Validates, analyses and prioritises identified weaknesses in the context of the environment. | Risk understanding and remediation prioritisation. |
| Penetration testing | Uses controlled manual and automated techniques to determine whether selected weaknesses can be exploited and what an attacker could achieve. | Exploitability, attack paths and demonstrated impact. |
| Red teaming | Runs an objective-driven exercise that tests people, processes and technology against a realistic adversarial scenario. | Prevention, detection, escalation, containment and response. |
CREST similarly distinguishes vulnerability assessment from penetration testing: penetration testing uses more intrusive, human-led techniques and considers context, including how weaknesses can be combined or trust relationships exploited. That is why a penetration test should not be replaced by an automated scan, and a vulnerability-management program should not be replaced by an occasional penetration test.
Penetration Testing vs Red Teaming
A traditional penetration test generally evaluates defined systems, applications or security controls. A red-team exercise is broader and objective-driven. It asks whether an organisation can prevent, detect and respond to a realistic attack scenario across people, processes and technology.
NSI Global makes the same distinction in its Adversary Emulation and Red Teaming services: penetration testing evaluates defined systems or controls, while red teaming tests whether the organisation can prevent, detect and respond to an objective-driven attack under agreed rules of engagement.
Physical access testing and social engineering can be appropriate in a separately authorised red-team or adversary-emulation exercise, but they should not be assumed to be part of a conventional cyber penetration test. Keeping those disciplines distinct improves safety, scope control and the usefulness of the final findings.
When Should an Organisation Conduct a Penetration Test?
The correct cadence depends on the system, threat environment, regulatory obligations and rate of change. ASD’s September 2026 Information Security Manual provides a useful Australian assurance benchmark: its current control states that vulnerability assessments and penetration tests should be conducted before deployment, before significant changes and at least every six months thereafter for systems operating within the ISM framework.
That six-month interval should not be treated as a universal legal requirement for every Australian business. The ISM is primarily written for government, critical infrastructure and larger organisations using ASD’s framework. For other organisations, penetration-testing frequency should be risk-based and matched to the importance and exposure of the systems being tested.
Testing should also be considered when a material change alters the attack surface, including:
- Before a new internet-facing application, portal or service is placed into production.
- After significant changes to network architecture, identity systems, authentication or remote-access infrastructure.
- During or after a major cloud migration or material change in cloud configuration.
- Following a merger or acquisition where networks, identities or applications are being integrated.
- After discovery of a serious vulnerability affecting important systems.
- After a cyber incident where the organisation needs assurance that remediation has closed the relevant attack paths.
- Before deployment of systems handling sensitive, regulated or mission-critical information.
- When contractual, regulatory or customer assurance requirements call for independent security testing.
Why Internet-Facing Systems Deserve Particular Attention
Internet-facing systems are continuously exposed to scanning and exploitation attempts. In July 2026, ASD warned that malicious actors may use AI to rapidly discover and exploit vulnerabilities, especially in internet-facing services such as websites. ASD also recommends using AI carefully, in secure and human-supervised ways, to assist penetration testing and vulnerability assessment.
This does not mean AI replaces skilled testing. Automated tools can increase speed and coverage, but context remains essential. A tester still needs to understand the application, business logic, authentication model, privilege boundaries, network relationships and the consequences of exploitation. The quality of the assessment depends on how well technical findings are interpreted in the environment being tested.
What Should a Penetration Test Assess?
The scope should follow the risk question. A test does not need to cover every system to be valuable, but it should be broad enough to answer the objective agreed with the organisation.
- External network exposure and internet-facing infrastructure.
- Internal network security and segmentation.
- Web applications, portals and APIs.
- Cloud services, identity platforms and access controls.
- Authentication, multi-factor authentication and privilege boundaries.
- Remote-access infrastructure and exposed management interfaces.
- Attack paths that allow one weakness to be combined with another.
- Selected mobile, IoT or specialist technology where included in scope.
- Industrial control systems and operational technology only under dedicated safety-aware scoping.
- AI and machine-learning systems where adversarial risks form part of the authorised assessment.
NSI Global’s broader Offensive Cyber Security services include penetration testing, AI and machine-learning security testing, ICS/OT security testing, cyber posture consulting and threat-informed adversarial assessments. The scope should be selected because it answers the client’s risk question, not because every available test is automatically required.
Operational Technology Requires a Different Testing Model
Penetration testing in industrial control systems and operational technology requires additional caution. Techniques that are acceptable in an enterprise IT environment may introduce unacceptable safety or availability risk in a live operational system.
For this reason, ICS and OT testing should be separately scoped with operational personnel. Depending on the environment, appropriate methods may include passive review, a staging or laboratory environment, configuration assessment, architecture review or carefully authorised active techniques with explicit stop conditions. Active testing of live operational systems should never be assumed.
What Must Be Agreed Before Testing Begins?
A professional penetration test starts with governance, not exploitation. The organisation and testing provider should define the engagement in enough detail that the tester knows what is authorised and the client understands what may occur.
- Objectives: define the security question the test is intended to answer.
- Systems in scope: identify authorised hosts, applications, accounts, cloud environments, locations or other assets.
- Systems out of scope: identify assets that must not be touched.
- Testing window: agree when active testing can occur and whether restrictions apply during production hours.
- Rules of engagement: define permitted techniques, credential use, exploitation limits and any prohibited actions.
- Safety constraints and stop conditions: identify circumstances requiring immediate suspension or escalation.
- Communication protocols: nominate technical and executive contacts and define how critical findings will be escalated.
- Sensitive data handling: determine how credentials, evidence, screenshots, logs and potentially sensitive client data will be protected.
- Reporting and retesting: agree how findings will be rated, what evidence will be supplied and whether remediation will be independently validated.
What a Penetration Test Should Not Be
Penetration testing is valuable because it provides evidence that other assurance activities may not provide. That value is lost when the engagement is treated as an uncontrolled attack or a compliance checkbox.
- It is not unauthorised hacking. Written authority and scope are fundamental.
- It is not the same as running a vulnerability scanner.
- It is not permission to attack every connected system.
- It is not a substitute for continuous vulnerability management, patching and monitoring.
- It is not automatically a test of the security operations team unless detection and response are explicitly in scope.
- It is not automatically a red-team exercise.
- It should not introduce unnecessary business, safety or evidentiary risk merely to make the test appear more realistic.
What Should the Final Report Contain?
A useful penetration-testing report should allow technical teams to reproduce and remediate findings while giving executives enough context to understand business impact and priorities. A list of scanner results is not sufficient.
- Executive summary and agreed scope.
- Methodology, constraints and assumptions.
- Verified vulnerabilities and affected assets.
- Evidence showing how exploitation occurred.
- Attack paths and privilege escalation where demonstrated.
- Technical and business impact.
- Severity or risk rating with clear rationale.
- Prioritised remediation actions.
- Material limitations or areas that could not be tested.
- Retest or validation recommendations.
ASD’s current security-assurance guidance also emphasises the need for security assessment reporting so system owners can undertake remediation and develop an action plan. A penetration test should therefore end with decisions and follow-up, not with a PDF that is filed away.
How to Choose a Penetration Testing Provider
The provider should be able to explain how testing is governed, how evidence is handled, how risk is controlled and how findings are validated. Technical credentials matter, but buyer assurance also depends on process, independence, reporting quality and the provider’s ability to understand the systems being tested.
CREST guidance for buyers similarly emphasises appropriate supplier selection, governance, testing constraints, effective test management and follow-up remediation. For high-risk or regulated environments, organisations should also consider whether independent accreditation, professional certification, security clearances or sector-specific experience are relevant to the engagement.
Frequently Asked Questions
How Often Should Penetration Testing Be Conducted?
ASD’s September 2026 ISM states that vulnerability assessments and penetration tests should occur before deployment, before significant changes and at least every six months thereafter for systems operating within that framework. Other organisations should set frequency according to risk, system importance, exposure, rate of change, customer requirements and applicable regulation or contracts.
Is Penetration Testing the Same as Vulnerability Scanning?
No. Vulnerability scanning primarily identifies known weaknesses, often through automation. Penetration testing uses controlled exploitation and human analysis to determine whether selected weaknesses can actually be used to achieve meaningful access or impact in the tested environment.
Is Penetration Testing the Same as Red Teaming?
No. A penetration test normally evaluates defined systems or controls. A red-team exercise is objective-driven and evaluates whether people, processes and technology can prevent, detect and respond to a realistic attack scenario within agreed rules of engagement.
Can a Penetration Test Disrupt Production Systems?
Active security testing can create risk if poorly scoped or executed. That is why rules of engagement, safety constraints, backups, communication channels, stop conditions and the treatment of critical systems should be agreed before testing. High-risk OT and ICS environments require additional safeguards.
Does Passing a Penetration Test Mean the Organisation Is Secure?
No. A penetration test provides assurance about a defined scope at a point in time. Systems change, new vulnerabilities emerge and attacker techniques evolve. Penetration testing should therefore sit within a broader security program that includes vulnerability management, secure configuration, logging, monitoring, patching, identity security, incident response and periodic reassessment.
Can AI Be Used in Penetration Testing?
Yes, with appropriate controls. ASD’s 2026 guidance supports careful, secure and human-supervised use of AI to assist vulnerability assessment and penetration testing. AI can increase speed and coverage, but it does not remove the need for authorised scope, expert judgment, validation and safe handling of client information.
Penetration Testing Should Produce Actionable Assurance
The value of penetration testing lies in demonstrating what is actually possible inside an agreed scope. The most useful engagements connect technical findings to business risk, distinguish theoretical exposure from verified exploitability and provide a clear path to remediation and retesting.
For organisations with mature cyber programs, penetration testing can validate important controls before deployment or after major changes. For organisations still building foundational security, it can reveal which weaknesses create the most meaningful attack paths. In both cases, the objective should be the same: evidence that helps decision-makers reduce risk before a malicious actor tests the environment without permission.
NSI Global provides authorised offensive cyber security assessments for corporate and government environments. Organisations seeking to define an appropriate test scope can review NSI Global’s Offensive Cyber Security capabilities or contact NSI Global for a confidential discussion about objectives, systems in scope, rules of engagement, safety constraints and reporting requirements.
Sources and Further Reading
- Australian Signals Directorate – Guidelines for Security Assurance (September 2026)
- Australian Signals Directorate – Guidelines for Cyber Security Documentation (September 2026)
- Australian Signals Directorate – Information Security Manual (September 2026)
- Australian Signals Directorate – Defending Against AI-Enabled Cyber Attacks (2026)
- Australian Signals Directorate – Cyber Skills Framework: Penetration Testing
- CREST – A Cyber Buyer’s Guide to Vulnerability Assessment and Penetration Testing
- CREST – Guide to Penetration Testing
- NSI Global – Offensive Cyber Security
- NSI Global – Adversary Emulation and Red Teaming
