I’ve been around application penetration testing for many years now and I’ve seen and heard many things when it comes to scoping a job. This post is a bit of a <rant>brain dump</rant> around this subject because, well, it’s an interesting one and based on the feedback we get, we take a slightly different approach to most others out there.
I scope application penetration tests almost every day (I’m pleased to say!) and it’s something which in some ways is pretty difficult but in other ways is pretty simple. Like most testing companies we charge a day rate, which inevitably means for us to deliver a test for you we need to know how many days we need to do it.
But the million dollar question is: what is it? Your it may be different to someone else’s.
When we’re scoping a penetration test to cover a compliance requirement, our hands are tied to some extent. It’s got to cover all of the platform and it’s got to cover certain types of tests. Then it comes down to a blend of experience, gut feel and good old fashioned metrics like input parameters and number of dynamic pages. Technologies and frameworks used also plays a part.
Where compliance isn’t so much of a factor, there’s quite a bit of noise on this subject.
I’ve seen other companies talking about pricing per input, per port (for infrastructure testing), per test performed in some effort to convince buyers they are getting “total coverage” but I call BS. Big BS when it comes to applications because apps just don’t play by those rules. And WTF is total coverage anyway? Do you even need it?
Total coverage is a myth
Total coverage doesn’t exist. It is not possible to test all inputs to even a modest application for all possible vulnerabilities within any sensible budget. You could test an average sized app for three months straight and still not find everything that might be a security issue.
Remember Heartbleed? Yes, I know, let’s drag its cold, dead corpse out for another sacrificial kicking but stick with me. There was just over a two year window in which that vulnerability was widely exploitable but not discovered. How many penetration tests of vulnerable systems occurred in that time period and didn’t identify it? I’d wager it was in the thousands, maybe even millions worldwide.
Testing has limits
So don’t bother penetration testing? No, no. The point is, there is a limit to testing. An application could be tested and tested and tested but the subtle nuances of a specific security bug may not be unearthed. Things which seem trivial in hindsight can be almost impossible to find without almost divine inspiration in the moment.
If you were running a ten year old version of your operating system, web server and application in production today (you’re not are you?), would you think it is secure? Hell no, it is without doubt chock full of holes. Well the really bad news is, there’s all manner of software you’re running today which has holes in we just don’t know about yet. In ten years time we probably will.
Security testing may find some of these issues but it won’t find all. Most of these bugs are discovered by researchers spending focused time on that specific software. Or bug bounty hunters who are often prepared to sit for weeks looking at an app, at their own expense. Which really brings me to the point of this post, is that why you’re getting a security test done in the first place?
Far better to consider your own threat model. Who are you going to get attacked by? What is their motivation? How long are they willing to spend doing it? What will they gain if they’re successful? Are bugs which take weeks to find really the ones you need to worry about? If so, relying solely on a security test is probably still not the most efficient way to find them.
This is why we like the OWASP Application Security Verification Standard and align our penetration testing to it. There are levels of testing we can perform based on what level of threat you are trying to protect from.
You don’t (shouldn’t) get a penetration test done to see if you are secure. If you are relying on a penetration test for this I’m afraid you have possibly already failed. Penetration testing, despite what many will have you believe, is just like any other technical testing. Yes, it’s a specialist skill but so is performance testing or good user acceptance testing, for which there are third-party specialist companies who can help you do it right.
If I said to you, build me an app that can handle 200 concurrent connections, would you hack away at it writing code and then wait until it’s performance tested to see if you’d done ok? Of course not. You’d consider concurrency, non-blocking operations, indexing, etc, etc. It’s no different with security. If you don’t define security requirements early in the build process and consider them while producing code, don’t be surprised when a sh**load of issues come up when it’s security tested.
We are pragmatic, think long term and recognise (and explain) the shortcomings of security testing. When we deliver a test, whilst we are very good at what we do and make every effort to get good coverage and work through our methodology as fully as possible, the reality of testing is that, depending on what you find you will use up time and overall coverage can suffer.
Many a penetration tester has a tale of that time they spent half a day working around some filtering or other restriction either to find the issue was not exploitable, or that it was but that, either way, they’d burned half-a-day on one small element of the overall test. And if it was an issue, don’t forget that adds time for reporting too. More time out of the limited allocation. You start to see why full coverage is a myth? Should the tester have kept trying for half a day? Hell yeh, they should if they’re making progress and believe it may be possible. A penetration test is not a scan, it deals in confirmation. The whole point is that potential issues are taken to conclusion as much as possible. We seek to confirm exploitability where we can, not list a bunch of theoretical issues. With applications, it’s not always possible - there is definitely a difference and many testers hate apps simply because it’s not black and white enough for them.
It’s about the long term and the root cause
Getting back to the scoping theme of this post, we worry less and less about “under-scoping” a test because, as we explain to our clients (and potential clients), it depends what you’re looking for. We can’t - and no-one can - give you a guarantee an app is secure. We can work through our methodology and try all our tests, use our creativity and even run some scans which will give you good assurance, absolutely, but it doesn’t mean the app is “secure”.
We are, with great success, moving organisations away from seeing penetration testing as some kind of definitive thing. Instead see it as the start of a journey together. Not “is it secure” but “how resilient are we to the attacks we are likely to face?”. “Are we trending up or down in terms of our resilience?”.
We encourage our clients to be less interested in the individuals findings but more the analysis we provide for them. As an example, if you have built a Node.js app using Express, or a Rails app, .NET app or Spring Java app, to name just a few, all of these are robust™ frameworks which handle the vast majority of security issues out of the box. It is rare - though it certainly does happen - to find a basic injection flaw, or a cross-site scripting issue, or a CSRF in an app built on any of these if the development team has even half an idea about secure coding.
What do nearly all these frameworks not handle out of the box? Access Control. This is pretty specific to your app so it means your developers need to think about it and have security awareness. This is where any lack of secure coding knowledge starts to get found out and this is where a penetration tester starts to earn their keep. I lose count of the number of relatively solid apps we test where authorisation/access controls issues are found due to relatively simple errors.
…and the devs
So, do your developers actually know how to code securely or are they profiting from good framework choices?
When we’re performing a test, this is what we’re really looking at. I don’t actually care how many SQL injection issues you have in your application total. If you’re running a .NET app or a Rails app and you have SQL injection, I already suspect you have more because they just shouldn’t be there. It’s almost harder to code them in than it is to code them out!
If this is your situation, yes, it could be one mistake but you don’t need more testing to find more SQLi, you need to understand how that bug got in there and address any root cause, be it a knowledge gap or a simple mistake. It’s far more efficient to start looking through the source code to find more examples of the same issue than paying us to find them blindly. If we have access to your source code (which is the case in the majority of our tests these days) we will actually do this for you.
True story, we tested a Rails app with a surprising amount of XSS not too long ago.
git blame showed it was all down to one dev who had since left. The issues got addressed and some changes were made to the peer review process to prevent a reoccurence, all good. When we looked at the dev’s Github page we found he maintained an open source and well used Rails project and guess what - yep, same bug throughout. Misunderstanding of HAML templating options. We let him know of course.
In summary, when delivering an application penetration test for you, we will ideally look to perform an initial baseline test and feedback to you on root cause of issues we find. We don’t/can’t guarantee 100% coverage because that’s impossible and frankly, not needed. We’d rather deliver you a modest first test to assess your current level then follow up with shorter, more focused tests around areas identified as being weaker. We also provide training for developers to help them understand the risks they face out on the Interwebs.
If you’re looking for an application penetration testing company who use technical testers to scope for technical testing work and are focused on long term, sustainable security improvements rather than selling you a big test with diminishing return on investment (meow) then get in touch with us to discuss your needs.