The Comedy & Tragedy of Automated Security Source Code Analysis - Act 1

The other day I was asked about how one might integrate Veracode's application security analysis solution into their nightly software build process. I get this question all the time because it is a popular idea among automated source code analysis tools and the common sense tells you that it's one of the best ways to go, but is it?

When I was responsible for product strategy at SPI Dynamics and later for the HP Application Security Center, we had a product called DevInspect that was designed to bring security testing to the developers desktop. This product was an experiment and a incubator for new ideas and technologies like Hybrid Analysis (A concept I fleshed out in 2004, but didn't invent, that honor goes to Microsoft Researcher Thomas Ball in I think 1999).

Our idea was that if we gave the developers the tools to do security testing from their IDE, the world would be a better place. But we were wrong. DevInspect didn't work out, getting developers to run tools or do things in addition to their day job was a non-starter, they just didn't have the time and running the tool required security knowledge to get good results, another non-starter. So we turned our attention to a new project called BuildInspect, this project was killed shortly before we decided to purse the Fortify acquisition and I left HP for other adventures. The thinking of course at that time was if it didn't work on the developers desktop, it has got to work at the build server right? It wasn't hard to come to this conclusion, almost all of our customers were telling us this is where they wanted to do the testing and we were listening.

But was the customer always right? In this case I don't think so, but it's not their fault, they were operating on a flawed assumption. That assumption was that on premise application scanning could truly be automated and not impact a developers tight schedule.

Fast forward to today, where I'm at Veracode now whose approach to application security is both familiar yet totally different at the same time. I'm not here to talk about Veracode other than to say Veracode is an automated SaaS solution for software security analysis that uses binary analysis among other methods and not a tool you install on your desktop. Veracode is something you could integrate with your build process if you really wanted to (there is an API for that) but the assessments can take 48 hours or more sometimes and that doesn't fit well with the nightly model. Now, it's valid to think I'm looking for a way to turn a perceived weakness into a strength but lets imagine for a moment that Veracode can perform assessments fast enough to be run with the nightly build (which isn't too far away actually, the other day we completed one assessment in under 15 minutes).

So the question is, putting aside what seems like common sense for a second, if you could integrate security source code analysis into your nightly build, would it be the right thing to do? To answer that question I started writing a long email. I hate long emails, but when you are trying to go against conventional wisdom you have to spend a lot of time covering all the bases, putting everything into context and then going after the flawed assumptions that are the source of the mess. I decided I needed another approach.

So I did what anyone would have done, I decided to write a screenplay. There are 4 acts, I'll be sharing all 4, here is act I for your enjoyment!

The Comedy & Tragedy of Automated Security Source Code Analysis

Written by

Erik Peterson

ACT I: Courtship

THE SOUND OF TYPING, almost like jungle drums beating from a distance, growing louder.



WE PAN Over a developer cube farm with a few build servers humming in the background, the developers are hard at work. The head developer manager walks from cube to cube overseeing the work. As we follow him we peer over his shoulder and see he is reading a security team pen testing report, the word FAIL is written in large red letters at the top of the report.

DEV MANAGER (Voice Over) These security team bastards are killing me, every day they are complaining about something and it’s ruining my ship schedule and my bonus. This stuff is all nonsense, how is cross site scripting even my problem? Enough already, this time I'm going to do something about it and put these guys in their place. All I need is something to just review my source code and problem solved right?


THE WHIRR OF A Projector as a security source code analysis sales person drones on about integrating security into the software development lifecycle and the importance of testing as early as possible for security defects.

The DEV MANAGER looks impatient and suddenly interrupts the sales person --

DEV MANAGER Look, that all sounds nice but all I want to know is there some automated way to find these security "defects" as you call them and help me fix them before the security team (waving to two men and a woman sitting quietly in the back of the room) gets their hands on my applications?

The SALES PERSON senses they have discovered the customers pain and can see the deal closing before their eyes, tries to control his excitement --

SALES PERSON Absolutely! We can provide a completely automated way to find all of your security defects quickly and efficiently and can integrate with all aspects of your development process!

The SALES PERSON turns to the SALES ENGINEER covertly with a questioning look and whispers, only to SALES ENGINEER --

SALES PERSON We can do that right?

The SALES ENGINEER Nods and rolls his eyes, he wonders if the sales person even knows the name of the product they are selling.

The SALES PERSON quickly turns back to the customers, he wonders why he even brings the SALES ENGINEER with him, he could do this job in his sleep, he bets the SALES ENGINEER doesn't even think he knows the name of the product --

SALES PERSON Our SourceSecure 2.0 product provides unparalleled ...

SALES ENGINEER (Interrupting) 4.0!

The SALES PERSON ignores the SALES ENGINEER and continues talking --

SALES PERSON ...provides unparalleled ability to automatically find security defects and provide your teams the information they need to quickly remediate them.

The DEV MANAGER is thinking to themselves now, the gears are turning, maybe he has found a way to find these security defects just like his compiler finds coding defects before they push to QA --

DEV MANAGER (V.O.) Hmmm, I have all my developers checking in code for nightly builds, wouldn't it be nice if I could just scan that code for security defects every night at the same time too and create security bugs for my developers to fix every day?

The DEV MANAGER looks at the SALES PERSON and asks --

DEV MANAGER What about nightly build integration? I don't want to have to force everyone on my team to install and run some tool, we just don't have time for that.

The SALES ENGINEER opens his mouth preparing to talk --

The SALES PERSON sees that the SALES ENGINEER is about to complicate his deal and start talking, likely for hours, about all sorts of technical details that are just going to to ruin his quarter, he has to act fast to contain this --

SALES PERSON Of course we can!

The SALES ENGINEER sighs loudly.

The DEV MANAGER has heard all that he needed to hear, if all he has to do is hook this thing into his build server and just have his developers fix the bugs that it identifies, he shouldn’t have anything to loose right? Why the heck is the security team even around if he can just automate all of this stuff anyway? Why didn’t anyone mention this to him earlier? He turns to the SALES PERSON --

DEV MANAGER Perfect! Finally someone around here who can help me automate this problem. Lets do an evaluation and see if this thing is as good as you say it is.

The SALES PERSON smiles to himself, his work is done, that SALES ENGINEER better work his magic and not screw this up, he turns to the DEV MANAGER --

SALES PERSON Excellent, we can start right away, I have some simple evaluation paperwork to sign, can you introduce me to your boss so we can go discuss it? Maybe we can all go play some golf tomorrow? My Treat! Lets leave my engineer to work out the details with one of your dev leads and your security team.

THE CAMERA PANS OVER the room as SALES PERSON and DEV MANAGER leave the conference room and the engineers behind who start to sketch out a plan on the whiteboard.