ACT II: The Deception
FADE IN:
WIDE SHOT — The server room, filled with blinking lights and racks of servers. We PAN and FOCUS in on the back of the SALES ENGINEER who is typing away at a terminal. A DEV LEAD sits nearby, looking bored.
SALES ENGINEER’S POV — The computer screen shows lines of code, the SALES ENGINEER is writing scripts to integrate SourceSecure 4.0 into the companies source code repository.
INT. SERVER ROOM
THE SERVER ROOM is a more of a lab than the real server room, used only for development it contains the central build servers used by the entire dev team. A small desk with two monitors and two chairs with their backs to the room look out a large glass window that overlooks the developers in their cubicles.
The SALES ENGINEER is sitting at one of the terminals, he turns to the DEV LEAD who is standing next to him –
SALES ENGINEER
Ok, that should take care of it, just by running this script SourceSecure will access your source code repository, check out the latest version and load it for automated analysis. It should take a few hours for that to work so why don’t we start it and go get some lunch?
THE DEV LEAD looks at his watch and then back at the SALES ENGINEER –
DEV LEAD
I thought we were integrating this into the nightly build process?
SALES ENGINEER
Oh yes, we are, everything I’ve done here will make that possible. Once we have completed this evaluation and assuming you are happy with the results we can easily wire this script into your nightly build and automate the entire process
LOOKING OUT of the windows A NERF DART flies over the cubes and strikes the glass, right next to the DEV LEADS FACE, it sticks cleanly to the glass, a perfect shot. The DEV LEAD throws a harsh stare at the shooter and mouths “back to work” before turning his attention back to the SALES ENGINEER who is trying not to grin –
DEV LEAD
So does this thing produce a report or something, where do all the bugs go?
SALES ENGINEER
Great question! We can hook this up to your defect tracking system as well so instead of a report, each developer can be automatically be assigned any security defects we find.
DEV LEAD
Perfect, when we get back, lets hook it up to HP Quality Center. Ok how does Mexican sound?
CUT TO:
EXT. OUTSIDE OF OFFICE BUILDING
THE OFFICE BUILDING is a 5 story nondescript brick building with large glass windows. All of the blinds are closed to keep the sun out. The parking lot is filled with old sedans and a few pickup trucks. There is one red Porsche, parked and looking out of place near the entrance. The SALES ENGINEER and the DEV LEAD walk across the parking lot towards the SALES ENGINEERS car which as they get closer you realize is the red Porsche. They quickly hop in and drive off towards the restaurants down the street.
FADE TO:
INT. SERVER ROOM
THE SOUND OF SERVERS humming in the background, THE CAMERA PANS and comes to rest on a lone terminal, the same one the SALES ENGINEER had been using, on the screen a progress meter shows a percentage counting up…28…29..30% and another figure, “defect count” is counting up as well…356…589…1345…
FADE TO:
EXT. OUTSIDE OF OFFICE BUILDING
The SALES ENGINEER and DEV LEAD return in their car from lunch. The SALES ENGINEER quickly parks the car near the entrance and they both hop out and walk quickly to the door of the office building.
CUT TO:
INT. SERVER ROOM
THE TERMINAL shows that the source code analysis has completed, the SALES ENGINEER is first to sit down and is quick to clear the screen with a few quick keystrokes that the DEV LEAD doesn’t quite catch. The DEV LEAD turns to the SALES ENGINEER –
DEV LEAD
Did it finish? Can’t wait to see the results, what do we have?
SALES ENGINEER
It looks like it’s all done, tell you what, why don’t I get the data ready for a report and then we can take a look. Can you give me a few min, just a few things to work out since this is the first scan and oh, didn’t you want that Quality Center Integration? Can you get me the details for your QC setup so I can configure it?
DEV LEAD
OK, sure, sounds good, let me go check some email real quick and then I’ll get you that QC Info.
SALES ENGINEER
Great, I’ll be right here when you are ready.
The DEV LEAD leaves the room, but even before the door has closed the SALES ENGINEER is looking at the results, 5439 security defects found. The SALES ENGINEER sighs and looks nervously around the room as he gets to work.
SALES ENGINEER (Thinking to himself)
ZOMG, 5,439 issues? Again? This app has less than 50,000 lines of code! No way those are all real, this thing is filled with false positives. Just another scan for me to clean up and make pretty before I can hand it off and I better tune this engine otherwise the next scan is going to be a mess too.
THE SALES ENGINEER starts to filter out false positives and tune the engine to not flag them again in the future, he has to work quick so he can clean this up before the DEV LEAD sees just how much work he has to do. As we fade out we see the SALES ENGINEER typing feverishly.
END ACT II – CONTINUED IN ACT III
No, build server technology based on static analysis like this is always going to get 99.94+% false positives. You can’t “tune away” the false positives.
Once the developers start to realize how much work this is going to add for them, they will start to “work around” your little build server security integration project. They know how static analysis works and what to do if they want it to come up with 99.95+% false negatives.
Plus, what’s the cost of this? Quality Center? This sort of integration? The costs are probably larger than ANY appdev project or series of projects is willing to cope with. The developers wouldn’t even be in the United States in this example! Good luck selling and shipping security products overseas.
This is a great story — but that’s all it is and ever will be: a story.
I have a million better ideas than where this is going…
You are giving away my ending
I’m going to have to get more creative. Share some of those ideas!
I love stories with unhappy endings! Can’t wait to hear about it. Make it nastier and grittier than I even did.
Can you do a horror next?
Awesome. Not sure how I missed the first post in this series. I worked in a Falsify shop for a couple years. Back of the envelope calcs put our fp rate above 90% of all issues… to say nothing of false negatives.
Story raises some real concerns. But you can tune away false positives (we do this all the time) but you can’t tune away ALL false positives. Also it depends on what technology you are using – some are better than others at this problem.