The Comedy and Tragedy of Automated Security Source Code Analysis - Act III

ACT III: Reality Sinks In

(Read Act IAct II)

THE SOUND of typing can be heard and then suddenly a loud groan of disappointment.

DEV LEAD

Argggggggggggggggg!!!

FADE IN:

INT. CORPORATE OFFICE

THE DEV LEAD is looking at his bug queue in HP Quality Center, the bug count is 6,894 defects, all assigned to him. The DEV LEAD yells over the cube wall to one of his DEVELOPERS.

DEV LEAD

Hey you screwed up the bug triage again man, I’ve got 6,833 bugs that shouldn’t be there in my queue, what the heck man?

DEVELOPER

I didn’t do a thing! We did the triage just like you said and moved all those bugs opened by marketing to the “blackhole” category you set up, you should only see 61 bugs in there.

DEV LEAD

Yes, but why then do I have 6,894 bugs?

THE DEVELOPER GETS UP and walks over to the DEV LEADS desk and looks over his shoulder.

DEVELOPER

Whoa, wait a min, these are all security bugs, they must have gotten loaded into the system with the nightly build, didn’t you check in a lot of changes yesterday? Sounds like you need to go back to security training (smirk)!

THE DEV LEAD sighs as he turns to his computer and starts quickly clicking from bug to bug triaging the thousands of bugs in his queue, it’s going to be a long day he says to himself.

THE CAMERA PANS from the DEV LEAD to a wall clock as we watch the hands move and 5 hours passes in accelerated time

FADE OUT:

INT. CORPORATE OFFICE

THE DEV LEAD has just completed his bug triage, he grumbles as he sits in front of his computer shaking his head.

DEV LEAD (Voice Over)

12 bugs, out of thousands only 12 were real, Jesus Christ I thought that SALES ENGINEER had properly configured this thing? Everything was working fine during the evaluation, what’s going on here?

AS THE DEV LEAD is looking at the source code analyzer configuration files A DEVELOPER WALKS over to the DEV LEAD’S desk.

DEVELOPER

Hey man, I’ve been sending you emails all morning, did you have a chance to review my design yet?

THE DEV LEAD starts speaking with THE DEVELOPER when the DEV MANAGER walks up to the DEV LEADS desk and interrupts the conversation and addresses the DEV LEAD --

DEV MANAGER

Did you see your security bug count this morning?! I just had to fight off our CSO who saw some security dashboard that I didn’t even know about and is suddenly very concerned about the high security defect count in the latest build, he’s worried we are not on board with the new corporate security mandate, please tell me you have a handle on this OK?

THE DEV LEAD shoots a quick “leave now why you still can” glance at the DEVELOPER who is still standing there but quickly gets the hint and runs off. The DEV LEAD now turns to the DEV MANAGER --

DEV LEAD

Sir, I do have an idea what’s going on and unfortunately I had to waste the last 5 hours just to figure it out. Seems our new source code analysis tool needs to be reconfigured to handle our latest build, it flagged over 6000 bugs last night, almost all of them false positives because it didn’t like my new external input filtering code. I’m going to have to spend the rest of the day getting it figured out but I should have it running right before tonights build.

DEV MANAGER

Good, keep me informed if anything should change, I’ll let the CSO know we are on top of it and that it will be addressed in tomorrows report.

THE DEV LEAD sighs, he doesn’t look happy. Her nervously glances at his watch and then turns to his computer and begins typing feverishly.

FADE TO:

INT. CORPORATE OFFICE AT NIGHT

THE DEV LEAD looks at the clock, it reads 11:32 PM.

DEV LEAD (Voice over)

OK That should just about do it for the security scan configuration, shouldn’t have any problems with my code after tonight. Time to get out of here, the wife is going to kill me.

THE DEV LEAD shuts off his monitor and heads for the door.

CUT TO:

INT. SERVER ROOM AT NIGHT

THE CLOCK on the wall ticks over to 12:00 AM as the camera pans to the right as the lights of a rack of servers come alive. The nightly build process has started and so has the automated security source code scan. It’s quiet...too quiet.

FADE OUT:

THE SOUND of keystrokes and multiple groans of surprise rising up from the cubes as the developers arrive and log into their workstations.

FADE IN:

INT. CORPORATE OFFICE IN THE MORNING

SLIVERS OF SUNLIGHT from the morning sun cut through the darkness of the office. The overhead lights all having had their florescent tubes removed long ago by the developers seeking darkness. Every month maintenance comes by in the night and fixes the overhead lights and the very next day the developers remove them again.

THE DEVELOPERS are just starting to filter in and sit at their cubes, many with cups of coffee or cans of Red Bull. One of the first DEVELOPERS to sit down loads up his bug queue and immediately lets out a small high pitched shriek --

DEVELOPER 1

HEY BOSS, why are there 435 bugs in my queue?

DEVELOPER 2

435? I’ve got 677!

DEVELOPER 3

Hey where did all these bugs come from??? I’ve got 8,456!

THE DEV LEAD walks into the office, later than usual, dark circles are under his eyes and he is immediately beset upon by his team wanting to know about all of these new bugs.

DEVELOPER 1

Hey Boss, something didn’t work right with that nightly build, my queue has more bugs than lines of code written by the new guy!

DEVELOPER 2

He isn’t the only one Boss, what’s going on?

DEV LEAD

Guys, a moment please, let me log on first and see what’s going on.

DEV LEAD (voice over)

Oh god, please don’t let it be my script, what time did I roll out of here last night? Crap there goes productivity for the day.

THE DEV LEAD is thinking he must have screwed up his script. He quickly logs in and checks his bug count. Less than a hundred, pretty much where he left it yesterday, he wonders what is going on?

DEV LEAD

Guys, are those bugs security bugs?

DEVELOPER 1 and 2 (together)

Yeah!

DEVELOPER 3

Looks like it boss

DEV LEAD

Look, just filter those out for now and ignore them for the moment, I’ll take care of it, everyone get back to work, remember we have out first milestone before the end of the month!

THE DEV LEAD is looking at a complete list of all the security bugs opened last night against the list of changes checked in by developer. Looks like everyone who checked in changes got dumped on by the security scanner just like he did the night before. Oh man, this is going to be another long day he thinks to himself. Little does he know it’s about to get worse.

THE DEV LEAD turns around to get coffee and is greeted by the DEV MANAGER who does not look happy.

DEV MANAGER

Buddy! You told me this was going to get taken care of! I just got chewed out by the CSO and CIO, the security defect count almost doubled from yesterday!

THE DEV LEAD starts to try and explain as we FADE OUT

- 6 MONTHS LATER -

FADE IN:

INT. DEV LEAD DESK

WE FOCUS IN ON THE DEV LEAD who is reviewing the bug counts from last nights security scan at his desk, 3 issues found, nothing serious, looks good. It took a while but after a lot of tuning and configuration and a few visits from that SALES ENGINEER they were able to get things back on track. 5 days ago the DEV TEAM released their final build for testing before release and the DEV LEAD is feeling pretty good.

CUT TO:

INT. CORPORATE OFFICE CUBE FARM

THE DEV MANAGER comes running in to the office, in his hand is a printed report, the look on his face doesn’t look good. He walks up to the DEV LEAD’s desk.

DEV MANAGER

We have a problem my friend.

THE DEV LEAD looks confused

DEV LEAD

A Problem?

THE DEV MANAGER drops the report he is carrying on the DEV LEAD’s desk. We can see that it is a pen testing report from the security team and it doesn’t look good.

DEV MANAGER

We failed the pen test! How did this happen? We have been testing our software for months, running that damn source code analyzer, you told me you had worked out the kinks! You told me you were fixing the bugs! What on earth is going on back here?!

DEV LEAD

We ran the scans every night! We fixed every bug!

DEV MANAGER

So why does this report look like a christmas tree then! Look at it! They drove a mac truck through the site, SQL Injection and XSS EVERYWHERE! How the heck didn't you catch that stuff?

DEV LEAD

Well we might have had to make a few tweaks to get the false positive count down...

DEV MANAGER

A FEW TWEAKS? It sounds like to me you turned the damned thing off!!!

DEV LEAD

We never did that...except that one time when the build was broken, but its fine now, look at all the incorrect equals operator issues it found and we fixed!

DEV MANAGER

Incorrect equals...what? WHO THE FRACK CARES! MAC TRUCK MAN, THE DROVE IT RIGHT THROUGH OUR HOUSE! We failed their pen test AGAIN! We are going to ship late...AGAIN! If you need me, i’ll be in my office, CRYING! Have you seen where I hid that bottle of scotch?

THE CAMERA PANS over the DEV MANAGERS face who stares blankly at the ground with the look of helplessness as he walks away towards his office. THE CAMERA focuses on the door to the DEV MANAGERS office as we watch him walk slowly into frame and into his office, he doesn’t look back as he closes the door behind him.

DEV MANAGER (Voice over)

Well there goes my bonus for the year, might even be time for me to work on my resume.

WE PAN OVER the CUBE FARM, developers everywhere are working hard at their desks. We see the DEV LEAD sitting at his desk, reading the pen test report and shaking his head.

DEV LEAD (Voice over)

Well that didn't go well. I really hate those security guys, there has to be a better way.

FADE OUT:

END ACT III - TO BE CONTINUED