The teflon crew at Pied Piper suffered quite a bit during Season 2 of SILICON VALLEY. But there was no greater indignity than being brought to their knees by a tequila bottle.
Since episode eight “White Hat/Black Hat” aired, many skeptical viewers have asked: how could something like this happen?
Could a mindless error of pressing a delete key really cause a venerable company like Intersite to lose over nine thousand hours of content (including an irreplaceable archive of
vintage yiffing videos)?
The short answer: yes.
When the producers of SILICON VALLEY reached out to me during the writing of the episode to help design the sequence, I knew it would be tough to make the technical intricacies track for the joke. But if the team had figured out
optimal tip-to-tip of efficiency with Stanford PhDs, I was up for the challenge.
The delete key hack is a perfect storm of bad system hygiene. The dozens of small errors that led to it are more common than most systems administrators care to admit.
There’s a lesson here. Away from the big headlines about hacks at Sony Pictures and Target, companies every day have their systems broken by stupid errors. Most as avoidable as not putting a tequila bottle on a laptop (or letting Russ Hanneman inside your home in the first place).
Below is the the post-mortem on how the hack went down and tres lessons we can learn from it.
The Bake-Off
Intersite set up the bake-off between Endframe and Pied Piper by providing access to an FTP server with the target files. Each company then downloads the individual files, encodes, compresses and roundtrips the file back to the Intersite server.
To speed up the process of the compression, the companies sequentially encode each file to save hard drive space and time waiting for them all to copy down, then they perform the operation.
This is where all the problems originate: to deliver the compressed file, Pied Piper was given full permission to Intersite’s FTP server (because of course it’s easier to just give all permissions than manage every folder’s permissions).
For those interested, the final Pied Piper solution for Intersite would look something like this:
The Hack
When Russ sets down the Tres Comas tequila bottle on Richard’s laptop he unwittingly initiates a massive delete sequence on Intersite’s server. Here are the steps:
- Richard navigates to bake-off directory or with the extended permissions he got into a parent folder with original content.
- Gilfoyle turns off the delete verification prompts in the custom software they used for the file transfer and conversion. Richard meanwhile has enabled CRC checking on the internet system, a protection measure for penetration into Pied Piper that backfires.
- Russ Hanneman arrives with the Tres Comas Tequila and mistakenly sets the bottle down on the keyboard initiating a delete command. Chaos ensues.
- As the delete sequence launches the large file size of video spins the disks at 100% and locks the system.
- Hands-on access with the bottle with the persistent delete requests plus the CRC checking creates fork bomb-like effects where the Pied Piper team cannot get back into the system to stop the delete sequence. Intersite is compromised and thousands of hours of video are deleted.
Tres Issues
A myriad of problems contributed to the delete key hack. Using the OWASP framework here are the top tres issues:
A7 - Missing Function Level Access Control “PITA Administration”
Intersite’s FTP server is set up to allow the bake-off users full permissions to the digital masters being used in the bake-off.
But why were they unable to kill the transfer? There are a bunch of reasons this could happen -- the SAN of which they were FTP’ed into was doing massive amounts of data deletion which can be HDD intensive (try deleting over 9000 files and watch your computer crawl).
But they would still be able to kill it on the Pied Piper end?! Not if the transfer agent was queuing up deletes as fast as it could, pegging out the server on the Pied Piper end as well. Remember, everything they had in their garage data center was going into making the conversions go as fast as possible, the transfer back was a lower priority.
A4 - Insecure Direct Object Reference “speed is everything”
By not locking out his screen and letting Russ Hanneman near his unsecured session where he was monitoring the transfer from, Richard allowed direct access to the “delete” key object.
But why didn’t it prompt for approving the delete?
Basically when you’re in a bake-off like this, you do absolutely everything to remove any possible obstacles, like ANY prompts or approvals that you might not be around to hit “OK” on.
A6 - Sensitive Data Exposure “We must run forward simply to stay in place”
Intersite did not do a proper backup of their files due to cost and size constraints on their current system. In Episode 7 “Adult Content,” Molly Kendall, Intersite’s CEO, talked about how the porn industry was barely making the bills due to “free” internet porn influx. Intersite was doing its best to stay afloat in its industry and, as such, cut corners not only in the lack of proper admin for their FTP server, but also in storage costs. A mistake they will probably not make again soon.
Tres Lessons
Don’t be an idiot with permissions:
The easiest way in is through the front door. Ensure that everyone is on a need-to-know basis with data. Standard protections like 2-factor authentication and whitelisting IP addresses mean nothing if the wrong people access sensitive information.
Back up your data:
It’s easy to take for granted but reliable storage is still expensive. Most small businesses don’t backup data on a daily basis. Even fewer do so with any form of redundancy or integrity checks.
Don’t work with assholes:
Insider attacks are the hardest to detect and protect against. Surveys estimate 59% go unnoticed until it is too late. The first line of defense is common sense, only hire people you trust — and definitely not anyone who put radio on the Internet.
Read More
Posts
