Companies open-sourcing their code often incur delays while they ‘clean up the code’ and the argument of ‘poor or not acceptable quality code’ is given as reason not to make it available to the public in the first place. Customers may want to ask themselves how they can trust products with hidden code given how frequently we hear about the low quality standards. Especially when it comes to security, code quality is a serious concern and a recent hack at Facebook shows why.
Last week, a security researcher disclosed how he hacked Facebook and found a running backdoor script from someone else. This event, and Facebook’s attempt to downplay its importance, illustrates some of the benefits of open-source and industry best practices around security as we employ them at ownCloud.
Motivation
Orange Tsai, an employee at the Taiwanese security firm Devcore, and previously known for a Remote Code Execution issue on uber.com, decided to spend some time trying to find a security issue at Facebook. He explained his motivation:
“With the growing popularity of Facebook around the world, I’ve always been interested in testing the security of Facebook. Luckily, in 2012, Facebook launched the Bug Bounty Program, which even motivated me to give it a shot.”
Besides showing the value of bug bounty programs again, it highlights what motivates many engineers: a challenge. The bigger and more impactful a technology, the more attention it gets. With more than 8 million users, ownCloud is a very significant project in the open-source community, and benefits from the attention that this generates.
Accellion’s Open Back Door into Facebook
Upon researching the networks related to Facebook, Orange found a mostly-hidden files.fb.com site running Accellion’s Secure File Transfer (FTA). After researching known exploits, he found this exploit disclosure from some months ago. While FTA Facebook used was no longer vulnerable to this particular issue, it gave Orange an important clue:
“from the fragments of source code mentioned in the Advisory, I felt that with such coding style there should still be security issues remained in FTA if I kept looking.”
His hunch turned out to be true. He quickly found 7 vulnerabilities which he then exploited to get into Facebook’s network. After getting in, he looked around – and found something strange: an error message in the Apache webserver logs.
Following the path, he discovered a script running on the server and gathering passwords, regularly retrieved from a third party server! Orange describes the actions of the other hacker quite well, and expressed some surprise that this was not discovered as he considered the setup he found pretty badly hidden.
Facebook’s Response
On Hackernews, Facebook responded publicly, thanking Orange for his work (he’ll be rewarded with a USD $10K bounty!) and claiming:
“After incident response, we determined that the activity Orange detected was in fact from another researcher who participates in our bounty program. Neither of them were able to compromise other parts of our infra-structure so, the way we see it, it’s a double win: two competent researchers assessed the system, one of them reported what he found to us and got a good bounty, none of them were able to escalate access.”
Now here, I have to be a bit cynical: if this ‘other researcher’ truly was looking to earn a bug bounty, he/she would not only have stepped forwards by now. More importantly, automatically gathering log-in data as described would not only violate the rules of the bug bounty program, it certainly goes further than a ‘white hat hacker’ is supposed to go. I can only guess that Facebook would rather not admit this…
How to do Better
A few interesting lessons come out of this disclosure.
First of all, bounty programs work – without it, the potentially malicious hacker would still be gathering passwords and the bugs in Accellion’s not-so-Secure File Transfer would not have been found.
Second, there’s clearly a code quality problem in a proprietary product widely advertised for its security. Anybody with industry experience knows the state the code base of many internal projects is. Just note how often code quality is mentioned when companies want to open-source one of their products. You’d do well to ask yourself how is it acceptable to sell something that the engineers themselves describe as ‘unacceptable’ just because the customer can’t actually see the code. It also makes clear that open-sourced companies are widely recognized as having a much higher standard of quality.
Why is this? Both for the pride of the company and its engineers, as well as more simple and practical considerations. Open communities like ownCloud have a large number of contributors joining and contributing code. Over 1000 people have contributed code to ownCloud, with between 80 and 90 doing so every month, about a quarter of whom are new! That’s right; we get to introduce two dozen new programmers to the ownCloud code base every month. You can imagine how your code base has to be clean, well laid out, well documented and well supplied with automated and manual tests for that to work out. This allows ownCloud’s new contributors to be able to get his or her code in within hours. Compare this to a typical proprietary project, where a new hire is expected to spend a few weeks getting up to speed. Our process and code base enables us to get 20+ volunteers to learn how to contribute to ownCloud each month!
Obviously, the many more eyes on ownCloud code, combined with the quality of the code base and the clear and transparent processes has its effect on security as well.
Does that mean our code is perfect? Not by any measure. We’re more than willing to admit that the code as written might not be much better than what gets produced at a proprietary company like Facebook or Accellion, despite our clear review process and pressure to document and test the code. But, unlike at a closed competitor, our code is under constant public scrutiny. Both software engineering, and especially security, is incredibly hard to do right. Thanks to our open process, we benefit from the collective insight of the smartest minds, and this input, review and oversight is how we continuously improve our code.
The fact that ownCloud is a big, open–source project, following security industry best practices and running a security bug bounty project quoted as “the benchmark for a well-executed, successful HackerOne launch,” gives our users and customers the confidence that, yes, they are running a secure product.
