Last week we wrote about the history and origins of Smashbox, a versatile testing tool for ownCloud developed by CERN. Today, we will discuss how Smashbox works, where it is helpful, how to develop tests, and how it will help ownCloud in the future.
How it Works
Smashbox essentially emulates multiple users doing a number of tasks. While it can be used to emulate larger numbers of users (for performance testing), it is at its best as a functional end to end testing tool, meaning that it is meant to extensively and rigorously test the functionality of ownCloud and detect where a change might have broken something.
As an example, consider the following scenario:
- User one shares a folder to user two with full access rights
- User two re-shares the folder with user three limiting access to read only
- Now user three should not be able to upload in the folder
There are many dozens of variations and more complex forms of the above. It is hard and laborious to test all of them by hand, but Smashbox automates this process and can thus make sure no change breaks any of its tested scenarios.
And Smashbox can do more than just API calls to ownCloud. One of the ways it can test is by initiating multiple sync clients doing parallel and do up- and downloads and then check if the files change and are modified correctly. This makes it a very versatile tool; useful in many scenarios.
Sharing and Smashbox
In general, testing file sharing in ownCloud is a big focus for Smashbox, and while steps have been made for ownCloud 8.1 to have very extensive testing in place, one can come up with an almost infinite number of challenging scenarios. One area which is begging for testing is the federated sharing API. The ownCloud QA team will get on that as soon as the final API is stable and published!
Smashbox for Testing Bug Fixes
Kuba pointed out that another area where Smashbox will be helpful, is bug reporting. It is often hard to reproduce bugs and users don’t always respond to requests for testing possible solutions. Developers then have to assume that the problem is solved based on their own testing or knowledge of the code. Smashbox, however, allows a developer to build a script which reproduces the bug so he/she can be confident that the bug has been fixed. Moreover, the script can then become part of the test suite, ensuring the problem doesn’t rear its ugly head again in the future.
For example, Smashbox has recently been helpful in tracking down a bug in the HTTP threading code in the Qt libraries (used by the desktop client) which could cause data loss in certain very specific circumstances.
Smashbox in Development
Smashbox is equally useful when developing new code. Upon re-factoring core infrastructure, Smashbox can test for regressions, as was its role when used to test the new high level file locking. It was also beneficial when ownCloud added end to end check summing (a client feature for now).
Smashbox and Load Testing
Smashbox can also be used to generate load on the system to test scalability or to trigger problems that only happen on a heavily loaded system. You can set up some background use, for example, you can have 200 users being active and then run the test cases and see if they still work.
Test Writing
Tests are written in Python, often by using the PyOC client library developed by Vincent Petry. Python takes care of constructing the API calls, saving the effort of manually having to write HTTP API requests.
Writing tests is lightly documented in the repository here, but the best way to write a new test would probably be to start with an existing one and building on it. Some efforts will be put in by explaining some of the existing tests better to help new test writers and the team currently working on the tests notes that Python knowledge shouldn’t be a barrier. One of our testers claims she didn’t know about it in January and was writing tests daily in March!
If you’re interested in increasing test coverage, there are some test cases in progress which are not yet developed – that might be an easy way in. Of course, if you have your own ideas, get going! Help is welcome in any way or shape.
The team is also working on merging the changes made back into the upstream project, tracking their progress in this github issue.
Future
Over the coming weeks and months, the ownCloud testing team will work on expanding the test coverage of ownCloud. An interesting side benefit of the work on Smashbox has been that it exposes functionality that is lacking in the public ownCloud API’s. Much of the functionality is in one of the multiple internal API’s used by components of ownCloud to talk to each other. To test this, an external API is needed. Smashbox not only forces the ownCloud developers to add any missing public API functionality, but to also think about what these API’s should look like and how they should relate with the internal API. It essentially provides a nice test case for each new API; providing feedback on its design and implementation.
Improvements to the ownCloud protocols and API’s are something Kuba also deeply cares about. He notes that at CERN “we are feet on the ground people” and that the usage of ownCloud and other IT services is exploding with new usage models, users, and growing expectations. The team is keen on exploring new ideas and possibilities but their foremost preoccupation is with a stable future for their services. He points out that Amazon S3 has been very successful, in no small part due to its successful ecosystem of solutions. This was made possible by a well-designed and documented protocol and he would like to see similarities between ownCloud and S3 in this regard.
Meanwhile, the CERN team is developing tests around protocol usage and implementation, which can help determine whether a problem is on the server or client side. Another new thing they are working on is ‚fault injection techniques,‘ where Smashbox will intentionally misbehave (within or outside of the defined protocol) in an attempt to uncover issues with the client.
As an ownCloud app developer, a core contributor, or even just a user who is interested in helping out with testing, you should consider having a good look at Smashbox and helping out with the expansion of test coverage. After all, while running a single test once is useful in finding bugs or verifying they are fixed, writing a test case and making it part of the Smashbox test suite provides test coverage of the functionality in the future. This makes the test also helpful for catching regressions and ensuring continued functionality. Additions to the ownCloud public API should especially be accompanied with an appropriate test.
You can find the Smashbox code in github and talk to the testing team on IRC at any point in the " target="_top" rel="noopener noreferrer">#owncloud-testing channel or on " target="_top" rel="noopener noreferrer">#owncloud-dev. People most involved with testing are @jennifer, @PVince81, @nickvergessen and @morrisjobke. And of course – join the ownCloud Contributor Conference next month in Berlin to discuss Smashbox with its developers and contributors in person!
Thanks to CERN and Jakub Moscicki for their time to answer our questions
