{"id":7752,"date":"2015-01-21T09:30:56","date_gmt":"2015-01-21T09:30:56","guid":{"rendered":"https:\/\/owncloud.com\/?p=7752"},"modified":"2015-01-21T09:30:56","modified_gmt":"2015-01-21T09:30:56","slug":"blackfire-io-and-recent-performance-optimizations-in-owncloud","status":"publish","type":"post","link":"https:\/\/owncloud.com\/de\/blogs\/blackfire-io-and-recent-performance-optimizations-in-owncloud\/","title":{"rendered":"Blackfire and recent performance optimizations in ownCloud"},"content":{"rendered":"

\"The<\/a>

The graph that started it<\/p><\/div>
\nA little under two weeks ago, a performance improvement in ownCloud<\/a> was accompanied by a graph<\/a> showing the difference in code execution time. The tool used to generate the graph was Blackfire<\/a>, a powerful and easy to use PHP profiler. Since that first graph, many more have been created and much code has been optimized.<\/p>\n

The first optimizations<\/h2>\n

The easy to read and intuitive Blackfire images quickly spread over Github and led to a speedup in the AppStore, thanks to caching the responses from apps.owncloud.com<\/a>. While the first fetching data from the app store takes a few seconds, subsequent page loads can rely on the cache, making a big perceptible difference for the end user, easily speeding up operations 20 times. The developer making the changes, Lukas, provided a nice graph<\/a> showing the difference.<\/p>\n

\"LukasReschke<\/a>

LukasReschke on Blackfire<\/p><\/div>\n

Getting serious<\/h3>\n

Lukas, who ‚discovered‘ Blackfire initially, started to simply upload files and use the profiler to see what took a lot of time. This resulted in a series of changes to file related operations, collected in this Github issue<\/a> — starting small with a 4% performance gain on the processing of `normalizeUnicode`<\/em><\/a>, adding a simplification to a check on paths<\/a> and ending with a performance boost of 24%<\/a> for simple searches.<\/p>\n

Speed up file deletion<\/h3>\n

Another extensive profiling session resulted in even more impressive results after Thomas noticed that deleting a folder with 3,000+ files in it took a huge amount of time to get processed. In total over 35 minutes, showing in the profiler as this graph<\/a>. It shows clearly that the bottleneck are nearly 40,000 SQL requests which are causing over 70% of the total request time.<\/p>\n

In the first iteration the request time went down to 10 minutes by reducing the SQL queries by a factor of 10<\/a>. Now, another interesting bottleneck showed up: the PHP method `strlen`<\/em> was called over 60,000 times by a component from a third party library that ownCloud uses, ZendSearch, from Zend Framework<\/a>. Focus moved to this library and it became clear that `strlen`<\/em> was used to just decide if a string has a defined length. Somebody came up with the idea of simply using the native language construct `isset`<\/em> rather than a function like `strlen`<\/em>. Instead of doing this…<\/p>\n

if (strlen($name) > 49) {\n\t...\n}<\/pre>\n
… you can do this which results in the exact same behavior, except that it is a lot faster:<\/p>\n
if (isset($name[49])) {\n\t...\n}<\/pre>\n
\"The<\/a>
The difference in ZendSearch<\/p><\/div>
\nWe provided a patch upstream<\/a> which reduces the load for this code path greatly.<\/p>\n
For the deletion use case, the improvement moved the instant processing to update the search index to a background job. This results in the execution time going down to just 1 minute<\/a>! There is still room for further improvements but it is a great first step and it took only a moderate amount of effort to achieve. The change from `strlen`<\/em> to `isset`<\/em> turned out to also be relevant in ownCloud code<\/a>, bringing a nice 20% increase in speed for processing a million path lookups.<\/p>\n
\"impact<\/a>
Impact of the first change in Punic<\/p><\/div>\n
Four Punic contributions<\/h3>\n
A debugging session led to finding another performance bottleneck<\/a> with very large folders and another set of upstream improvements. The first patch going in the Punic<\/a> localization library was caching getTimezoneNameNoLocationSpecific<\/em><\/a>, followed by this<\/a> and this<\/a>. Familiar by now might be a patch to use isset()<\/em> instead of strlen()<\/em><\/a>! On the ownCloud side, reusing the array key of mimetypes<\/a> helped improve performance and the session also found a bug<\/a> which was subsequently fixed<\/a>. More ideas are still coming in<\/a> while this issue is tackled.<\/p>\n
Architectural impact<\/h3>\n
Not all performance issues can be relatively easily fixed. Lukas discovered that the custom OC_Autoloader in ownCloud is quite a bit slower<\/a> than the Composer loader we are transitioning to. These loaders bring in functionality from third party libraries ownCloud uses when needed in a piece of code. The OC_Autoloader is the ‚old‘ way ownCloud has been handling that and the fact that OC_Autoloader wasn’t fast had been noted before<\/a>. ownCloud 8 will introduce the Composer loader<\/a>, a dependency manager strongly inspired by Node’s npm and Ruby’s bundler. Clearly, it not only saves the ownCloud team the work of maintaining their own loader but also offers performance improvements! Moving completely<\/a> to the Composer loader is already on the agenda for ownCloud 8.1<\/a>.<\/p>\n
Another change coming in ownCloud 8.1 is related to memcache<\/a>, something Lukas bumped into when going through code with Blackfire. It is a relatively big change but there is already code<\/a> and some testing has been done. Due to the impact, however, it will have to wait until the next release.<\/p>\n
Impact on reporting of issues<\/h3>\n
Blackfire is also starting to be used by users<\/a> who report performance issues. When asked to run a profiling run<\/a>, the results prompted a first optimization already<\/a> and more are in the pipeline<\/a>.<\/p>\n
Nice things about Blackfire<\/h2>\n
As most of us here at ownCloud work remotely (as in „not in one city at all“) this is a nice way to share the results with other contributors. Blackfire makes it really easy to inspect all the calls that are made during a request and to spot bottlenecks in our application even if you didn’t run the profiler.<\/p>\n
It has an easy to use user interface that just works and the profiler has a minimal setup effort. Running an agent, loading a PHP module and clicking on a browser plugin button. That’s all you need to get some nice graphs with really valuable metrics about your request. Of course we have some wish list items already, like the ability to more easily grab images from the result graphs (or export them to other formats) and a Blackfire Firefox plugin! Last but not least, it would be great if it could do the performance optimizations automatically. For this, integration with Skynet was suggested.<\/p>\n
Enthusiasm around the tool is becoming very visible, with a large number of performance related pull requests showing up with Blackfire graphs<\/a>. We expect more improvements in ownCloud performance thanks to Blackfire – a great tool!<\/p>\n
This post also appeared on the Blackfire blog<\/a>. Thanks to Morris Jobke and Lukas Reschke for providing much of the content of this blog. And of course to everybody for the awesome work! If you like to follow ownCloud development, keep an eye on our development reports – see our overview over December<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"
A little under two weeks ago, a performance improvement in ownCloud was accompanied by a graph showing the difference in code execution time. The tool used to generate the graph was Blackfire, a powerful and easy to use PHP profiler. Since that first graph, many more have been created and much code has been optimized. […]<\/p>\n","protected":false},"author":7,"featured_media":78665,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","inline_featured_image":false,"footnotes":""},"categories":[342,332,333,335,339],"tags":[],"class_list":["post-7752","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud","category-coding","category-community","category-development","category-owncloud"],"acf":[],"_links":{"self":[{"href":"https:\/\/owncloud.com\/de\/wp-json\/wp\/v2\/posts\/7752","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/owncloud.com\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/owncloud.com\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/owncloud.com\/de\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/owncloud.com\/de\/wp-json\/wp\/v2\/comments?post=7752"}],"version-history":[{"count":0,"href":"https:\/\/owncloud.com\/de\/wp-json\/wp\/v2\/posts\/7752\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/owncloud.com\/de\/wp-json\/wp\/v2\/media\/78665"}],"wp:attachment":[{"href":"https:\/\/owncloud.com\/de\/wp-json\/wp\/v2\/media?parent=7752"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/owncloud.com\/de\/wp-json\/wp\/v2\/categories?post=7752"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/owncloud.com\/de\/wp-json\/wp\/v2\/tags?post=7752"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}