I've finally gotten around to making a comparative performance test of CWF. As a benchmark, I've chosen CodeIgniter framework, since it is fairly popular, known for its good performance and easy to install. The test system was a Windows machine with AMD 64 X2 processor and XAMPP Lite 1.7.2 installed. (FYI, this version of XAMPP comes with PHP 5.3.)
The tests will consist of outputting "Hello Wold" and setting the page title to some value. Of course, some people would say that bare-bone test applications like this don't show the real-life performance of a system. That is true. However, they are a pretty good way to measure the system's overhead.
First, here is the baseline test that runs a single PHP file containing only <? echo 'Hello World';?>. It uses Apache ab utility to run 200 requests with the concurrency of 5 (the same settings as for all the other tests).
# ab -n 200 -c 5 http://localhost/baseline.php
...
Concurrency Level: 5
Time taken for tests: 0.438 seconds
Complete requests: 200
Failed requests: 0
Write errors: 0
Total transferred: 50600 bytes
HTML transferred: 2200 bytes
Requests per second: 457.14 [#/sec] (mean)
Time per request: 10.938 [ms] (mean)
Time per request: 2.188 [ms] (mean, across all concurrent requests)
Transfer rate: 112.95 [Kbytes/sec] received
...
After this, everything was in order to run the real benchmarks. The version of CodeIgniter I downloaded was 1.7.1. I didn't write the view and controller, but rather copied them from a tutorial on devshed.com.
codeigniter\system\application\controllers\helloworld.php:
<?php
class HelloWorld extends Controller{
function HelloWorld(){
parent::Controller();
}
function index(){
$data['title']='My first application created with Code Igniter';
$data['message']='Hello world!';
// load 'helloworld' view
$this->load->view('helloworld', $data);
}
}
?>
codeigniter\system\application\views\helloworld.php:
<html>
<head>
<title><?php echo $title;?></title>
</head>
<body>
<h1><?php echo $message?></h1>
</body>
</html>
Pretty straightforward. Here are the results of running Apache ab:
# ab -n 200 -c 5 http://localhost/codeigniter/index.php/helloworld
...
Concurrency Level: 5
Time taken for tests: 4.359 seconds
Complete requests: 200
Failed requests: 0
Write errors: 0
Total transferred: 75600 bytes
HTML transferred: 27000 bytes
Requests per second: 45.88 [#/sec] (mean)
Time per request: 108.984 [ms] (mean)
Time per request: 21.797 [ms] (mean, across all concurrent requests)
Transfer rate: 16.94 [Kbytes/sec] received
...
Of course, for testing Caffeine Web Framework I used the latest development version. It is marked as revision 58 in SVN. I wrote a module and a template that do roughly the same as the CodeIgniter components above.
framework\libs\HelloWorld.php:
<?php
class HelloWorld extends LightModule{
protected $permittedActions = array('index'); //users can run this
function index(){
Html::$title[] = 'My nth application created with CWF';
Template::show('HelloWorld/index', array('message' => 'Hello World'));
}
}
framework\templates\HelloWorld\index.php:
<h1><?=$arg['message']?></h1>
I did't need to add HTML boilerplate code, since the CWF does that automatically. There is a default Http::$prism callback, which surround the output with standard HTML tags, and populates the title from $Html::title.
Here are the benchmark results for CWF:
# ab -n 200 -c 5 http://localhost/framework/?HelloWorld
...
Concurrency Level: 5
Time taken for tests: 1.234 seconds
Complete requests: 200
Failed requests: 0
Write errors: 0
Total transferred: 155800 bytes
HTML transferred: 101600 bytes
Requests per second: 162.03 [#/sec] (mean)
Time per request: 30.859 [ms] (mean)
Time per request: 6.172 [ms] (mean, across all concurrent requests)
Transfer rate: 123.26 [Kbytes/sec] received
...
That's 3.5 times more requests per second, while outputting more data. Additional testing had shown that it's possible to gain extra 20 requests per seconds with CWF by setting Http::$prism to NULL and rewriting the template to output the entirety of HTML. Considering that CodeIgniter is (one of?) the fastest major Framework out there, those are pretty good results.
According to XDebug, both frameworks spent most of the time loading files. Apparently, using __autoload() function, putting everything into separate files, and trying to shrink them down as much as possible paid off. I don't intend to stop here, though. Just by running these test I got a few ideas about removing and merging certain "core" files, which should reduce the overhead even further.
