Code coverage

at 2006-05-12 in Unittests, Experiments by friebe (0 comments)

The XP framework's unittesting API has been experimentally extended by a userland code coverage implementation. For an example of a generated code coverage for the util.Binford class, have a look here.

How it works
The trick was to use the declare statement in combination with register_tick_function(), trigger_error() and set_error_handler().

Abbreviated example:

<?php 
  function coverage($level, $message= NULL, $file= NULL, $line= NULL) {
    // Gory details snipped
  }

  declare(ticks= 1);
  set_error_handler('coverage');
  register_tick_function('trigger_error', 'coverage', E_USER_NOTICE);

  // The sourcecode we want covered
?>

Implementation details
The coverage() function will now be passed E_USER_NOTICE, "coverage" as well as file name and line number (the latter two from the Zend Engine's internals functions zend_get_executed_lineno() and zend_get_executed_filename).

By storing these in a function-static variable and later retrieving them by calling coverage() with a level of -1, we are able to generate a trace of files and lines executed.

The fetched results look like this:

{
  "lang/Object.class.php" => {
    25 => 2
    26 => 1
  }
}

...meaning line 25 in lang/Object.class.php was executed twice and line 26 once.

About statements and blocks
Take the following sourcecode:

<?php 
  $a= 1;
  $b= 1;
?>

This will produce a tick in line 1 and a tick on line 2, which seems absolutely logical.

The "mess" starts when statements span multiple lines:

<?php 
  $greeting= (strlen('Hello') == 5
    ? 'Hello'
    : 'Moto'
  );
?>

In this case, a tick is generated at line #4.

It gets worse with blocks of code, such as the following:

<?php 
  if (TRUE) {
    $executed= TRUE;    // tick
  } else {              // tick
    $executed= FALSE;
  }                     // tick
?>

<?php 
  $executed= 0;         // tick
  for ($i= 0; $i &lt; 5; $i++) {
    $executed++;        // tick (5)
  }                     // tick (6)                
?>

<?php 
  switch (strlen('Hello')) {
    case 5: $result= TRUE; break;             // tick
    default: return throw(new PrerequisitesNotMetError('strlen() broken!'));
  }                                           // tick
?>

For a complete overview of when ticks are produced and when not, have a look at the TickTest class' sourcecode.

To correctly mark lines as executed (or not) we will have to perform an analysis of the sourcecode, parsing it into its fragments. Look for instance at the ternary operator example. Here we need to mark all four lines as executed instead of only coloring the last line.

Parsing PHP code into fragments
To parse PHP sourcecode into fragments (an expression, a block, a comment, ...) we use the tokenizer functions. The basic workings are easy:

When we encounter a semicolon (;), an expression ends
When we encounter an opening curly brace ({), a block starts
When we encounter a closing curly brace (}), a block ends
For tokens which may possibly contain newlines, we increase the line number by the amount of line breaks we find

Of course, it's a bit trickier than that:

Curly braces can also be used for string offsets ($string{0}) or expression evaluation ($this->{$key})
A heredoc string is not a single token but has T_START_HEREDOC and T_END_HEREDOC token objects (and others inbetween)

Putting it all together
Now that we have the line numbers ticked and by comparing those to code fragments, we can create the coverage report.

Take the following code:

<?php 
  $a= substr(
    $string,
    $offset,
    $length
  );
?>

As stated before, a tick will be generated at the end of the function call to substr() in line 5, making the coverage information look like this:

{
  "substr-example.php" => {
    5 => 1
  }
}

By parsing the sourcecode into fragments, we'll have:

[
  Expression(code= "$a= substr(...)", linestart= 1, lineend= 5)
]

...and can then see we'll also need to mark lines 1 - 4 as executed.

That's it! Besides a couple of oddities still needing to be adressed (return $a; doesn't produce a tick, how would we know it was executed?) this should be added to the XP framework's unittesting functionality soon. It's a userland implementation, so no need for external tools (except as browser to look at the reports

You can subscribe to the XP framework's news by using RSS syndication.

Code coverage

Subscribe

Categories

Related