Abstraction Benchmarks

Evaluating the costs of object-oriented features applied to Fortran 90 Programs

We have developed a benchmark program to measure the performance of the new features of Fortran 90, including their usage in object-oriented software development. The benchmark measures various forms of abstraction, based on ideas from the Stepanov C++ Benchmark. The Stepanov benchmark performs a simple computation in a loop, adding STL (Standard Template Library) style iterators of various kinds in multiple combinations to increase the level of abstraction a compiler must address. In the ideal world, the compiler should remove these abstractions so that the modified tests execute as quickly as the original basic test. The Stepanov benchmark performs 13 tests with increasing amounts of abstraction at each level.

Our benchmark also performs a simple computation in a loop as the initial test, however, the subsequent 20 tests add abstraction that more appropriately reflect how actual programs may be developed. We return three kinds of "abstraction measurements" that indicate the effects of the tests: abstraction of Fortran 90 array-types, inheritance abstraction, and the overall abstraction.

Sample benchmark runs and availability

This benchmark is still in development, please return to this site to learn when it will be available to the general public. We have applied the benchmark to compilers and machines from various vendors with varying results. Some compilers cannot build the benchmark at all while others perform well. Since different architectures were used, we will include a sample run from the IBM test (since comparison across different architectures would not be appropriate).

Iterations (2000 minimum) : 2000
Working .................... 
 
 Test    Time    Base Ratio
   1     0.290     1.000
   2     0.280     0.966
   3     0.300     1.034
   4     0.320     1.103
   5     0.300     1.034
   6     0.400     1.379
   7     0.280     0.966
   8     0.270     0.931
   9     0.280     0.966
  10     0.430     1.483
  11     0.380     1.310
  12     0.270     0.931
  13     0.620     2.138
  14     0.640     2.207
  15     0.290     1.000
  16     0.620     2.138
  17     0.570     1.966
  18     0.290     1.000
  19     0.580     2.000
  20     0.590     2.034
 
 Array Syntax Abstraction Measure:    1.178
 Inheritance Abstraction Measure:     1.500
 Overall Abstraction Measure:         1.303

The "base ratio" compares the speed of the first test (test 1) to each subsequent test by division. Note that Test 6 returns 0.000 regardless of the iteration length, so we assumed a value of 0.400 seconds which may be indicative of the actual value of this test. Test 6 involves Fortran 90 automatic arrays with use association. An abstraction measure close to 1.000 is desired.

UP