Benchmarking Qizx

Since one of Qizx's distinctive characteristics is its querying speed, we give concrete execution measurements on real-size examples.

• Qizx is compared with two open-source products on the XMark benchmark.
• An example of multi-criteria search is given on a large database of small documents.

Please remember that like for any benchmark, figures given here are highly dependent on test conditions (details below), hence subject to variations from one platform to another, and are given for information purpose only.

See also: performance facts

Test platform

All following tests have been performed on the same commodity PC:

• processor: Intel i7 860 quad core at 2.8 GHz (one core is actually used)
• memory: 8 Gigabytes,
• hard-disk: SATA 3 Gb, 7200 rpm,
• JVM: Sun Hotspot™ 1.6.0 64-Bit Server,

• OS: GNU/Linux OpenSuse 11.2 (kernel 2.6.31). Measurements on Windows 7™ are very close.

XMark

XMark is a XML database benchmark designed by the CWI Amsterdam and other organizations. It consist of 20 queries on a single document (whose reference size is 112 Megabytes, but can be scaled at will).

Note: another comparison of several products on XMark is available on the site of Monet DB, but it does not include Qizx and is performed on a different platform.

XMark favors massive scans and joins, and puts little emphasis on value-based queries (unlike example in the section above).

The following test compares Qizx with:

• eXist, a widely used open-source Java native XML database.
• Qizx/open, which is not a database engine: it parses documents in memory and works without indexes.
• Monet DB XQuery 5.22, the XML Query database also developed by CWI. Monet DB is written in C and has been in the past years deemed the fastest XQuery database engine available. [Timings not yet available]
• Berkeley DB XML 2.5.16, an XML Query database now owned by Oracle. [Timings not yet available]

These 3 first products share the same characteristic that queries can be executed "out of the box" without need to define indexes manually.

For legal reasons, we cannot not include tests performed on commercial products (some vendors explicitly prohibit it), but that does not make a real difference.

Test conditions:

• each query is executed 2 times and the average execution time is computed.

• XMark Queries are slightly modified to return the count of result items without actually performing XML output, but we made sure that both systems actually materialize results (we did not simply rely on the count() function which is sometimes specially optimized).

• Measurements are made after an initial "warm-up" phase (executing all queries once).This is made necessary by Java dynamic class loading and compilation, and implies that caches are not empty. Such conditions are deemed realistic in a server environment
• All times are in milliseconds.
• DNF means Did Not Finish within 15 minutes. AL means "terminating but abnormally long".
• In all respects, please remember that measured times in Qizx are highly dependent on caching effects. As a matter of fact, the second and following executions of a query can be much faster than the first execution, due to caching.

Remarks:

1. The geometric average is more meaningful when aggregating short queries and long queries.