MICROSOFT'S SQL SERVER 2000 database, which is set to ship within a month, suffered a double embarrassment this week. Record TPC-C benchmark numbers, which Microsoft has touted for the past five months, were scratched from the official records after the Transaction Processing Council found the results "noncompliant." And IBM published numbers twice as high as Microsoft's with its own DB2 database.

   ADVERTISEMENT
  

Free IT resource

Open Source Business Conference (OSBC) May 22-23, 2007

Sponsored by OSBC

Free IT resource

TechNet: More ways to know it, share it, and keep it running.

Sponsored by Microsoft

RELATED LINKS
»  Canonical chases deals to ship Ubuntu Server preinstalled
»  Sun delivers first UltraSparc T2-based servers
»  IT trainer offers master's degree for hackers
»  Platforms RSS feed 

IDG ENTERPRISE NETWORK
• More Operating Systems News...  (ComputerWorld)
• CrossOver Office aims to ease a switch to Linux  (ComputerWorld)

TOP NEWS 


IT SOLUTION SEARCH
At the launch of Windows 2000 in February, Microsoft Chairman and Chief Software Architect Bill Gates touted the performance numbers, which showed SQL Server 2000 and Windows 2000 processing 227,079.15 transactions per minute (TPM-C), the highest ever measured. The result, which beat high-end RISC servers from IBM and Sun Microsystems, was achieved with a cluster of 12 Compaq ProLiant 8500 servers. The TPC-C numbers have figured highly in presentations by Microsoft executives over the last five months.

But at a meeting of the TPC last Thursday, the benchmark "was found noncompliant to our policies," said TPC chairman Jerrold Buggert. "Given the nature of the problem, the tests need to be redone."

The Compaq/Microsoft benchmark was challenged because the tested configuration didn't allow for the primary key of a distributed database to be updated. TPC declined to identify who challenged the results, citing the confidentiality of the challenge procedure.

Steve Murchie, group product manager for SQL Server at Redmond, Wash.-based Microsoft, said the TPC's rules are ambiguous on this issue and claimed that previous benchmarks published by Houston-based Compaq Computer's Tandem unit and by Oracle, in Redwood City, Calif., have also failed to implement primary key updates. Officials from those companies weren't available for comment by the deadline for this article.

According to Murchie, Microsoft has now modified the SQL Server 2000 code to include the required feature, and it will be in the product when it ships in the next few weeks. Murchie said the missing feature did not materially affect performance in the TPC-C benchmark. Microsoft will rerun the tests with the newest SQL Server 2000 code and publish a new benchmark in about a month, Murchie said.

It is not unprecedented for benchmark results to be removed in this way, said Jonathan Eunice, an analyst at Illuminata, in Nashua, N.H. TPC-C benchmark results are open to peer review. "You are policed by your competitors," Eunice said. "It's not pretty, but there is a kind of efficiency to it."

Rerunning the benchmarks could easily cost Microsoft and Compaq $1 million, Eunice estimated.

The SQL Server-on-Windows 2000 benchmark would have lost its first-place ranking this week anyway, with Armonk, N.Y-based IBM's publication on Wednesday of a record 440,870 TPM-C result for DB2 running on its NetFinity servers.

The IBM benchmark does return Windows 2000 to the top of the TPC-C ranking, since it was run using Windows 2000 Advanced Server.

Eunice said TPC-C benchmarks performed on clusters -- such as the IBM NetFinity and Compaq ProLiant clusters -- aren't as important as single-server numbers. "From a manageability perspective, people don't like to run a 16-way cluster if they can get a single SMP (symmetric multiprocessing) server [with the same performance]," Eunice said.