Words from FOSS4G 2010 and Famous WMS Shoutout Winner

First, I want to apologize, I just got Internet access while at FOSS4G, so I couldn’t provide good coverage so far… But don’t worry, I’ll make sure you get all the most important announcements and news, probably some time next week. At least for the elements other Slashgeo editors won’t have already shared.

Meanwhile, here’s my notes and the results of the famous WMS Shoutout, which just ended minutes ago.

– WMS benchmarking, 8 teams this year instead of the usual 2.
– This year with Cardcorp, GeognoSIS, Constellation-SDI, ERDAS APOLLO, GeoServer, – Mapnik, MapServer, Oracle MapViewer, QGIS MapServer.
– Both open source and proprietary solutions compared with the same datasets, the same WMS requests, same output formats, etc.
– Testing 18 gigs of vector data.
– Testing 120 gigs of raster data.
– Using real Spain data.
– From 1 to 64 parallel clients requests (2152 requests total).
– Main winners are the users, because we are the ones who will benefit from the improvements done to the WMS servers (one rule of the Shootout is that modifications must become available to the users, I guess this also applies to proprietary software competitors).
– This year, benchmark includes bottlenecks from CPU, disk access, network access and remote database access.
– If you wonder, these guys really went deep in the benchmarking analysis (such as OS-level caching), take a look at the slides. They do try to do a serious benchmarking comparison.
– There was serious differences in servers which were disk-bound vs disk-unbounded (CPU-bound).
– Tests included dbf files of over 2 gigs.
– Of course, there is performance in terms of time to answer requests, but there is also the quality of the rendered tiles, which is more subjective.
– All contestants agree that this was a lot of work to participate to the Shootout, but it was worth.
– GeoServer got many great improvements for the sake of the Shootout, same for MapServer.
– Mapnik: even built their own new WMS server named Paleoserver.
– Oracle MapViewer: found quite a few bugs and bottlenecks, and yes, they plan to participate next year.
– QGIS MapServer: integrating desktop GIS and server. Missing time to get consistent results.
– Constellation-SDI: it’s open source software (LGPL). First serious work in performance, many challenges found.
– Cadcorp GeognoSIS: Windows-only server. Added many improvements for the sake of the Shootout.
– ERDAS APOLLO: exciting benchmarking project. Made many improvements too. They’re refusing to publish results because CPU-bound results cannot be compared with Disk-bound results. Yeah, contestants can step aside from displaying results for any reason they see fit. 

– And now the results…     
– MapServer wins first test, Mapnik second
– second test, MapServer first, GeoServer second
– third test, Constellation, map server GeoServer all doing good
– fourth test, MapServer generally faster
– fifth test, still MapServer ahead
– on the last test, Mapnik doing good
– Overall, one can say that MapServer can be considered the faster of them all at the moment. Of course, considering that some servers were able to cache in RAM better than others.
– For next year, the whole WMS process will start much earlier.
– The show ended by the WMS-song.. Pretty funny.
– Obviously, the full results will be available soon on the OSGeo website (search Benchmarking on their wiki).      

One comment

  1. Hi,

    While working through the performance benchmarks, it was noticed that results between applications could not be directly compared and were not reflecting the relative performances of the servers. Not enough time was available to normalize the benchmark and produce homogeneous results measuring the exact same scenario for all participants. The following movie (hosted on the OSGeo server) was produced showcasing a rendering example between different severs:

    This screen demo was captured on 8th September when no other testing was taking place on the server machines. The test client is built using OpenLayers widgets and all maps are synchronized to the top left master widget. There is no delay in requesting maps so it is a true performance indicator based on when maps are returned and rendered.

    Two servers are running on the Windows box (MapServer Windows and ERDAS APOLLO). The two other servers are running on Red Hat Enterprise Linux (MapServer RHEL and GeoServer). This movie also demonstrates the efforts that were done by each team to reach rendering quality, and styling compliance with the defined benchmark specifications.

    Regards,

    Luc Donea