Network activity 5:

Standardised data protocols and central database

Objectives and expected impact

An important goal of EUSAAR is to improve accessibility to European monitoring data and to provide users with better tools to extract information from archiving systems.
The exact achievements of the activity will be defined according to a functionality requirements document during the first year of the project, but the main components will include:

Central server hosting the actual database application. This will be a more or less unchanged version of the current system and it will be possible to continue the operational work of the EMEP/CCC in parallel to the new developments.

File processing system. An automatic procedure for import of measurement data will be set up to handle both regular EMEP data and additional aerosol data submitted through EUSAAR.

Database driven web portal. A web portal will be implemented on top of the database and file processing system, and will provide the users with an easily accessible interface to both EMEP and EUSAAR data.

Interface towards external services. In order to increase the use of monitoring data, it is necessary to provide external data services with proper access to the system. This will include a machine-to-machine interface, enabling an automated operational service to contact the database, query the DB for available measurements and retrieve the requested data on a standardised format.

An essential part of the database developments is to make it able to run operationally after the end of the project. The efforts put down in NA5 aims to improve the routines for data flow and dissemination, while the EMEP programme will fund the continued operation. Monitoring activities may then also include advanced aerosol parameters from European super-sites.

Description of work

The implementation of an improved database needs to be done according to a set of properly defined user requirements. Most of the users needs were acquired through the scoping part of the GMES initiative where the EC funded project(s) CREATE/DAEDALUS performed user-surveys of science and policy needs for aerosol products. The output of this project (CREATE) will feed into NA5 and serve as a starting point for discussions concerning functionality requirements of the database. A workshop for the whole EUSAAR consortium will be arranged during the first year of the project and a functionality requirements document will be produced to define what the database will do and how it should be presented. Following the requirements for functionality, a system design document will be produced. This will include a detailed specification of all system components, functions, data flow and operation procedures of the data centre. In addition it is necessary to define standardised protocols for submitting data to EUSAAR. These will be based on already existing formats and the developments through the CREATE project, the database will support data files in (at least) two formats, namely NASA-Ames (EMEP) and NARSTO (WDCA). These format needs to be supplemented with specification for new instrumentation and methods.

The developments of an enhanced data centre will commence after the completion of the system design document. A web portal will be set up to enable users to search for and access selected datasets and data may be downloaded through a web browser. The first version of the database driven web portal is to be delivered without all the specification in the system design document and it will therefore not include any statistical tools or graphical modules for on-line visualisation data. These are to be implemented in later versions of the portal. The database driven web portal will be operational by month 18, but developments will continue to implement facility for automatic upload of data. This will check the format of incoming data, insert the files in the database and mark them as preliminary. The first statistical and graphical tools (basic toolbox) will furthermore be developed and implemented by month 24.

A second database workshop will be arranged after implementation of the basic toolbox. The focus will be on current status, and feedback from the consortium will allow functionality specifications to be further refined. Implementation of high level statistical and graphical tools will commence after the workshop and will be completed by month 30. These developments will have to be done in collaboration with JRA3 since both efforts require similar types of functionality.

Halfway through the project, the database will have matured enough to allow super-sites to upload data and for users to retrieve and analyse the measurements. As for upload of data, these also include the measurements from the MAX-DOAS instruments of JRA1. Developments of interfaces towards other data services and external users will thus commence. The developments performed in JRA3 will have enabled automatic data retrieval of advanced aerosol parameters, but a connection between the EMEP database and the near real time (NRT) system needs to be established. An interface between the two systems will be developed, enabling the PIs of the super-sites to analyse and control their raw-data, aggregate the measurements into time-averaged data sets and submit the files to the EMEP data centre. All will be possible on-line. The NRT system will have developed a machine-to-machine interface for automatic distribution of data to external data services such as the GEMS project of the ECMWF. A similar mechanism needs to be implemented also for the EMEP database and will allow for future integration with other services, possibly through a data GRID.