9.7.2 Steps in DP&M

Data processing, validation, and management procedures for an upper-air meteorological monitoring program would typically include the following steps, which should be described in the QAPP:

• Collection and storage on-site (as appropriate) of the “raw” signals from the upper-air sensors, followed by real-time processing of the “raw” data by the data acquisition system to produce reduced, averaged profiles of the meteorological variables. The reduced data are stored on the data acquisition system's computer, usually in one or more ASCII files.



• Transfer of the reduced data to a central data processing facility at regular intervals (e.g., daily). Once the data are received at the central facility, they should be reviewed by an experienced data technician as soon as possible to verify the operational readiness of the upper-air site. Backup electronic copies of the data should be prepared and maintained on-site and off-site.

Data collected by the remote sensing systems can usually be obtained by polling the data system at a site from the central facility using a personal computer, modem, and standard telecommunications software. Other options that are available for communications with a remote upper-air site include leased-line telephone service, local or wide area network (LAN, WAN) connections, Internet access, and satellite telemetry. For immediate turnaround of radiosonde data, the upper-air operator can transfer the sounding data to the central facility using a personal computer equipped with a modem and communications software. There must be a bulletin board system (BBS) operating at the central facility, or some other means provided to receive the data (e.g., via an Internet access). Alternatively, if a one- or two-day delay is acceptable, the operator can mail the sounding data to the data center.

Please note that the initial review of the data is not very time consuming, but it is an extremely important component of a successful upper-air program. It is at this stage that most problems affecting data quality or data recovery will be detected. If the upper-air data are not reviewed at regular, frequent intervals, the risk of losing valuable information increases. If the data are reviewed frequently, then problems can be detected and corrected quickly, often the same day, thereby minimizing data losses. At a minimum, the operational readiness of an upper-air monitoring site should be checked regularly. Likewise, maintaining backup copies of the data at each stage of processing is extremely important. Backup copies should be kept at the central data processing facility and at a separate, off-site location(s) to ensure that no data are damaged or lost.

• Additional post-processing is performed as required (e.g., reformatting the data using a different database format than that produced by the data acquisition system) to produce the version of the data that will be subjected to final quality control validation.



• At this stage, the data are usually said to be at “Level 0” quality control validation, meaning that they are ready for quality control screening and final validation.



• Quantitative screening of the data can be performed using quality control software to identify outliers or other observations that are possibly in error or otherwise appear questionable.



• A final review of the data should be performed by an experienced meteorologist who understands the methods used to collect the data and who is knowledgeable about the kinds of meteorological conditions expected to be revealed in the data.

This is the process that brings the data to what is usually referred to as “Level 1” quality control validation, meaning that the data have been subjected to a qualitative (and often quantitative) review by experts to assess the accuracy, completeness, and internal consistency of the data. At this stage, data that have been determined to be in error are usually removed from the database, and quality control flags are assigned to the data values to indicate their validity. It is also at this stage that final calibrations should be applied to the data as necessary, as well as any changes required as the result of the system audits. Additional screening of the data based on comparisons to other independent data sets may be performed, which is part of the process to bring the data to “Level 2” quality control.

• Some final processing may be necessary to convert the data to the format that will be used to submit the information to the final data archive. 

Final documentation should be prepared that summarizes sampling strategies and conditions; describes the results of audits and any actions taken to address issues raised by the audits; identifies any problems that adversely affected data quality and/or completeness; and describes the contents and formats of the database. Typically, a copy (electronic and/or paper) of this documentation accompanies the submittal of the data to the final data archive. Once the above steps are completed, the data are ready to be submitted to the upper-air archive. Several options for creating an archive are available, ranging from a simple repository to complex database management systems (DBMS).

9. UPPER-AIR MONITORING 
9.1 Fundamentals  
      9.1.1 Upper-Air Meteorological Variables  
     9.1.2 Radiosonde Sounding System  
     9.1.3 Doppler Sodar 
     9.1.4 Radar Wind Profiler 
     9.1.5 RASS  
 9.2 Performance Characteristics  
     9.2.1 Definition of Performance Specifications  
     9.2.2 Performance Characteristics of Radiosonde Sounding Systems 
     9.2.3 Performance Characteristics of Remote Sensing Systems  
 9.3 Monitoring Objectives and Goals  
     9.3.1 Data Quality Objectives  
 9.4 Siting and Exposure
 9.5 Installation and Acceptance Testing 
 9.6 Quality Assurance and Quality Control 
     9.6.1 Calibration Methods  
     9.6.2 System and Performance Audits  
     9.6.3 Standard Operating Procedures 
     9.6.4 Operational Checks and Preventive Maintenance  
     9.6.5 Corrective Action and Reporting  
     9.6.6 Common Problems Encountered in Upper-Air Data Collection 
 9.7 Data Processing and Management (DP&M) 
      9.7.1 Overview of Data Products  
     9.7.2 Steps in DP&M 
     9.7.3 Data Archiving  
 9.8 Recommendations for Upper-Air Data Collection