|
|
# Introduction
|
|
|
Data quality issues are often found some time after data is collected, ingested, archived and then delivered to an end user. The Data Quality Report (DQR) web service allows a user application to automatically query the DQR database in order to retrieve the most up-to-date information regarding the quality of their data holdings. The service returns a list of time blocks (start/end time pairs) that delineate when a user should not use or should be suspicious of their ARM data. Optionally, additional information about the DQR responsible for the time blocks in the report can be returned. This might include the DQR identifier string and the subject details.
|
|
|
|
|
|
## Build the Query
|
|
|
Generate a URL with the datastream and variable names encoded as parameters to the webservice.
|
|
|
For example, to search for DQRs affecting the datastream **sgplssondeC1.c1** and the variable **rh**, use this URL:
|
|
|
|
|
|
http://www.archive.arm.gov/dqrws/ARMDQR?datastream=sgplssondeC1.c1&varname=rh
|
|
|
|
|
|
When submitted, you will see output something like this:
|
|
|
<pre>
|
|
|
944199060|944285400
|
|
|
1031333220|1031338440
|
|
|
1031290020|1031296680
|
|
|
1033644000|1033651200
|
|
|
1126265520|1126272720
|
|
|
</pre>
|
|
|
|
|
|
The default is a list of start and end times in seconds since Jan-1-1970, the same time units used in ARM data files. You can specify exactly the type of DQR information you would like returned, the time format, and whether you are interested in known incorrect (aka red), suspicious (aka yellow), or both types of DQR.
|
|
|
|
|
|
For example, if you would like to see the DQR id and prefer to see time as encoded in ARM filenames instead, add the dqrfields and timeformat parameters to the URL as so:
|
|
|
|
|
|
http://www.archive.arm.gov/dqrws/ARMDQR?datastream=sgplssondeC1.c1&varname=rh&timeformat=YYYYMMDD.hhmmss&dqrfields=starttime,endtime,dqrid
|
|
|
|
|
|
<pre>
|
|
|
19991203.053100|19991204.053000|D000127.2
|
|
|
20020906.172700|20020906.185400|D020909.1
|
|
|
20020906.052700|20020906.071800|D020909.2
|
|
|
20021003.112000|20021003.132000|D021006.1
|
|
|
20050909.113200|20050909.133200|D050912.2
|
|
|
</pre>
|
|
|
|
|
|
## Automated Query
|
|
|
|
|
|
In the next sections we will delve into retrieving machine readable data quality information using web queries from several popular programming languages. Full documentation of the web service options can be found in [[Parameter Descriptions]]. These examples convey the basic techniques in performing a query and parsing the simple delimited response. For fully unattended code, it is advisable to check error codes from all connection attempts to ensure that:
|
|
|
|
|
|
* web service host is reachable (no network issues)
|
|
|
* valid parameters were passed
|
|
|
* web service and associated database is up and running
|
|
|
|
|
|
Examples provided in this repository will help you check that these conditions are met. By doing these checks, you can be confident that a blank response is due to an absence of DQRs rather than a problem with the service or the way it is being called.
|
|
|
|
|
|
[[Examples Using R]] |